AAOS OrthoInfo provides more accessible information regarding femoroacetabular impingement than ChatGPT-4 while information accuracy is comparable

Abstract

Abstract As artificial intelligence (AI) Large Language Models (LLM) like ChatGPT become more common in healthcare, patients increasingly use them to find medical information. While ChatGPT may support health literacy, its readability and accuracy compared to established resources remain unclear. The purpose of this study is to evaluate ChatGPT-4 responses on femoroacetabular impingement (FAI) and its surgical management compared to AAOS OrthoInfo content. We hypothesize that ChatGPT may deliver medically accurate information, but its readability may fall short when compared to OrthoInfo. Nine questions based on the OrthoInfo FAI page were submitted to ChatGPT-4, with and without a readability prompt. Topics included anatomy, pathology, cause, symptoms, workup, imaging, treatment, role of hip arthroscopy, and outcomes. Readability was assessed using validated indices. Accuracy was independently rated using a 4-point scale. Statistical comparisons were made using t-tests and ANOVA (P < 0.01 threshold). OrthoInfo content had a mean reading grade level of 8.0 and a Flesch Reading Ease score of 60.7. Unprompted ChatGPT responses were significantly less readable (grade level 16.2; Flesch score 21.7; P < 0.001). Prompting for readability improved ChatGPT outputs (grade level 10.6; Flesch score 58.2), making them comparable to OrthoInfo (P < 0.09). Accuracy was high across all sources, though OrthoInfo's response on FAI causes scored slightly lower (3.5 versus 4.0; P < 0.46). ChatGPT provides accurate information on FAI. However, without prompting, its complexity may hinder patient understanding. Prompting for readability enables ChatGPT to match established resources like OrthoInfo. Tailored prompting is key to using LLM effectively in patient education and promoting health literacy.

Ähnliche Arbeiten