Assessing the Quality and Accuracy of ChatGPT-3.5 Responses to Patient Questions About Hip Arthroscopy

Abstract

<i>Background and Objectives</i>: Artificial intelligence-driven large language models such as ChatGPT increasingly influence patient education in surgical fields. This study evaluates the quality and accuracy of ChatGPT-3.5-generated responses to patient questions regarding femoroacetabular impingement syndrome (FAIS) and hip arthroscopy (HAS). <i>Materials and Methods</i>: In this descriptive observational study, ChatGPT-3.5 generated and answered 20 representative patient questions about FAIS and HAS (n = 20 question-answer pairs). No patient-derived questions or data were used. Each response was independently evaluated by two fellowship-trained orthopedic surgeons across four domains: relevance, accuracy, clarity, and completeness, using a five-point Likert scale. Inter-rater reliability was calculated using the intraclass correlation coefficient (ICC), and descriptive inter-rater agreement percentages were reported. Additional qualitative impressions from the reviewers were recorded to contextualize areas in which responses were rated slightly lower, particularly regarding explanatory depth and postoperative variability. <i>Results</i>: Mean ratings across all domains ranged from 4.85 ± 0.24 (95% CI: 4.74-4.96) to 5.00 ± 0.00. Relevance achieved a perfect mean score of 5.00, while accuracy and clarity each obtained 4.98 ± 0.11 (95% CI: 4.91-5.00). Completeness demonstrated the lowest scores (4.85 ± 0.24). Due to pronounced ceiling effects, ICC values were non-informative; however, descriptive agreement between raters was high, with 100% concordance for relevance and 90% agreement for accuracy and clarity. No factually incorrect or unsafe information was identified. Overall, responses were concise, structured, and clinically appropriate, though occasionally lacking in granularity concerning individual recovery trajectories and procedure-specific nuances. <i>Conclusions</i>: ChatGPT-3.5 demonstrates significant potential as a supplementary patient education tool in hip preservation surgery. While its responses were consistently accurate and easy to understand, their occasional lack of detail, particularly concerning postoperative expectations and variability in outcomes, indicates that the findings apply primarily to synthetic, standardized questions in a controlled setting. Further validation is required before generalizing these results to real-world patient interactions. Future studies should incorporate authentic patient questions, diverse evaluator groups, and longitudinal assessment across different LLM versions to better define clinical applicability and safety.

Ähnliche Arbeiten