Evaluation and comparison of the quality and readability of online information for patients using the Chat GPT (artificial intelligence) platform versus other online sources regarding hip fractures management

Abstract

Objectives The purpose of this study was to use standardised scoring systems in the analysis and comparison of the quality and readability of information regarding hip fractures and their management when querying an AI software (ChatGPT). Methods An open AI model (ChatGPT) was used to answer 25 commonly asked questions from patients regarding hip fractures as described by three international societies. Subsequent evaluation and comparison was carried out by two reviewers for medical accuracy, quality and readability using the JAMA Benchmark criteria, DISCERN score, Flesch-Kincaid Reading Ease Score (FRES) & Grade Level (FKGL). Results For the ChatGPT responses, the JAMA Benchmark criteria score was 0, which is the lowest score indicating no reliable resources cited. The DISCERN score was 60, which is considered a good score. The FRES was 27.7 (a ‘very confusing’ score), and the FKGL was considered to be that of a ‘college graduate’. In contrast, the other sources analysed had a mean FRES of 70.3 (a ‘fairly easy’ score), with the FKGL considered to be that of ‘middle school student’. Conclusion Using ChatGPT resulted in very high quality answers given on questions relating to hip fractures and their management, although the authors noted that a much higher reading level was required to comprehend the generated information on the topic when compared to other credible sources. Thus, while the information was readily accessible, it was not easily understandable. Furthermore, the lack of citations generated is an ongoing source of concern regarding the reliability of responses. Whilst ChatGPT offers an accessible information resource for patients and their families, this study highlights the need for vetted, understandable and reliable literature to be made available by caregivers to inform our patients. Level of Evidence V

Ähnliche Arbeiten