Evaluation of Reliability, Repeatability and Confidence of ChatGPT With Regard to Interstitial Lung Disease in Patients With Systemic Autoimmune Rheumatic Diseases

Abstract

Abstract Rationale: ChatGPT, as a newly released Large Language Model (LLM), has many applications in medical practice, but whether it can manage patients with Systemic Autoimmune Rheumatic Diseases associated Interstitial Lung Disease based on guideline is uncertain. Methods: 103 questions based on 2023 American College of Rheumatology (ACR)/American College of Chest Physicians (CHEST) Guideline for Interstitial Lung Disease in People with Systemic Autoimmune Rheumatic Diseases were used to benchmark ChatGPT (GPT-4, GPT-4o mini and GPT-4o) on three separate attempts. Accuracy and answer choices among attempts were compared to assess reliability (accuracy over time) and repeatability (agreement over time). In addition, ChatGPT was prompted to rate its confidence from 1-10 (with 10 being the highest level of confidence and 1 being the lowest). Results: Neither version showed a difference in accuracy over three attempts: for the first, second, and third attempt, accuracy of GPT-4 was 86.4% (89 of 103), 75.7% (78 of 103), and 79.6% (82 of 103), respectively (P =.146), GPT-4o mini was 77.7% (80 of 103), 70.9% (73 of 103), and 74.8% (77 of 103), respectively (P =.533), and GPT-4o was 82.5% (85 of 103), 83.9% (86 of 103), and 78.6% (81 of 103), respectively (P =.637). GPT-4 and GPT-4omini have moderate interrater agreement (Kendall-W coefficient of concordance (KW)= 0.747 and 0.765, respectively), while GPT-4o have strong interrater agreement (KW = 0.816). All three versions showed statistical difference in rated “high confidence” (≥8 on the 1-10 scale) for three attempts: GPT-4o, 63.1% (65 of 103), 86.4% (89 of 103), and 100.0% (103 of 103), respectively (P <0.01), GPT-4o mini 84.5% (87 of 103), 89.3% (92 of 103), and 42.7% (44 of 103), respectively (P <0.01), GPT-4,67.0% (69 of 103), 35.9% (37 of 103), and 58.3% (60 of 103), respectively (P <0.01). Conclusions: All GPT-4, GPT-4o mini and GPT-4o are reliably accurate across three attempts, with GPT-4o being more repeatable than GPT-4 and GPT-4o mini, but their confidence is frequently unstable.

Ähnliche Arbeiten