Advancing Rheumatology Practice with AI Assistance: Evaluating ChatGPT's Performance in Real-world Cases

Abstract

<title>Abstract</title> Background The launch of ChatGPT, an advanced language model created by OpenAI, has sparked attention throughout the world. ChatGPT offers human-like responses and the potential for a wide range of applications, including medical decision-making. However, questions remain regarding its validity, the spread of false information, and its ethical implications for healthcare. While other studies have investigated ChatGPT's diagnostic capabilities, this study fills a research gap by assessing how well it performs in real-world rheumatology case scenarios, offering light on its possible use in managing rheumatologic patients. Methods The study encompassed 32 challenging rheumatology cases. Data for each case was divided into four categories: 1) initial presentation, history, and review of systems; 2) physical examinations; 3) workup results; and 4) final diagnosis. Data was transformed into prompts for ChatGPT, simulating real-time interaction. Four stages of questioning were used to progressively build the case. Recommendations were evaluated based on correctness, completeness, and potential harm or cost implications. Results The percentage of comprehensive answers (totally correct, totally complete, no extra-costs, no harm) for physical examinations, workups, differential diagnosis, and treatment were 65.6%, 50%, 40,6% and 40,6% respectively. ChatGPT was successful in 65.6% of the cases to suggest the correct diagnosis first in the list of differential diagnoses. None of ChatGPT responses included suggestions that would result in unnecessary costs or harm to the patient. ChatGPT recommendations for physical examinations, workups, differential diagnosis and treatment were totally correct in 75%, 65.63%, 53.13% and 50% of instances; and they were totally complete in 81.25%, 62.50%, 62.50%, 59.38% of instances respectively. Conclusions Our study highlights the effectiveness of ChatGPT in supporting rheumatology practice by offering precise and comprehensive suggestions across various stages of patient cases. While the AI model displays potential, its performance is inconsistent when faced with higher levels of scenario complexity.

Ähnliche Arbeiten