Comparative evaluation of ChatGPT-4, Claude 3.5 Sonnet, and Gemini 1.5 Advanced for patient education on chronic obstructive pulmonary disease (COPD): a global expert assessment of artificial intelligence (AI)-generated responses

Abstract

<bold>Background:</bold> AI models like ChatGPT-4, Claude 3.5 Sonnet, and Gemini 1.5 Advanced are increasingly used to generate information, but their effectiveness in delivering accurate and reliable health content related to COPD remains insufficiently explored. <bold>Objective:</bold> To evaluate and compare the response quality of AI-generated answers to frequently asked questions about COPD. <bold>Methods:</bold> 30 COPD-related questions, based on the 2024 Global Initiative for Chronic Obstructive Lung Disease strategy document, were input into three AI platforms in September 2024. The 90 responses were evaluated by expert pulmonologists from six continents, blinded to the AI platform, and assessed on a Likert scale (1–5) across five criteria: completeness, accuracy, terminology, accessibility, and safety. Group differences were assessed using the Kruskal–Wallis test, followed by Dunn’s test with Bonferroni correction for multiple comparisons. <bold>Results:</bold> 61 experienced pulmonologists assessed the survey. Statistical analysis showed that Gemini outperformed the others in response completeness (p < 0.01–0.03), while Claude achieved higher accuracy in information delivery and medical terminology (p = 0.002–0.05). No differences were found for accessibility or safety (all p > 0.05). <bold>Conclusions:</bold> All three AI platforms provided potentially useful information, though performance varied. Caution is advised when using them as COPD guides for patients and families. While AI has the potential to support global respiratory health education, further research is needed to ensure accuracy and validation.

Ähnliche Arbeiten