Evaluating the accuracy and reliability of AI chatbots in patient education on cardiovascular imaging: a comparative study of ChatGPT, gemini, and copilot

Abstract

Abstract Background The integration of artificial intelligence (AI) chatbots in medicine is expanding rapidly, with notable models like ChatGPT by OpenAI, Gemini by Google, and Copilot by Microsoft. These chatbots are increasingly used to provide medical information, yet their reliability in specific areas such as cardiovascular imaging remains underexplored. This study aims to evaluate the accuracy and reliability of ChatGPT (versions 3.5 and 4), Gemini, and Copilot in responding to patient inquiries about cardiovascular imaging. Methods We sourced 30 patient-oriented questions on cardiovascular imaging. The questions were submitted to ChatGPT-4, ChatGPT-3.5, Copilot Balanced Mode, Copilot Precise Mode, and Gemini. Responses were evaluated by two cardiovascular radiologists based on accuracy, clarity, completeness, neutrality, and appropriateness using a structured rubric. Inter-rater reliability was assessed using Cohen’s Kappa. Results ChatGPT-4 achieved the highest performance with 78.3% accuracy, 86.87% clarity and appropriateness, 81.7% completeness, and 100% neutrality. Gemini showed balanced performance, while Copilot Balanced Mode excelled in clarity and accuracy but lagged in completeness. Copilot Precise Mode had the lowest scores in completeness and accuracy. Penalty assessments revealed that ChatGPT-4 had the lowest incidence of missing or misleading information. Conclusion ChatGPT-4 emerged as the most reliable AI model for providing accurate, clear, and comprehensive patient information on cardiovascular imaging. While other models showed potential, they require further refinement. This study underscores the value of integrating AI chatbots into clinical practice to enhance patient education and engagement.

Ähnliche Arbeiten