Reliability of Gemini 2.5 Pro, ChatGPT 4.1, DeepSeek V3, and Claude Opus 4 in generating standardized CMR protocols

Abstract

Artificial intelligence (AI) and large language models (LLMs) are increasingly integrated into radiology, offering new possibilities for advanced imaging techniques, including cardiovascular magnetic resonance (CMR). This proof-of-concept study assessed four high-performing LLMs (Gemini 2.5 Pro, ChatGPT 4.1, DeepSeek V3, and Claude Opus 4) on their ability to generate CMR protocols for 140 hypothetical cardiac cases. AI-generated protocols were compared against a reference standard established by a consensus between two experienced cardiovascular radiologists, following the Society for Cardiovascular Magnetic Resonance (SCMR) recommendations. Descriptive statistics were used to quantify the concordance of LLM-generated sequences with the SCMR guidelines. Statistical agreement was measured using Cohen and Fleiss κ statistics. Gemini 2.5 Pro achieved the highest concordance, aligning with the SCMR guidelines in 71.5% of all evaluated scenarios. Overall, LLMs showed moderate agreement with the SCMR protocols, with Gemini 2.5 Pro again performing best (Cohen κ = 0.55). Agreement was substantial for mandatory CMR sequences (Fleiss κ ∈ [0.69, 0.74]) and predominantly fair for optional sequences. The tested LLMs demonstrate a potential to generate efficient and pathology-adapted CMR protocols. Under expert supervision, this capability could streamline the imaging workflow and help extend CMR to primary healthcare centers through protocol automation. RELEVANCE STATEMENT: The potential of Gemini 2.5 Pro, ChatGPT 4.1, DeepSeek V3, and Claude Opus 4 to suggest pathology-adapted CMR protocols could improve imaging throughput and help to expand access to advanced cardiac diagnostics in primary healthcare centers. KEY POINTS: The tested large language models show potential for generating CMR protocols. Substantial agreement on mandatory CMR sequences promises more efficient examinations. Automation of CMR protocols could help to improve access to this advanced technique outside major medical institutions.

Ähnliche Arbeiten