Evolution of AI in anatomy education study based on comparison of current large language models against historical ChatGPT performance

Abstract

The integration of Large Language Models (LLMs) in medical education has gained significant attention, particularly in their ability to handle complex medical knowledge assessments. However, a comprehensive evaluation of their performance in anatomical education remains limited. To evaluate the performance accuracy of current LLMs compared to previous versions in answering anatomical multiple-choice questions and assessing their reliability across different anatomical topics. We analyzed the performance of four LLMs (GPT-4o, Claude, Copilot, and Gemini) on 325 USMLE-style MCQs covering seven anatomical topics. Each model attempted the questions three times. Results were compared with the previous year's GPT-3.5 performance and random guessing. Statistical analysis included chi-square tests for performance differences. Current LLMs achieved an average accuracy of 76.8 ± 12.2%, significantly higher than GPT-3.5 (44.4 ± 8.5%) and random responses (19.4 ± 5.9%). GPT-4o demonstrated the highest accuracy (92.9 ± 2.5%), followed by Claude (76.7 ± 5.7%), Copilot (73.9 ± 11.9%), and Gemini (63.7 ± 6.5%). Performance varied significantly across anatomical topics, with Head & Neck (79.5%) and Abdomen (78.7%) showing the highest accuracy rates, while Upper Limb questions showed the lowest performance (72.9%). Only 29.5% of questions were answered correctly by all LLMs, and 2.5% were never answered correctly. Statistical analysis confirmed significant differences between models and across topics (χ² = 182.11-518.32, p < 0.001). Current LLMs show markedly improved performance in anatomical knowledge assessment compared to previous versions, with GPT-4o demonstrating superior accuracy and consistency. However, performance variations across anatomical topics and between models suggest the need for careful consideration in educational applications. These tools show promise as supplementary resources in medical education while highlighting the continued necessity for human expertise.

Ähnliche Arbeiten