Response to ‘Performance of large language models at the MRCS Part A: a tool for medical education?’

Abstract

This follows the May 2025 issue of the Bulletin, which was dedicated to artificial intelligence (AI) technologies in surgery.The topic is of clear ongoing interest for the college, 2,3 and we would like to offer our reflections.The authors have rightly identified a central question facing the surgical education community: can emerging AI technologies, particularly large language models (LLMs), be integrated reliably as educational tools?The results are of course impressive, with ChatGPT-4 achieving a score sufficient to pass the MRCS Part A examination.However, this achievement is tempered by the finding that these models produced inaccurate but authoritative answers for 15 to 36 percent of questions, depending on the platform assessed.Although there are no data available on the performance of the latest LLM models on the MRCS Part A, it is reasonable to anticipate some degree of error.Even if the error rate were reduced to around 5 percent, this would still result in a significant number of incorrect responses given the large volume of questions typically attempted in preparation for this examination.In practical terms, a candidate using these tools as a learning resource would be exposed to incorrect information at a rate that cannot be overlooked.This is particularly concerning given the confident tone with which LLMs tend to deliver their responses.To echo the authors' question: a tool for medical education?On balance, perhaps we are not quite there yet.This article also highlights a subtle but important difference between AI-supported clinicians and clinicians who have been educated with the help of AI-enabled tools.These tools can help make the large and challenging volume of material an aspiring surgeon must learn more engaging and memorable.LLMs could help identify areas of weakness, put together bespoke curricula to address gaps, and individualise learning.

Ähnliche Arbeiten