ChatGPT Performance in the UK Medical Licensing Assessment: How to Train the Next Generation?

Abstract

Artificial intelligence (AI) will inevitably play an ever-increasing role in our daily lives, but its role in medical education remains uncertain.1Topol E.J. High-performance medicine: the convergence of human and artificial intelligence.Nat Med. 2019; 25: 44-56Crossref PubMed Scopus (2028) Google Scholar Language models such as Chat Generative Pretrained Transformer (ChatGPT) have the potential to augment clinical education, and it is becoming increasingly clear medical examinations may have to adapt accordingly and move away from simple deductive logical reasoning.2Kung T.H. Cheatham M. Medenilla A. et al.Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models.PLOS Digit Health. 2023; 2e0000198Crossref PubMed Google Scholar The UKMLA, set to be introduced in 2024, aims to serve as a standardized examination for international medical graduates and medical students in the United Kingdom.3Medical Licensing Assessment (MLA)General Medical Council.https://www.gmc-uk.org/registration-and-licensing/join-the-register/plab/plab-1-guide/medical-licensing-assessment-mlaDate accessed: May 31, 2023Google Scholar It consists of 2 parts: an applied knowledge test (AKT) and a clinical and professional skills assessment. The AKT section comprises 200 single best answer questions divided across two 2-hour papers. We aimed to study the performance of ChatGPT in the AKT section of the UKMLA to help guide contemporary medical education leaders. From the UK Medical Schools Council website, we identified 200 sample UKMLA AKT single best answer questions.4Practice materialsMedical Schools Council.https://www.medschools.ac.uk/studying-medicine/medical-licensing-assessment/practice-materialsDate accessed: May 31, 2023Google Scholar Questions were categorized according to 1 of 2 tasks, diagnostic workup or management, and mapped to 24 areas of clinical practice.3Medical Licensing Assessment (MLA)General Medical Council.https://www.gmc-uk.org/registration-and-licensing/join-the-register/plab/plab-1-guide/medical-licensing-assessment-mlaDate accessed: May 31, 2023Google Scholar Items with images were excluded. With the current public version of ChatGPT, we provided the following prompts: “Please answer the following questions. Each question has five answer options,” followed by each question stem. If ChatGPT could not commit to one answer, a further prompt of “please select one answer” was used. The passing mark was estimated from studies examining the retiring examination, the Professional and Linguistic Assessments Board (PLAB), which the General Medical Council states is very similar to the UKMLA blueprint.3Medical Licensing Assessment (MLA)General Medical Council.https://www.gmc-uk.org/registration-and-licensing/join-the-register/plab/plab-1-guide/medical-licensing-assessment-mlaDate accessed: May 31, 2023Google Scholar,5McManus I.C. Wakeford R. PLAB and UK graduates’ performance on MRCP(UK) and MRCGP examinations: data linkage study.BMJ. 2014; 348: g2621Crossref PubMed Scopus (0) Google Scholar From 191 sample questions (9 excluded), ChatGPT identified the correct option in 73.3% (140/191) of cases. ChatGPT demonstrated better performance in diagnostic workup compared with management at 78.9% (90/114) and 64.9% (50/77), respectively. In most of areas of clinical practice, ChatGPT’s accuracy met or exceeded the passing range (Figure). It achieved the 100% accuracy in Emergency Medicine, Palliative Care, and Otolaryngology but had only 40% accuracy in Ophthalmology (Figure). ChatGPT performed well on the UKMLA AKT and likely approaches or falls within the passing range across a range of specialties, which mirrors its initial performance on the USMLE.2Kung T.H. Cheatham M. Medenilla A. et al.Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models.PLOS Digit Health. 2023; 2e0000198Crossref PubMed Google Scholar Although the pass mark is not publicly known, a study reported a pass mark ranging from 58.9% to 68.5% in the PLAB, the predecessor to the UKMLA.5McManus I.C. Wakeford R. PLAB and UK graduates’ performance on MRCP(UK) and MRCGP examinations: data linkage study.BMJ. 2014; 348: g2621Crossref PubMed Scopus (0) Google Scholar Although we could not assess ChatGPT performance in the clinical and professional skills assessment, these results highlight the forthcoming impact of AI and the need to train future physicians. These findings provide a unique insight for education leaders. Our study revealed a publicly available natural language AI platform performs well at simple deductive logical reasoning underpinned by its vast encyclopedic recall knowledge. However, more nuanced clinical decisions requiring input of patient preferences in the context of applied clinical skills is an area where AI’s current role remains limited, and hence, a greater emphasis in this domain should be placed upon in contemporary medical education. The authors report no competing interests.

Ähnliche Arbeiten