Artificial Intelligence Chatbots Taking American Board of Endodontics Simulated Oral Board Examination

Abstract

Introduction The aim of this study was to assess the overall performance of artificial intelligence (AI) chatbots in taking the American Board of Endodontics (ABE) simulated Oral Board Examination. Methods Three oral board cases were constructed by two academic board-certified endodontists. Each case included a comprehensive patient profile consisting of medical history, dental history, and results of clinical testing, followed by 20 consecutive open-ended oral board-style questions. Two publicly accessible AI chatbots were selected to take the exam: GPT-4o and Gemini-2.5 Pro. Responses were scored based on a comprehensive rubric on a four-point ordinal scale (0-3) by the same two examiners independently: response validity, citation validity, and overall performance score. A Cumulative Link Mixed Model (CLMM; proportional odds) was used with fixed effects for chatbot and case, and random intercepts for reviewer and question to analyze and compare the performance of models, i.e., inter- and intra-chatbot comparisons. Results Gemini-2.5 Pro and GPT-4o achieved high mean overall performance scores of 2.83 (±0.42) and 2.73 (±0.51), respectively. CLMM showed no significant difference between the two chatbots in probability of receiving an excellent score (i.e., 3) in response validity (odds ratio [OR]=2.44, 95% confidence interval (CI) [0.98–6.06], p=0.054) or in overall performance (OR=2.04, 95 % CI 0.97–4.30, p=.061). There was a positive correlation between response validity and citation validity for GPT-4o (ρ=0.21, p=0.019). Conclusions Both chatbots scored considerably high in the simulated ABE Oral Board Examination. Results of this study support the concept of using AI chatbots as aid in endodontic education.

Ähnliche Arbeiten