Performance Assessment of ChatGPT for the Board Qualification Examination of the Japanese Society for Oral and Maxillofacial Radiology

Abstract

Abstract The aim of this study is to assess the performance and utility of ChatGPT for the board qualification examination of the Japanese Society for Oral and Maxillofacial Radiology (JSOMR). We assessed ChatGPT responses to 149 multiple-choice questions written in Japanese for the board qualification examination of the JSOMR for the 3 years from 2020 to 2022. The questions were directly entered into ChatGPT-3.5 and ChatGPT-4 models manually one by one as a prompt. The accuracy rate was calculated and classified by year, type of multiple-choice question, and level of intellectual ability, and significant differences were noted. The accuracy rate of GPT-3.5 for the 3 years was 45.0% (51.0% for 2020, 34.0% for 2021, and 50.0% for 2022), while the accuracy rate of GPT-4 was 68.5% (73.5% for 2020, 62.0% for 2021, and 70.0% for 2022) for the board qualification examination of the JSOMR. GPT-4 had a significantly higher accuracy rate than GPT-3.5 in each year. On performance classified by the type of multiple-choice questions, GPT-4 performed significantly better than GPT-3.5. However, neither model performed well with questions that required interpretation or knowledge of Japanese law. The performance of GPT-4 was significantly superior to GPT-3.5 in the board qualification examination of the JSOMR, suggesting that the use of Chat GPT, especially ChatGPT-4, would be effective as a tool for learning and preparing for the examination.

Ähnliche Arbeiten