Performance of Generative Artificial Intelligence Chatbots on the Pulmonological Medical Licensing Testing

Abstract

<bold>Background:</bold> Generative artificial intelligence chatbots have performed highly in various professional tests. However, it is still unclear how effective are these chatbots in respiratory medicine medical licensing testing. <bold>Objective:</bold> To evaluate the performance of generative artificial intelligence chatbots on pulmonological medical licensing testing in Ukraine <bold>Methods:</bold> We used the three most popular generative artificial intelligence (AI) chatbots in Ukraine: ChatGPT, Gemini and Microsoft Copilot to provide answers for tests from the Ukrainian pulmonological medical licensing examination. From all tests 1104 test, we excluded 9 tests which contained figures or images, thus, 1095 tests were inputted into generative AI chatbots: 1036 single choice tests, and 59 multiple choice tests. <bold>Results:</bold> ChatGPT accuracy was 95% (n=1037), Microsoft Copilot – 92% (n=1008), and Gemini – 81% (n=890). ChatGPT accuracy for single choice tests was 97% (n=1002), for Microsoft Copilot – 96% (n=884), Gemini – 85% (n=992). ChatGPT accuracy for multiple-choice tests was 59% (n=35), for Microsoft Copilot – 27% (n=16), and for Gemini – 10% (n=6). <bold>Conclusion:</bold> Overall, generative AI chatbots performed well, scoring 89.3% on the Ukrainian pulmonological medical licensing testing. ChatGPT presented the best results, demonstrating consistent performance across single-choice and multiple-choice tests.

Ähnliche Arbeiten