Using chatbots to develop multiple-choice questions. We got evidence, but we ain't there yet!

Abstract

Abstract Developing accessible assessment tools is crucial for educators. Traditional methods demand significant resources such as time and expertise. Therefore, an accessible, user-friendly approach is needed. Traditional assessment creation faces challenges, however, new solutions like automatic item generation have emerged. Despite their potential, they still require expert knowledge. ChatGPT and similar chatbots offer a novel approach in this field. Our study evaluates the validity of MCQs generated by chatbots under the Kane validity framework. We focused on the top ten topics in Infectious and Tropical diseases, chosen based on epidemiological data and expert evaluations. These topics were transformed into learning objectives for chatbots like GPT-4, BingAI, and Claude to generate MCQs. Each chatbot produced 10 MCQs, which were subsequently refined. We compared 30 chatbot-generated MCQs with 10 from a Peruvian medical examination. The participants included 48 medical students and doctors from Peru. Our analysis revealed that the quality of chatbot-generated MCQs is consistent with those created by humans. This was evident in scoring inferences, with no significant differences in difficulty and discrimination indexes. In conclusion, chatbots appear to be a viable tool for creating MCQs in the field of infectious and tropical diseases in Peru. Although our study confirms their validity, further research is necessary to optimize their use in educational assessments.

Ähnliche Arbeiten