The Use of Large Language Models Tuned with Socratic Methods on the Impact of Medical Students' Learning: A Randomised Controlled Trial (Preprint)

Abstract

<sec> <title>BACKGROUND</title> Large Language Models (LLM) are AI models that can generate conversational content based on a trained specified source of information (corpus). </sec> <sec> <title>OBJECTIVE</title> The aim is to use these corpus-trained LLMs to limit the content offered by LLM, then using prompt engineering to teach using Socratic methods. </sec> <sec> <title>METHODS</title> Two chatbots were created and deployed, powered by OpenAI’s GPT-3.5 model, with a medical-school textbook corpus. The first chatbot generates a brief summary and open-ended question. The second chatbot generates a case vignette from its pre-trained clinical cases, prompting users for a diagnosis. Both chatbots reply to the user’s response, commenting on the accuracy and asks further questions to encourage critical thinking. A randomised controlled trial was conducted on two groups comprising third year medical students. One group used both chatbots for 10 minutes while the other read the medical textbook. A 15-question test was administered to both groups before and after the intervention. </sec> <sec> <title>RESULTS</title> Forty students participated in the study. The average of the group before and after reading the textbook (n=20) are 3.9 +/- 1.0 and 7.6 +/- 1.5 respectively (p&lt;0.001). The average of the group before and after using the bot (n=20) are 3.9 +/- 0.9 and 12.8 +/- 1.6 respectively (p&lt;0.001). The respective increase in results was 3.7 and 8.9. </sec> <sec> <title>CONCLUSIONS</title> Medical students’ learning showed a better performance using a LLM based chatbot compared to self-reading of medical information assessed using a standardised test. More studies are required to determine if LLM-based pedagogical methods are superior to standard education. </sec>

Ähnliche Arbeiten