Which curriculum components do medical students find most helpful for evaluating AI outputs?

Abstract

<title>Abstract</title> <bold>Introduction</bold> The risk and opportunity of Large Language Models (LLMs) in medical education both rest in their imitation of human communication. Future doctors working with generative artificial intelligence need to judge the value of any outputs from LLMs to safely direct the management of patients. We set out to evaluate our students’ ability to validate LLM responses to clinical vignettes, identify which prior learning they utilised to scrutinise the LLM answers, and whether they were aware of ‘clinical prompt engineering’. <bold>Methods</bold> A content analysis cohort study was conducted amongst 148 consenting final year medical students at Imperial College London. A survey asked students to evaluate answers provided by GPT 3.5 in response to ten clinical scenarios, five of which GPT 3.5 had answered incorrectly, and to identify which prior training enabled them to determine the accuracy of the GPT 3.5 output. <bold>Results</bold> The overall median student score in correctly judging the answers given by GPT 3.5 was 61%, with 65% demonstrating sound clinical reasoning for their decision. Students reported interactive case-based discussions and pathology teaching to be the most helpful for AI output evaluation. Only 5% were aware of ‘clinical prompt engineering’. <bold>Conclusion</bold> Artificial intelligence is a sociotechnical reality, and we need to validate the new pedagogical requirements for the next generation of doctors. Our data suggest that critical analysis taught by pathology clinical case teaching is currently the self-reported best training for medical students to evaluate the outputs of LLMs. This is significant for informing the design of medical training for future doctors graduating into AI-enhanced health services.

Ähnliche Arbeiten