AI-generated feedback in social robotic virtual patients and medical student performance: A nonrandomized clinical trial (Preprint)

Abstract

<sec> <title>BACKGROUND</title> Virtual patients (VPs) demonstrate effectiveness in improving clinical reasoning (CR) skills, yet traditional VP platforms often lack individualised feedback mechanisms. Effective feedback represents a cornerstone of medical education, particularly in developing CR skills. Advances in large language models (LLMs) enable automated analysis of student-VP interactions, providing scalable feedback on clinical performance. While AI-enhanced social robotic VPs show promise for CR training, no studies have examined whether integrated AI-generated feedback improves clinical performance objectively. Determining whether AI-generated feedback translates into improved performance in clinical examinations could provide essential evidence for the educational value of these emerging technologies. </sec> <sec> <title>OBJECTIVE</title> To evaluate whether the integration of AI-generated post-consultation feedback into social robotic VP interactions improves medical students’ clinical performance. </sec> <sec> <title>METHODS</title> A quasi-experimental study with 115 sixth-semester medical students (73.2% of eligible students) was conducted at Karolinska Institutet, Stockholm, Sweden, during spring 2025. Sixth semester medical students were allocated to either receive (n=61) or not receive (n=54) AI-generated feedback following interactions with a Social AI-enhanced Robotic Interface (SARI). All students completed nine VP cases; students in the intervention group received approximately one page of structured written feedback after each VP case using SARI. The AI feedback system employed multiple LLMs and followed a two-stage algorithm: first assessing student-VP dialogues using an assessment rubric, then generating structured feedback on medical history-taking performance. Students in both groups participated in case-specific follow-up seminars led by consultant rheumatologists following each VP encounter. Clinical performance was assessed through an eight-minute OSCE-like evaluation with a standardised patient portraying axial spondylarthritis, evaluated by a consultant rheumatologist blinded to group allocation using a 10-point rubric across five domains: communication at consultation start, generic medical history, targeted medical history, diagnostics and management reasoning, and communication at consultation end. </sec> <sec> <title>RESULTS</title> Students receiving AI-generated feedback achieved significantly higher total OSCE scores (7.39±0.86 versus 6.68±1.04 points; mean difference: 0.70; 95% CI: 0.35–1.06; P&lt;.001; Cohen's d=0.74). Domain-specific analysis revealed significant improvement in generic medical history after Bonferroni correction (2.46±0.65 versus 2.03±0.79 points; P=.004; r=0.27), while other domains showed no significant differences: communication at start (P=.134; r=0.14), targeted medical history (P=.605; r=0.05), diagnostics and management (P=.149; r=0.14), and communication at consultation end (P=.312; r=0.09). Pass rates were significantly higher in the feedback group (96.7% versus 79.6%; OR: 7.55; 95% CI: 1.51–72.2; P=.006), with a number needed to treat of six students. </sec> <sec> <title>CONCLUSIONS</title> AI-generated feedback following social robotic VP interactions significantly improved medical students' clinical performance in standardised examination, particularly in generic medical history-taking. These findings support integrating validated AI feedback systems in VP platforms for clinical skill training and demonstrate the feasibility of scalable, automated feedback for medical education. The domain-specific improvement in generic medical history components highlights the importance of targeted, competency-specific feedback design in VP education. </sec> <sec> <title>CLINICALTRIAL</title> NCT07277829 </sec>

Ähnliche Arbeiten