Impact of an Ambient AI Scribe on Medical Student Observed Structured Clinical Examination Notes: A Nonrandomized Clinical Trial (Preprint)

Abstract

BACKGROUND: Ambient artificial intelligence (AI) scribes for chart documentation have seen rapid adoption in clinical practice, but their educational impact on medical students has not been described. OBJECTIVE: The purpose of this study is to determine the impact of an AI scribe on pre-clerkship medical student note writing. METHODS: In this prospective non-randomized pre/post design study, all first-year medical students (n=104) at a single U.S. medical school submitted "human-only" notes based on a summative observed structured clinical examination (OSCE) station in May 2025. An AI scribe generated independent AI notes post-OSCE from recorded audio. A sub-group of students (n=47) consented to complete a second "hybrid" note by revisiting their human-only note and incorporating AI notes as perceived necessary, followed by a brief survey about the AI notes. Trained, blinded fourth-year medical student raters were randomly assigned to score all notes on 10 elements using QNOTE acceptability criteria (0=Unacceptable, 50=Partially, 100=Fully). A post-hoc, exploratory element-level review was then conducted. RESULTS: Across all elements, median evaluation scores of human-only notes were high (range 81.3 - 100) and were similar between students who participated in "hybrid" notes and those who did not. In paired analyses between "human-only" and "hybrid" notes, the only notable element-level change was a decline in Chief Complaint scores (P=.05). Symptom duration was omitted in the Chief Complaint section in 8 of 47 (17%) AI notes. No score differences were observed in QNOTE elements requiring documentation of pertinent findings and prioritized lists. Participants agreed that the AI note "was more concise than my note" (37/47, 79%) and would be "helpful as a first draft" (31/47, 66%); 26 out of 47 (55%) agreed that the AI note "left out important details", and 10 out of 47 (21%) agreed that the AI note "may reduce my ability to learn how to write a good note." CONCLUSIONS: Interaction with AI notes among pre-clerkship medical students had little impact on quality of "hybrid" notes. Chief Complaint scores likely declined due to conciseness in AI notes that often omitted symptom duration. Our findings suggest that among students who predominantly write close to fully acceptable "human-only" notes, there was no detriment to clinical reasoning, and students were discerning in balancing AI's conciseness and its omissions. The lack of impact on note quality may have been due to the workflow employed in this study, in which students were required to generate independent judgments before exposure to AI-generated content. Future work must explore longitudinal use of such tools using standard workflows seen in clinical settings, where AI notes serve as true first drafts. Especially for lower-performing students, AI scribes could enhance students' own note writing, though educational safeguards are necessary given the potential for harm due to overreliance on automated systems. CLINICALTRIAL: The study was granted approval by the Yale School of Medicine Committee to Review Student Participation in Research on January 14, 2025, and an exemption from full review due to the minimal-risk educational nature of the project by the Yale Human Research Protection Program on January 23, 2025 (IRB Protocol ID 2000039478).

Ähnliche Arbeiten