Assessing the quality of AI-generated clinical notes: validated evaluation of a large language model ambient scribe

Abstract

Background: Generative artificial intelligence (AI) tools are increasingly being used as "ambient scribes" to generate drafts for clinical notes from patient encounters. Despite rapid adoption, few studies have systematically evaluated the quality of AI-generated documentation against physician standards using validated frameworks. Objective: This study aimed to compare the quality of large language model (LLM)-generated clinical notes ("Ambient") with physician-authored reference ("Gold") notes across five clinical specialties using the Physician Documentation Quality Instrument (PDQI-9) as a validated framework to assess document quality. Methods: -tests or Mann-Whitney tests. Results: = 0.01). Despite these limitations, reviewers overall preferred ambient notes (47% vs. 39% for gold). Conclusion: LLM-generated Ambient notes demonstrated quality comparable to physician-authored notes across multiple specialties. While Ambient notes were more thorough and better organized, they were also less succinct and more prone to hallucination. The PDQI-9 provides a validated, practical framework for evaluating AI-generated clinical documentation. This quality assessment methodology can inform iterative quality optimization and support the standardization of ambient AI scribes in clinical practice.

Ähnliche Arbeiten