Improving Radiology Report Error Detection Using a Multi-Pass LLM Framework (Preprint)

Abstract

<sec> <title>BACKGROUND</title> Large language model (LLM) proofreaders for radiology reports generate many false positives (FP) due to the low prevalence of errors. </sec> <sec> <title>OBJECTIVE</title> This study aimed to determine whether an optimized LLM framework could improve both precision and cost-efficiency without compromising error detection capability. </sec> <sec> <title>METHODS</title> In this retrospective study, 1,000 radiology reports (radiography, ultrasonography, CT, and MRI; 250 each) were sampled from the Medical Information Mart for Intensive Care III (MIMIC-III) database. Two public chest radiography corpora (CheXpert and Open-i) served as external test sets. Three LLM frameworks were evaluated: single-prompt detector (Framework 1); report extractor plus single-prompt detector (Framework 2); and extractor, detector, and false positive verifier (Framework 3). Precision for each framework was assessed using positive predictive value (PPV) and detected errors per 1,000 reports (DE/1k). Overall efficiency was estimated using model inference computational costs. </sec> <sec> <title>RESULTS</title> PPV increased from 0.063 [95% CI, 0.036–0.101] in Framework 1 to 0.079 (0.049–0.118) in Framework 2 and 0.159 (0.090–0.252) in Framework 3 (P&lt;.001). Despite improved PPV, detected errors remained stable (DE/1k: 12–14). Human review burden decreased from 192 to 88 reports. Framework 3 also reduced costs to $5.58 per 1,000 reports (vs $9.72 and $6.85 for Frameworks 1 and 2; 42.6% and 18.5% reductions). External validation confirmed similar improvements. </sec> <sec> <title>CONCLUSIONS</title> A three-pass LLM framework more than doubled precision and halved the cost of radiology report error detection without compromising error detection capability, offering sustainable strategies for AI-assisted quality assurance in both radiological practice and research. </sec>

Ähnliche Arbeiten