Large language model diagnostic assistance for physicians in a lower-middle-income country: a randomized controlled trial

Abstract

Diagnostic errors remain a major source of preventable patient harm, particularly in low- and middle-income countries. Large language models (LLMs) have the potential to bridge these gaps in diagnostic capacity, but can generate inaccurate information, necessitating artificial intelligence (AI)-literacy training. However, whether AI-trained physicians can leverage LLMs to improve diagnostic reasoning compared with conventional resources is unknown. Here we conducted a single-blind randomized controlled trial involving 60 licensed physicians from multiple medical institutions in Pakistan between January and May 2025. Participants completed a 20-hour AI-literacy curriculum covering LLM capabilities, appropriate use and limitations. Post-training, physicians were randomized to either LLM access plus conventional resources or conventional resources only, with 75 minutes to review up to 6 clinical vignettes. The primary outcome was diagnostic reasoning score using an expert-validated grading rubric; the secondary outcome was time per vignette. Of 58 physicians completing the study, those with LLM access achieved a mean diagnostic reasoning score of 71.4% versus 42.6% with conventional resources alone, an adjusted difference of 27.5 percentage points (95% confidence interval (CI), 22.8 to 32.2; P < 0.001). Time per case was similar (mean difference −6.4 s; 95% CI, −68.2 to 55.3; P = 0.84). While a secondary exploratory analysis showed that LLM alone outperformed LLM-assisted physicians (11.5 percentage points; 95% CI, 5.5 to 17.5; P < 0.001), in 31.4% of cases, the physician-plus-LLM group exceeded the median LLM-alone performance, indicating complementarities. Among AI-trained physicians, access to an LLM substantially improved diagnostic reasoning without slowing case review, suggesting that effective LLM utilization could help address diagnostic gaps in resource-limited settings. ClinicalTrials.gov registration: NCT06774612 . In a randomized controlled study involving 58 physicians in Pakistan, assistance by a large language model in diagnostic reasoning resulted in a 27.5% increase in performance on 6 clinical vignettes.

Ähnliche Arbeiten