Real-World Evaluation of Artificial Intelligence-Assisted Triage for Same-Day Appointments: A Mixed-Methods Study in UK Primary Care (Preprint)

Abstract

<sec> <title>BACKGROUND</title> Artificial intelligence (AI) is increasingly used to support clinical decision-making, particularly in primary care triage. However, few studies have assessed clinical concordance between AI triage tools and general practitioner (GP) assessments in real-world settings. </sec> <sec> <title>OBJECTIVE</title> This study evaluated the agreement between AI-enabled triage urgency ratings (Visiba Triage) and GP urgency assessments for same-day appointment (SDA) requests in the NHS in England. Secondary aims included assessing perceptions of safety, accuracy and usability from both clinician and patient perspectives. </sec> <sec> <title>METHODS</title> A pre-study established the extent of automation bias when GPs had prior sight of the AI urgency rating before making their own urgency assessment. The main trial was a mixed-methods study conducted using live data from patients requesting SDA between January and June 2024. A total of 649 participants were included in this study after completing an AI-enabled triage protocol upon initial presentation to the primary care facility including a brief satisfaction survey. For each case, a GP also made a separate urgency assessment based on the medical information available. The concordance of the urgency ratings generated by the AI tool with GP assessment was analysed using Cohen’s kappa, correlation, and the confusion matrix. Ordinal logistic regression assessed associations between patient satisfaction scores and patient demographics. Thematic analysis of semi-structured interviews with eight GPs explored perceptions of the AI tool’s performance in terms of utility and safety. </sec> <sec> <title>RESULTS</title> There was no difference in the distribution of urgency ratings for GP assessments whether GPs were aware of the AI recommendation a priori or not (Wasserstein distance P-value: .81 ± .0005), so risk of automation bias was considered low. For the main trial, there was 83.7% categorical agreement across three urgency levels (κ 0.69, P 0.001), with strong correlation between AI and GP urgency ratings on an 8-point scale (ρ=0.796, P 0.001). The AI system demonstrated safety-conscious design, with a greater likelihood of overtriage. No cases deemed non-urgent by AI were reclassified as emergencies by GPs. Patient satisfaction scores for using the AI triage tool varied significantly by age, with older adults (60+) reporting lower satisfaction (aOR 0.25, 95% CI 0.12-0.52). Thematic analysis of interviews with GPs indicated overall satisfaction with the perceived accuracy and safety of the AI system and its usefulness in supporting their triage process. </sec> <sec> <title>CONCLUSIONS</title> This study demonstrates a high rate of GP agreement with the urgency ratings output by an AI-enabled triage tool, offering the potential for a safe, scalable solution to manage demand for same-day care in primary care settings. Current limiters for the AI tool include suboptimal integration with full patient medical records. Safe adoption of AI triage tools in healthcare should include real-world assessment of performance and comparison with consensus clinician judgement in real-time. </sec>

Ähnliche Arbeiten