SHAPE-AI: Development and Expert Validation of a Survey for Human–AI Performance Evaluation in Healthcare

Abstract

Abstract Objective To develop and content-validate a brief, expert-informed S urvey for H uman– A I P erformance E valuation (SHAPE-AI) for near-real-time assessment of how clinical AI affects human performance. Background AI-enabled clinical decision support can improve outcomes only when aligned with clinician workflows, and cognitive demands. Existing evaluations measure technical performance and adoption, providing limited assessment of how AI shapes human performance. There is a lack of concise, operationally feasible instruments to measure AI impact on these human factors outcomes in clinical settings. Method We used a construct-driven, multi-stage development process. A literature review and prior qualitative work with users of a deployed sepsis prediction tool identified core human performance constructs. Preliminary items were created and iteratively refined through two expert panels. Six clinical informatics experts evaluated representativeness and clarity using content validity indices (CVI). Seven human factors experts then refined constructs, item wording, and response formats through ratings and focus groups, emphasizing discriminant validity, cognitive bias mitigation, and feasibility for deployment within 24 hours of AI use. Results A concise 10-item instrument was created, comprising perceived impact, interpretability, agreement with the AI’s findings, agreement with the AI’s recommendations, trust, workload, provider–patient and provider–team relationships, unexpected outcomes. Conclusion The SHAPE-AI instrument is a theoretically grounded, operationally feasible tool to monitor human performance as relates to AI use. Application Health care organizations can deploy SHAPE-AI as a rapid, standardized probe to detect workflow misalignment, mis-calibrated reliance, communication disruptions, and unintended consequences of AI, informing safer design, implementation, and optimization of clinical AI tools. Précis SHAPE-AI is a brief, expert-validated survey designed to capture clinicians’ near-real-time perceptions of how AI-enabled decision support affects human performance such as their situational awareness, decision-making, workload, and trust. SHAPE-AI offers health systems a practical, standardized way to monitor and understand the impact of AI on human performance.

Ähnliche Arbeiten