Promise or Peril: Evaluating the Performance of OpenEvidence in Clinical Decision Making in the Global Context. (Preprint)

Abstract

<sec> <title>BACKGROUND</title> Clinicians face the impossible task of keeping up with evidence-based literature to inform clinical decision making. OpenEvidence holds great promise by providing latest evidence to inform patient care.AI solutions are increasingly used to support physicians during clinical decision making. Clinicians need to critically analyze and evaluate outputs and be aware of biases that exist prior to application, especially when caring for patients in global contexts. </sec> <sec> <title>OBJECTIVE</title> To evaluate performance of OpenEvidence (OE) in global context. </sec> <sec> <title>METHODS</title> In this study a sample of 11 clinicians from the United States (n=3) and six other countries (n=8) performed standardized searches for 10 common clinical conditions. Search results were evaluated using the Rubric Assessment Scale and additional questions compared U.S. based results versus other countries. Data were analyzed using descriptive statistics, and Wilcoxon rank-sum test for comparisons of median differences. All statistical tests were two-tailed p-value &lt; .05 was considered statistically significant. </sec> <sec> <title>RESULTS</title> A total of 80 evaluations showed 88% accuracy and 89% efficiency with significant geographic disparities. U.S. based searches demonstrated alignment with management guidelines (p=0.02), higher perceived overall value (p=0.02) and inclusion of local experts(p&lt;0.0001). The global group reported wider, less relevant differential diagnoses (p=0.03) and demonstrated lack of regional and socio-cultural contexts. </sec> <sec> <title>CONCLUSIONS</title> OE is a valuable tool but exhibits significant geographic bias, favoring U.S. clinical contexts with limitations for global applicability. There is critical need for training AI models on diverse and representative datasets to ensure accurate performance. Clinicians should critically appraise AI outputs prior to application to model responsible use of AI. </sec> <sec> <title>CLINICALTRIAL</title> NA </sec>

Ähnliche Arbeiten