CLEVER: Clinical Large Language Model Evaluation by Expert Review (Preprint)

Abstract

<sec> <title>BACKGROUND</title> The proliferation of both general-purpose and healthcare-specific Large Language Models (LLMs) has intensified the challenge of effectively evaluating and comparing them. Data contamination plagues the validity of public benchmarks; self-preference distorts LLM-as-a-judge approaches; and there’s a gap between the tasks used to test models and those used in clinical practice. </sec> <sec> <title>OBJECTIVE</title> In response, we propose CLEVER: A methodology for blind, randomized, preference-based evaluation by practicing medical doctors on specific tasks. </sec> <sec> <title>METHODS</title> We demonstrate the methodology by comparing GPT-4o against two healthcare-specific LLMs, with 8B and 70B parameters, over three tasks: clinical text summarization, clinical information extraction, and question answering on biomedical research. </sec> <sec> <title>RESULTS</title> Medical doctors prefer the Small Medical LLM over GPT-4o 45% to 92% more often on the dimensions of factuality, clinical relevance, and conciseness. </sec> <sec> <title>CONCLUSIONS</title> The models show comparable performance on open-ended medical Q&amp;A, suggesting that healthcare-specific LLMs can outperform much larger general-purpose LLMs in tasks that require understanding of clinical context. We test the validity of CLEVER evaluations by conducting inter-annotator agreement, inter-class correlation, and washout period analysis. </sec> <sec> <title>CLINICALTRIAL</title> n/a </sec>

Ähnliche Arbeiten