Overview of the ClinIQLink 2025 Shared Task on Medical Question-Answering

Abstract

In this paper, we present an overview of ClinIQLink a shared task, collocated with the 24th BioNLP workshop at ACL 2025, designed to stress-test large language models (LLMs) on medically-oriented question answering aimed at the level of a General Practitioner. The challenge supplies 4 978 expert-verified, medical source-grounded question-answer pairs that cover seven formats - <i>true/false</i>, <i>multiple choice</i>, <i>unordered list</i>, <i>short answer</i>, <i>short-inverse</i>, <i>multi-hop</i>, and <i>multi-hop-inverse</i>. Participating systems, bundled in Docker or Apptainer images, are executed on the CodaBench platform or the University of Maryland's <i>Zaratan</i> cluster. An automated harness (Task 1) scores closed-ended items by exact match and open-ended items with a three-tier embedding metric. A subsequent physician panel (Task 2) audits the top model responses.

Ähnliche Arbeiten