Evaluation Methods for LLM-based Applications for Non-professional Users in Healthcare: A Scoping Review (Preprint)

Abstract

<sec> <title>BACKGROUND</title> Large Language Models (LLMs) are increasingly used in healthcare. Non professionals, i.e., individuals without formal training in health-related professions, use LLM-based applications for health information, symptom assessment and disease management. These applications should fulfill their intended use while ensuring user safety and minimizing harm. Therefore, they must be evaluated in an appropriate manner. This is particularly important when applications are used by non-professionals, who may not have the medical knowledge to recognize errors, making them more susceptible to misinformation and harmful decisions. To date, guidance to evaluate LLM-based applications for non-professional users remains limited and fragmented, leaving researchers and developers without a scientifically grounded set of quality dimensions, metrics, and measurement tools to guide them. </sec> <sec> <title>OBJECTIVE</title> This protocol outlines a scoping review that maps approaches for evaluation of LLM-based applications that are used for health purposes by non-professionals. It identifies current methods and maps them thematically by assigning them to evaluation dimensions, metrics, and measurement instruments. The review provides a comprehensive overview of evaluation methods currently in use. </sec> <sec> <title>METHODS</title> The Scoping Review follows the Joana Briggs Institute (JBI) approach for conducting scoping reviews and reports in accordance with the PRISMA-ScR guideline. The inclusion criteria comprise studies that evaluate LLM-base applications that are used in the context of healthcare by non-professionals. The search is conducted in PubMED, CINAHL, PsycINFO and IEEE Xplore databases. Results since 2021 are considered. Data is summarized and interpreted qualitatively. Publication Screening and data extraction is conducted by two independent reviewers in a blinded manner, with discrepancies settled by discussion. </sec> <sec> <title>RESULTS</title> As of January 2026, a steering committee of 6 researchers has been chosen to for the realization of the review. An initial search resulted in 8538 Results after removing duplicates. We expect the screening and data extraction results in the first quarter of 2026. </sec> <sec> <title>CONCLUSIONS</title> The scoping review aims to identify and map current evaluation methods for LLM-based applications used in healthcare by non-professionals. It gives a systematic overview of the current state of research and gives insights into quality dimensions, metrics, and measurement instruments. The findings provide directional guidance for further research and development in the field of quality assurance for LLM-based applications used by non-professional users. </sec>

Ähnliche Arbeiten