Enhancing Real-World Data Extraction in Clinical Research: Evaluating the Impact of Implementing Large Language Models in Hospital Settings

Abstract

<title>Abstract</title> <bold>Background:</bold> The application of artificial intelligence (AI) and large language models (LLMs) in the medical sector has gained momentum. <bold/> The widespread adoption of electronic health record (EHR) platforms has created a demand for efficient extraction and analysis of unstructured data, known as real-world data (RWD). The surge in medical free-text data has emphasized the significance of natural language processing (NLP) in extracting insights from EHRs, making it a crucial tool in clinical research. The development of LLMs specifically designed for biomedical and clinical text mining has further propelled the capabilities of NLP in this domain. Despite these advancements, the specific utilization of LLMs in clinical research remains limited. <bold>Objective:</bold> This study aims to assess the feasibility and impact of implementing a LLM for extracting RWD in hospital settings. The primary focus is on evaluating the effectiveness of LLM-driven data extraction compared to manual processes used by Electronic Source Data Repositories (ESDR) system. Additionally, the study aims to identify challenges in LLM implementation and gain practical insights from the field. <bold>Methods:</bold> Researchers developed the ESDR system, integrating LLM, electronic Case Report Forms (eCRF) and EHR. The Paroxysmal Atrial Tachycardia Project, a single-center retrospective cohort study, served as a pilot case. The study involved deploying the ESDR system on the hospital LAN. Localized LLM deployment utilized the Chinese open-source ChatGLM model. The research design compared the AI-assisted process with ESDR manual processes in terms of accuracy rates and time allocations. Five eCRF forms, predominantly comprising free-text content, underwent evaluation, involving 630 subjects with a 10% sample (63 subjects) for assessment. Data collection involved electronic medical and prescription records from 13 departments. <bold>Results:</bold> While the discharge medication form achieved 100% data completeness, some free-text forms exhibited data completeness below 20%. The AI-assisted process showed an estimated efficiency improvement of 80.7% in eCRF data transcription time. The AI data extraction accuracy rate was 94.84%, with errors mainly related to localized Chinese clinical terminology. The study identified challenges in prompt design, prompt output consistency, and prompt output verification. Addressing limitations in clinical terminology and output inconsistency involves integrating local terminology libraries and offering clear output format examples. Enhancing output verification can be achieved by probing the model's reasoning, assessing confidence on a scale, and highlighting relevant text snippets. These measures mitigate challenges in understanding the model's decision-making process within extensive free-text documents. <bold>Conclusions:</bold> The research enriches academic discourse on LLM in clinical research and provides actionable recommendations for practical implementation in RWD extraction. By offering insights into LLM integration within clinical research systems, the study contributes to establishing a secure and efficient framework for digital clinical research. Continuous evolution and optimization of LLM technology are crucial for its seamless integration into the broader landscape of clinical research.

Ähnliche Arbeiten