Employing A Large Language Model to Assess Practitioner Rationale for Bypassing Clinical Decision Support System Alerts for Duplicate Imaging (Preprint)

Abstract

<sec> <title>BACKGROUND</title> Clinical Decision Support System (CDSS) alerts offer summarized computerized knowledge to aid clinical decision-making. However, if prompted inappropriately, these alerts can consume practitioner time and contribute to alert fatigue. A thorough examination of practitioner feedback to these alerts, although time consuming, can help to promote changes in prompts to minimize inappropriate alerts. With the recent developments of large language models (LLMs), these models can be employed to analyze textual data much more rapidly than human review. </sec> <sec> <title>OBJECTIVE</title> To analyze the current capabilities of an LLM to analyze practitioner free-text responses for bypassing our institution's CDSS alert. </sec> <sec> <title>METHODS</title> These alerts are prompted by the order of a chest X-Ray for a patient that had a separate chest X-Ray or computed tomography in the past 48 hours. All 4,789 practitioner free-text rationales for bypassing this alert were entered into OpenAI’s Chat-GPT-4o mini LLM, which was prompted to classify the responses into five categories: 1) imaging before or after a procedure, 2) imaging after a change in clinical status, 3) imaging to monitor a previously imaged condition, 4) unintelligible responses, 5) other. Categories 1 and 2 were deemed appropriate, 3 and 4 were considered inappropriate, with category 5 indeterminate. Manual review served as the reference standard. </sec> <sec> <title>RESULTS</title> 4,789 free-text responses from 933 providers were included in this study. The mean length of provider responses was 12.8 characters and 2.2 words per response. The LLM categorized over three-fourths of responses correctly (3,791 responses, 77.8%). ChatGPT accurately determined the appropriateness of 85.6% of responses. </sec> <sec> <title>CONCLUSIONS</title> Our findings suggest LLMs may be able to aid clinical quality improvement projects by analyzing free-text responses to CDSS alerts. However, caution should be taken when interpreting their outputs given the presence of inaccuracies. </sec>

Ähnliche Arbeiten