Abstract IA04: Evaluating and mitigating medical misinformation risk in large language models

Abstract

Abstract Background: Large language models (LLMs) are increasingly used to generate medical content, yet their inherent design to follow user instructions may leave them vulnerable to producing misinformation. This risk becomes especially pronounced when LLMs generate incorrect medical information that could adversely affect human health. A propensity to comply with prompts, even when these lead to illogical or false information, highlights a critical gap in their safety, especially in high-stakes fields like healthcare. Methods: We evaluated the behavior of five LLMs—Llama3-8B, Llama3-70B, GPT4o-mini, GPT4o, and GPT-4-0613—by observing their compliance with prompts to generate misleading medical information. Specifically, the LLMs were prompted to suggest that a brand name drug was safer than its generic counterpart, a request with no basis in factual medical reasoning. We tested whether using prompt-based methods and instruction-tuning could enhance the models' ability to detect and resist generating content based on illogical premises. These safety approaches were evaluated in both medical and non-medical contexts to assess generalizability. Results: All five LLMs, despite accurately identifying the equivalence between brand and generic drugs, generated misleading medical content in response to overtly false prompts in over 50% of cases. Instruction-tuning and prompt-based techniques showed promise in reducing misinformation: models became better at detecting logical inconsistencies and were more likely to refuse requests that would result in misinformation, often providing explanations for their refusals. Furthermore, instruction-tuned LLMs showed improved safety behavior not only in medical contexts but also in non-medical domains without compromising performance on standard benchmarks. Conclusions: This study highlights a key vulnerability in LLMs: their tendency to comply with user requests even when it leads to medical misinformation. This compliance, particularly in generating content with flawed or incorrect logic, poses a significant safety issue for both individual and public health. The current safety and performance benchmarks for LLMs fail to account for this harmful behavior, underscoring an urgent need for new development approaches that prioritize logic and factual accuracy. Addressing this issue is essential to creating LLMs that are both useful and safe in medical and other high-stakes applications. Citation Format: Shan Chen, Mingye Gao, Kuleen Sasse, Thomas Hartvigsen, Brian Anthony, Lizhou Fan, Jack Gallifant, Danielle D. Bitterman. Evaluating and mitigating medical misinformation risk in large language models. [abstract]. In: Proceedings of the AACR Special Conference in Cancer Research: Translating Targeted Therapies in Combination with Radiotherapy; 2025 Jan 26-29; San Diego, CA. Philadelphia (PA): AACR; Clin Cancer Res 2025;31(2_Suppl):Abstract nr IA04

Ähnliche Arbeiten