Comparison of Antiobesity Medication Decisions: Large Language Models and Obesity Medicine Expert Clinicians

Abstract

Background: Artificial intelligence, specifically deep learning-based large language models (LLMs), are quickly emerging as a tool for supporting clinical decision-making. We asked whether current LLMs can accurately and safely perform clinical decision-making for obesity medication prescribing. Methods: Twenty hypothetical patients visiting weight management clinics were created with detailed medical and personal histories. Four obesity medicine experts were asked to choose their first-choice weight loss medication, as were four LLMs: GPT-4o, Llama3, Gemini1.5, and Claude3.5. All the LLMs underwent identical prompt tuning and were configured with a uniform temperature setting of 1.0. All medication choices by the obesity medicine experts were presumed correct and a binary system was used to compare whether the LLMs did or did not select one of the medications proposed by the experts. Hallucination incidence and high-risk medication choices are evaluated. Results: Gemini1.5 and GPT-4o had the highest matching rate with experts (55%), followed by Claude3.5 Sonnet (50%) and Llama3 (30%). Only Gemini1.5 avoided high-risk or contraindicated medications in all cases. Among the LLMs, Llama3 had a statistically significant high hallucination incidence rate ( p < 0.01). Some of the LLMs’ choices and rationales were creative and reasonable, offering insights that experts had not considered. Conclusion: LLMs can offer valuable insights into clinical decisions for obesity treatment. However, safety and accuracy concerns persist due to hallucinations. It is premature to use LLMs as clinical support tools. They need refinement through fine-tuning, detailed prompt tuning, real-world data incorporation, and rigorous validation to ensure reliability and safety.

Ähnliche Arbeiten