Comparative Analysis of a Custom Lightweight LLM Versus General-Purpose LLMs for Medical Query Handling

Abstract

Large language models (LLMs) have demonstrated remarkable versatility in natural language understanding and generation, yet their reliability in specialized medical domains remains uncertain. We focus on the use of lightwight and specialized LLMs instances on cardiological medical data corpora. We present a comparative evaluation of four general-purpose LLMs, i.e., ChatGPT, Gemini, Claude AI, and PerplexityAI trained on cardiology clinical data. We assess response quality on set of clinically relevant queries with a score in the [0,1] interval. Score were grouped in three classes: good (scores greater than 0.75), sufficient (scores in the interval 0.51-0.74), insufficient (scores less than 0.5). Results show that the lightweight model achieved the highest proportion of sufficient answers evaluated as sufficient (54%) and a greater share of answers evaluated as good (23%) than general-purpose counterparts, while maintaining a relatively low proportion of insufficient responses <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$(23 \%)$</tex>. In contrast, general-purpose models exhibited greater variability, alternating between highly accurate and critically lacking outputs, with PerplexityAI performing weakest overall. These findings suggest that targeted domain adaptation, even with lightweight architectures, can yield more stable and clinically reliable outputs than large general-purpose systems. The study underscores the potential of lightweight, specialized LLMs as trustworthy components in medical decision support frameworks, where consistency and factual grounding are paramount.

Ähnliche Arbeiten