Retrieval-Augmented Large Language Models for a Chronic Kidney Disease Patient Education Chatbot

Abstract

Chronic Kidney Disease (CKD) is rising globally, underscoring the need for scalable and accurate patient education. This paper presents the design and evaluation of a CKD education chatbot specifically for hemodialysis cases in Indonesia that runs on-premises and employs a Retrieval-Augmented Generation (RAG) framework. We conducted a comparative study of a general-purpose LLM (Llama 3.1) and a language adapted LLM (Sahabat AI). Both models were used in identical RAG pipelines and paired with several retrieval embeddings. Using 70 unique questions evaluated across 10 runs per configuration (n = 700), we assessed semantic similarity (METEOR), factual correctness, and RAG-specific metrics (faithfulness, answer relevancy, context precision) with a Gemini 2.0 Flash evaluator. RAG consistently improved performance over non-RAG baselines, with statistically significant gains across metrics. Sahabat AI outperformed Llama 3.1, posting the highest METEOR (0.4116, mContriever) and correctness (4.0293 with BGE-M3). It also exhibits higher, more stable faithfulness (∼0.95), indicating closer adherence to the source evidence. These findings suggest that, for healthcare education, localized LLMs combined with carefully chosen embeddings can yield more effective and trustworthy systems.

Ähnliche Arbeiten