An Intelligent Virtual Assistant for Symptom Assessment and Healthcare FAQ Resolution

Abstract

Medical chatbots often face challenges related to answering clinical questions contextualized appropriately, since the medical language is often complex and variable. The purpose of our work is to explore how multi-agent chatbots can improve medical question-answering through the delegation of tasks. The multi-agent system for medical chatbot learning includes three agents: an intermediary Controller Agent that provides access to all agents in the system; a Retrieval Agent providing a consistent, semantic search of medical documents; and a Generator Agent providing responses based upon the text returned from the Retrieval Agent. In a second configuration of the system, the Generator Agent and Summarizer Agent were replaced with a Symptom Checker Agent and Treatment Suggestion Agent, with a view toward supporting clinical actionability. To train and evaluate our models, we developed a consistent subset of the Medical Chatbot Dataset with 3,000 Q&A pairs. All documents were pre-processed to normalize the text and create a semantic embedding of each document. For evaluation, the BERTScore was used to evaluate 3 different sample sizes (10, 20, & 50 queries). The baselining model achieved F1 scores ranging from 0.7881 to 0.7905, while our enhanced model achieved an F1 score of 0.8138 and an improved recall from 0.7926 to 0.8560. The results of our study indicate that increased agent specialization of an agent or chatbot's functionality leads to improved performance of the agent; these results further support the increased utility of scalable and clinically relevant AI-assisted medical dialogue systems.

Ähnliche Arbeiten