HerCare: Development and Formative Evaluation of a Dual-Source Retrieval-Augmented Generation Chatbot for Women’s Health (Preprint)

Abstract

<sec> <title>BACKGROUND</title> Conversational agents for women’s health often fail to meet user needs, offering either clinically sterile advice or unreliable peer anecdotes. This limitation creates a tension between the need for factual safety and emotional resonance in sensitive health contexts. </sec> <sec> <title>OBJECTIVE</title> We aimed to address this gap by developing and conducting a formative evaluation of HerCare, a conversational agent built on a novel dual-source Retrieval-Augmented Generation (RAG) architecture. The system systematically fuses expert medical knowledge with peer narratives to act as a “trust-calibration mechanism” through transparent source attribution. </sec> <sec> <title>METHODS</title> We conducted a remote, single-session field study with 243 participants to evaluate the system's usability, trustworthiness, and affective dynamics. Participants interacted with the agent regarding a women's health topic of personal interest. We employed a mixed-methods approach, combining standardized self-report metrics—the Chatbot Usability Questionnaire (CUQ) and Net Promoter Score (NPS)—with computational linguistic analyses (VADER sentiment analysis and NRC Emotion Lexicon) of 1,191 conversational turns. </sec> <sec> <title>RESULTS</title> Participants reported high system usability, with a mean CUQ score of 75.67 (SD 15.50). The system achieved a Net Promoter Score (NPS) of 60.0, indicating strong willingness to recommend the tool. In the Post-Interaction Questionnaire (5-point scale), participants rated Helpfulness (mean 4.55) and Trustworthiness (mean 4.35) highly. Computational analysis revealed a consistent conversational polarity shift: while user queries often skewed neutral-to-negative (compound scores down to -0.181), agent responses were consistently and strongly positive (compound scores +0.555 to +0.826). Emotion profiling identified a recurring “Validate, then Redirect” strategy, where the agent acknowledged distress (Sadness) before pivoting to a frame of Trust and Anticipation. </sec> <sec> <title>CONCLUSIONS</title> The dual-source architecture successfully balanced clinical accuracy with emotional support, resulting in high perceived empathy and trust. These findings demonstrate the feasibility of weaving clinical sources with lived experiences to create safer, more resonant health AI, offering a transferable design pattern for future empathy-attuned systems. </sec>

Ähnliche Arbeiten