Evaluating ChatGPT’s responses to vaccine-related questions: the impact of question framing on content and quality

Abstract

Background: Vaccine hesitancy, fueled by mistrust, fear, and misinformation, remains a major public health challenge. Generative artificial intelligence tools such as ChatGPT have emerged as new information sources, particularly for younger users. It is essential that these tools provide medically accurate content, especially when responding to negatively framed vaccine questions. This study aims to examine how ChatGPT responds to vaccine-related questions, focusing on questions with negative or skeptical framing. Methods: test; qualitative feedback was thematically analyzed. Results: 3 [3-4]. No significant differences were found for any item. Among 81 free-text comments, the most frequent concerns were "bias toward COVID-19 vaccines" (n=38), "insufficient explanation" (n=19), and "potentially misleading expressions" (n=9). Examples included overemphasis on COVID-19 in unrelated contexts and problematic phrasing regarding human papillomavirus vaccine adverse events. Conclusions: ChatGPT maintained comparable quality in Japanese responses to both supportive and critical vaccine questions, suggesting resilience to negative framing. However, expert reviewers identified thematic biases, occasional inadequacy of detail, and linguistic issues that could mislead lay readers. These findings underscore the need for continued human oversight, refinement of Japanese-language outputs, and algorithmic adjustments to reduce topical bias.

Ähnliche Arbeiten