Comparative Evaluation of ChatGPT and Gemini in Answering Questions on Vaccines and Immunization

Abstract

Background/Aims: This study aimed to evaluate and compare the adequacy, reliability, and reproducibility of ChatGPT-4 Plus and Gemini Advanced chatbots in answering questions about vaccines and immunization. Given the growing role of artificial intelligence in healthcare communication, evaluating the performance of these models is crucial for their effective integration into public health strategies. Methods: A total of 56 questions were selected from World Health Organization educational materials and a national guideline on adult vaccination. Questions were categorized into four domains: Myths and Misconceptions, Vaccination and Immunization, Basic Principles of Vaccination, and Claims and Facts about Vaccines. Two infectious disease specialists independently rated chatbot responses using the Global Quality Score (1–5). Discrepancies were resolved by a third evaluator. Inter-rater reliability, reproducibility, and correlation of responses were also assessed. Results: ChatGPT-4 Plus received a perfect score (5 points) in 83.9% of responses, while Gemini achieved this in 71.4% of responses. Mean scores were similar (4.70±0.72 vs. 4.61±0.75, p=0.116). ChatGPT-4 Plus outperformed Gemini in the "Claims and Facts" category (p=0.012), while other categories showed no significant differences. Both models demonstrated substantial inter-rater agreement (κ=0.540 for ChatGPT, κ=0.572 for Gemini) and moderate correlation (r=0.521, p

Ähnliche Arbeiten