Generative artificial intelligence chatbots versus physicians on answering patients’ questions about nosocomial pneumonia

Abstract

<bold>Background:</bold> Generative artificial intelligence (AI) chatbots are commonly used to receive medical advice, but there is no data about its performance in answering questions about nosocomial pneumonia. <bold>Objective:</bold> To assess patients` attitudes to information about nosocomial pneumonia generated by generative AI chatbots and physicians. <bold>Methods:</bold> We received five answers to questions about nosocomial pneumonia from ChatGPT, Gemini, Microsoft Copilot and 2 physicians (PhD, MD). During January-February 2025 we presented these blinded answers to 20 patients with nosocomial pneumonia (mean age 31,7±11,7 years, 55% women, 45% men) and asked them to assess the overall score, willingness to recommend to other patients, completeness, usefulness, and Turing test (whether patients can recognize AI) using 10-point Likert scale. <bold>Results:</bold> The usefulness of answers generated by Gemini was significantly higher than by physicians (8.41 ± 1.3 vs. 7.72 ± 2.0; p=0.012). The completeness of physicians’ answers (7.49 ± 1.7) was significantly worse than all chatbots: ChatGPT (8.16 ± 1.3 (p=0.006)), Gemini (8.54 ± 1.4 (p<0.001)), Copilot (8.10 ± 1.3 (p=0.015)). The willingness to recommend Gemini`s answers was significantly higher than physicians’ (8.53 ± 1.4 vs. 7.92 ± 2.0; p=0.039). Gemini's performance in the Turing test (36%) was significantly worse than ChatGPT (75%, p<0.001), Copilot (64%, p<0.001), or physicians (67%, p<0.001). <bold>Conclusion:</bold> Overall, patients` attitudes to all generative AI chatbots’ answers were high. Gemini received the highest score in willingness to recommend answers. ChatGPT showed the best result in the Turing test, gaining an advantage even over physicians.

Ähnliche Arbeiten