Who’s an Author? Public Perception of Medical Advice from ChatGPT versus Medical Professionals

Abstract

BACKGROUND: ChatGPT is an artificial intelligence (AI) model that can generate text and provide responses to questions, including answers that include medical knowledge and advice. Prior studies have established that text written by the AI model is often indistinguishable from human generated text. The purpose of this study was to gauge public opinion on responses to a medical question written by actual medical professionals versus simply generated by ChatGPT. METHODS: Amazon’s Mechanical Turk crowdsourcing service and REDCap’s survey manager were used to recruit survey participants and collect responses. An anonymous 13-question survey was distributed that provided two different responses to a plastic surgery-related question about which patients are appropriate candidates for liposuction. One response was written by a medical professional while the other was generated by artificial intelligence (AI) machine ChatGPT. Responses were graded for various qualities and participants chose if they thought the response was written by AI or a medical professional. Five-point Likert scales were converted to binary variables for tabulation and a logistic regression analysis was performed. RESULTS: A total of 578 participants were included for analysis. All participants were in the United States and had a mean age of 35.3 years. When assessing both responses on warmth, conciseness, thoroughness and clarity, there were no statistically significant differences between medical advice provided by AI writers versus medical professionals (p>0.05). Overall, only 41% of the public was correctly able to identify ChatGPT’s response as having been written by AI. The public was better able to identify that the medical professionals’ response was human generated (70% vs. 41%, p<0.0001). Further, as respondent age increased, there was a significant increase in the ability to correctly identify AI-written text (p = 0.00082). Oppositely, as age increased, there was also a significant decrease in the ability to correctly identify that a response was written from a medical professional (p = 0.00082). CONCLUSION: When assessing responses to a plastic surgery-related question, respondents felt that the answers from both a medical professional and from ChatGPT were equally warm, concise, thorough and clear. Likewise, less than half of the public could identify a response as AI generated. However, significant limitations to ChatGPT have been made apparent, including the possibility for incorrect or misleading information. Therefore, considering its ability to produce convincing responses, increased vigilance by healthcare professionals is necessary to ensure that medical information relayed to the public is accurate.

Ähnliche Arbeiten