S928 Accuracy of a Chat-Vased Artificial Intelligence Model in Answering Frequently Asked Questions and Providing References Related to Inflammatory Bowel Disease

Abstract

Introduction: With the rise of easily accessible Artificial Intelligence (AI)chatbots like ChatGPT, patients now have novel ways to search for health-related information. However, their accuracy in answering disease-specific questions is unexplored. We sought to assess the accuracy of ChatGPT-4 in providing appropriate responses and references to frequently asked IBD questions. Methods: Using Google Trends, an online tool that tracks and analyzes the popularity of specific search queries, we formulated 20 questions based on frequently searched terms related to IBD over the past 5 years. The questions were entered into ChatGPT-4 on 5/5/2023, and the model was asked to provide 2 references for each response. Three gastroenterologists independently reviewed the answers and references for accuracy. They were asked to provide a detailed grade (100% accurate, 100% inaccurate, partly inaccurate, accurate with missing information) and overall grade (accurate, inaccurate). References were graded as suitable, unsuitable (existent but unrelated to answer) or nonexistent. When references for the same answer had different grading, the overall grade was given according to the least accurate reference (nonexistent >unsuitable >suitable). Answers with disagreement were discussed by the reviewers, permitting potential adjustments post-discussion. Agreement level was computed using free-marginal Kappa coefficients (κ). Results: There was 100% agreement for the overall grade (accurate or inaccurate; κ=1) and almost perfect agreement for the detailed grade (86.7%, κ=0.82, 95%CI [0.66, 0.98]) following consensus discussion. On average, 55% of ChatGPT-4 responses were rated as 100% accurate, 40% partly inaccurate and 5% accurate with missing information on detailed assessment. Overall, the accuracy level of chatGPT-4 responses was 75% (Table 1). In terms of references, 15% of questions had at least one reference that was unsuitable and 50% of questions had at least one reference that was non-existent. An example is presented in Figure 1. Conclusion: This study demonstrates the potential role of ChatGPT-4 as a healthcare tool addressing online IBD-related patient queries. Although the model's accuracy was not perfect, an overall accuracy level of 75% is acceptable, indicating a promising potential. However, before contemplating the possibility of a comprehensive integration of ChatGPT-4 into healthcare, we must recognize the necessity for improvement, particularly in providing suitable references.Figure 1.: Example of a question entered into ChatGPT-4 and assessment of the answer’s accuracy and references’ suitability by one gastroenterologist reviewer. Table 1. - List of the 20 IBD-related questions entered into ChatGPT-4 and grading of the responses’ overall accuracy and references suitability Question Overall accuracy References 1 How is IBD diagnosed? Provide 2 references. Accurate Unsuitable 2 What is the difference between Crohn’s disease and ulcerative colitis? Provide 2 references. Inaccurate Unsuitable 3 What are the treatment options for IBD? Provide 2 references. Accurate Nonexistent 4 How can I manage fatigue associated with IBD? Provide 2 references. Accurate Nonexistent 5 What is the impact of IBD on mental health? Provide 2 references. Accurate Suitable 6 Can IBD affect fertility or pregnancy? Provide 2 references. Inaccurate Unsuitable 7 How can I explain IBD to friends and family? Provide 2 references. Accurate Suitable 8 What is the link between IBD and arthritis? Provide 2 references. Accurate Suitable 9 Can probiotics help manage IBD symptoms? Provide 2 references. Accurate Suitable 10 What are the surgical options for treating IBD? Provide 2 references. Accurate Suitable 11 How can I find support and resources for living with IBD? Provide 2 references. Accurate Suitable 12 What is the difference between IBD and Irritable Bowel Syndrome (IBS)? Provide 2 references. Inaccurate Suitable 13 What is the best diet for IBD? Provide 2 references. Inaccurate Nonexistent 14 What are the causes of IBD? Provide 2 references. Accurate Nonexistent 15 What is calprotectin? Provide 2 references. Accurate Nonexistent 16 How to recognize an IBD flare-up? Provide 2 references. Accurate Nonexistent 17 What is ileitis? Provide 2 references. Accurate Nonexistent 18 Do cigarettes help with IBD? Provide 2 references. Inaccurate Nonexistent 19 I was recommended a biologic for IBD. Will this lead to infections and cancer? Provide 2 references. Accurate Nonexistent 20 Which biologic is the best to treat ulcerative colitis? Provide 2 references. Accurate Nonexistent

Ähnliche Arbeiten