Unraveling ChatGPT's Performance in Addressing ESKD: Implications for Artificial Intelligence (AI)-Assisted Healthcare

Abstract

Background: ChatGPT, an artificial intelligence language model, is at the forefront of cutting-edge technology. It has shown abilities in natural language processing tasks, producing responses resembling those crafted by human beings. While there is discourse about the potential of ChatGPT as a substitute for physicians, its abilities in the field of nephrology, particularly in ESKD including dialysis, remains uncertain. The objective of this study is to assess the performance of ChatGPT in addressing fundamental inquiries pertaining to ESKD. Methods: We conducted an evaluation of ChatGPT's accuracy in answering questions related to CKD, ESKD, including hemodialysis, and peritoneal dialysis, using the ASN eLEARNING CENTER (nephSAP vol1-No2 and Dialysis Core Curriculum 2021). There were 95 questions included. Each question set was executed twice using ChatGPT (Mar 14 version, OpenAI), and the level of agreement between the initial and subsequent run, conducted two weeks apart, was determined. Also, an assessment was performed using ChatGPT using the query, “Based on these findings, what is ChatGPT's performance, and is ChatGPT ready to provide answers pertaining to ESKD?” Results: In our study evaluating ChatGPT's performance in answering questions related to CKD and ESKD, we found that on the two different question banks combined, ChatGPT achieved accuracies of 54% and 57% on the first and second runs, respectively. The overall agreement between the two runs was 71%. The study revealed that the level of agreement between the initial and subsequent runs of ChatGPT was higher for correct answers compared to incorrect ones, concordance of 46% vs 24%, respectively. Among the 28 instances where ChatGPT provided different responses, it changed from incorrect to correct in 10 questions (36%), from correct to incorrect 7 times (25%). ChatGPT acknowledged these results, further highlighting its limitations in accurately addressing questions related to ESKD. Conclusions: The current study demonstrates that ChatGPT's accuracy in answering questions related to ESKD is below the minimum passing threshold of 75% set by the ASN for nephrologists, with an accuracy of 55% (average of the two runs), indicating the need for further development and training to improve its accuracy and consistency.

Ähnliche Arbeiten