#2996 Pilot insights from an online survey study interpreting clinical trials outcome; nephrologist recall vs. artificial intelligence

Abstract

Abstract Background and Aims Clinical trials (CTs) frequently provide extensive datasets from study outcomes that hold significant value in clinical practice. However, retaining and integrating this volume of information effectively can pose a considerable challenge for nephrologists. Artificial intelligence (AI) has shown substantial promise in analyzing or providing various forms of accurate medical data. We conducted an online survey to evaluate the accuracy of AI in delivering correct responses according to the recent nephrology CTs outcomes compared to the recall accuracy of nephrologists. Method We developed an online questionnaire on Google Forms, consisting of 10 questions from recent nephrology CTs on IgA nephropathy, lupus nephritis, C3 glomerulopathy, and polycystic kidney disease. The selected trials included APPLAUSE-IgAN, NefIgArd, PROTECT, STOP-IgAN, BLISS-LN, ALIGN, VALIANT, APPEAR-C3G, TEMPO 3:4, and REPRISE trials. During the two-day Loma Linda Rare Kidney Diseases Conference, held in Palm Springs in December 2024, these clinical trials were discussed in detail during various sessions. Conference participants, including nephrologists and nephrology trainees, were invited to complete the survey after the conference. The responses from the questionnaire were subsequently submitted to four AI models for comparison, including two free versions (Gemini 1.5 Flash and ChatGPT-4) and two paid versions (Gemini 1.5 Pro and ChatGPT-4 Turbo). The accuracy of AI-generated responses was recorded and compared to the answers provided by nephrologists. Results Thirty-one participants completed the anonymized online survey, with 24 individuals affiliated with academic institutions, including nephrology fellows, assistant professors, and associate professors. The median (IQR) years of practice since completing nephrology training was 5 (2.5–10) years. While no significant differences were observed for most questions, AI outperformed human subjects in answering a few questions more accurately. Overall, AI demonstrated significantly superior accuracy in responses compared to human participants (median (IQR): 95% (85%–100%) vs. 40% (30%–80%), P = 0.005). The study assessed four AI platforms, including free and paid ChatGPT and Gemini versions. Paid versions demonstrated higher accuracy than free versions; ChatGPT-4 (80%), ChatGPT-4 Turbo (100%), Gemini 1.5 Flash (90%), and Gemini 1.5 Pro (100%). Conclusion AI consistently demonstrated superior performance to nephrologists, with paid versions achieving 100% accuracy. This finding highlights the potential for clinicians to utilize AI to access specific data efficiently, reducing the need to spend significant time searching the literature, particularly given the demands of their busy schedules.

Ähnliche Arbeiten