MP03-14 COMPARING AI-GENERATED VS. COCHRANE REVIEW SUMMARIES IN INCONTINENCE RESEARCH: A COMPREHENSIVE ANALYSIS

Abstract

You have accessJournal of UrologyUrodynamics/Lower Urinary Tract Dysfunction/Female Pelvic Medicine: Male Incontinence (MP03)1 May 2024MP03-14 COMPARING AI-GENERATED VS. COCHRANE REVIEW SUMMARIES IN INCONTINENCE RESEARCH: A COMPREHENSIVE ANALYSIS Angelo Cadiente, Catherine Implicito, Andre Ho, Abinav Udaiyar, Christopher Wan, Jamie Chen, Charles Palmer, Qilin Cao, Michael Raver, and Mubashir S. Billah Angelo CadienteAngelo Cadiente , Catherine ImplicitoCatherine Implicito , Andre HoAndre Ho , Abinav UdaiyarAbinav Udaiyar , Christopher WanChristopher Wan , Jamie ChenJamie Chen , Charles PalmerCharles Palmer , Qilin CaoQilin Cao , Michael RaverMichael Raver , and Mubashir S. BillahMubashir S. Billah View All Author Informationhttps://doi.org/10.1097/01.JU.0001009488.55564.85.14AboutPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissionsReprints ShareFacebookLinked InTwitterEmail Abstract INTRODUCTION AND OBJECTIVE: As the volume of medical literature continues to expand, the provision of artificial intelligence (AI) to give concise, accessible summaries has the potential to lead to improved efficacy. Using Cochrane Review, a systematic review database, this project assesses the readability and quality of summaries generated by ChatGPT in comparison to the Plain Text Summaries from Cochrane Reviews in incontinence research. METHODS: 73 abstracts from the Cochrane Library tagged under "Incontinence" were summarized using ChatGPT-3.5 (July 25 Version) and compared with their corresponding Cochrane Plain Text Summaries. Readability was assessed using six metrics: Flesch Kincaid Reading Ease, Flesch Kincaid Grade Level, Gunning Fog Score, Smog Index, Coleman Liau Index, and Automated Readability Index. With the exception of the Flesch Kincaid Reading Ease score, each measure describes the amount of formal education required to understand a given response. A two-tailed t-test was used to compare ChatGPT-3.5-generated summaries and Cochrane Plain Text Summaries. Each summary was also evaluated by two blinded, independent graders on a 5-point scale for accuracy and adherence to the abstract, with their combined grades compared between datasets. RESULTS: Compared to ChatGPT, Cochrane Review's Plain Text Summaries scored higher in the Flesch Kincaid Reading Ease score and lower in the five other readability metrics with statistical significance, indicating better readability. However, ChatGPT earned a better mean grade of 4.25 compared to Cochrane Review's mean grade of 4.05 with statistical significance (Table 1). CONCLUSIONS: Cochrane Review's Plain Text Summaries provide clearer summaries of the incontinence literature when compared to ChatGPT; yet, ChatGPT generates more comprehensive summaries. While ChatGPT can effectively summarize the medical literature, further studies focused on prompt engineering ought to be undertaken to improve reader accessibility to these summaries. Source of Funding: None to disclose © 2024 by American Urological Association Education and Research, Inc.FiguresReferencesRelatedDetails Volume 211Issue 5SMay 2024Page: e28 Advertisement Copyright & Permissions© 2024 by American Urological Association Education and Research, Inc.Metrics Author Information Angelo Cadiente More articles by this author Catherine Implicito More articles by this author Andre Ho More articles by this author Abinav Udaiyar More articles by this author Christopher Wan More articles by this author Jamie Chen More articles by this author Charles Palmer More articles by this author Qilin Cao More articles by this author Michael Raver More articles by this author Mubashir S. Billah More articles by this author Expand All Advertisement PDF downloadLoading ...

Ähnliche Arbeiten