Faster, better?: Testing artificial intelligence accuracy for neurosurgical literature analysis

Abstract

With the rapid rise of artificial intelligence tools, applications like ChatPDF are seen as promising for supporting academic tasks in neurosurgery, such as literature review, summarization, and question generation. However, its accuracy and relevance remain to be critically assessed. This study assesses ChatPDF's accuracy in interpreting neurosurgical research articles, aiming to identify its strengths and limitations. Articles from the 10 highest-ranked neurosurgical journals were reviewed by selecting the first original research article from each journal's 2023 volume. Ten detailed questions were independently generated by 2 researchers based on each article's content. Each article was then uploaded to ChatPDF, which generated its own questions and provided responses to both its questions and those posed by the researchers. Responses were categorized as completely correct, partially correct, or incorrect. Source reliability was also evaluated to determine ChatPDF's performance. An overall accuracy rate of 89% was achieved by ChatPDF across 100 questions, with 89% of responses classified as completely correct, 5% as partially correct, and 6% as incorrect. Source reliability averaged 83%, although variability was noted, particularly in journals such as the Journal of Neurosurgery: Spine and Neurosurgery Clinics, which showed lower reliability rates. Substantial accuracy and potential were demonstrated by ChatPDF as a supplementary tool for neurosurgical literature review. However, limitations such as inconsistent source reliability and lack of visual content analysis highlight the need for ongoing refinement. While promising, ChatPDF should be used alongside manual verification to ensure comprehensive and accurate literature interpretation in neurosurgical research.

Ähnliche Arbeiten