An in-depth analysis of AI-generated scientific review articles and its potential implications on the future of medical journal publications

Abstract

• AI-generated articles are of poor quality, based on SANRA scores ranging from 0.75 to 5.38 out of 12 possible points. • None of the AI-generated articles achieved a score comparable to the human-written review or a score above the threshold required for acceptance by a journal's editors. • Inaccurate and false citations are a major problem: Many citations in AI-generated were unverifiable or misattributed, compromising academic credibility of an article. • Some AI-generated articles had high similarity indices (up to 52%), raising concerns about plagiarism and academic integrity. • With the improvement of key functions, AI programs could eventually reduce the intellectual contribution of authors of narrative review articles to a negligible minimum, but not with the technology currently available. Key issues related to the use of artificial intelligence (AI) in scientific writing include intellectual property, authorship, falsification and AI hallucination/bias. The concerns of using AI in scientific writing are premised on the proposition that AI can indeed generate high-quality research. The aim of the study was to determine whether AI writing programs can independently write and reference a high-quality narrative review article on the quantitation of plasma cells in bone marrow samples and compare it to a human-written narrative review article on the same topic. The AI-generated articles and the human-written review were critically evaluated by four raters using the Scale for the Assessment of Narrative Review Articles (SANRA). The plagiarism detection tool, Turnitin was used to assess originality of the content. The AI-generated articles scored poorly on the SANRA scale ranging from scores of 0.75 - 5.38, while the human-written review, on the other hand, scored 11.13 out of 12 possible points. Inter-rater reliability was excellent (ICC = 0.977 [95% CI 0.94-0.99]), confirming consistent application of the SANRA scoring across raters. AI-generated outputs for narrative reviews have significant quality limitations, therefore oversight by human subject experts is required to ensure scientific integrity and rigor. Future improvements in AI technology could increase its utility for scientific writing.

Ähnliche Arbeiten