The Potential for Artificial Intelligence to Augment Peer Review

Abstract

Artificial intelligence (AI) is a new technology that represents a promising tool to augment the scientific peer-review process, particularly by addressing the challenges of scalability, consistency and bias in the review process. The term AI was formally introduced by computer scientist John McCarthy in 1955 as a part of a proposal for the famous 1956 Dartmouth Conference [1]. He defined AI as ‘the science and engineering of making intelligent machines’ [2]. This conference is widely considered the founding event of AI as a formal academic discipline. One of the primary benefits of AI lies in its potential to support the triaging and initial screening of journal submissions. With the rapid growth in submission volumes, journals face pressure to manage and filter manuscripts more efficiently [3]. AI-based tools, such as large language models (LLMs), can screen manuscripts for issues like plagiarism, adherence to formatting guidelines and even alignment with specific journal standards. An AI agent could also confirm clinical trial registration, pull the registered prespecified primary and secondary outcomes and compare them with the manuscript to flag discrepancies. This pre-review process can save reviewers significant time and allow them to focus on evaluating the scientific merit of submissions [4]. Manuscript authorship has evolved rapidly due to the use of LLMs. A large-scale analysis of over 15 million PubMed abstracts estimated that at least 13.5% of 2024 abstracts display LLM-style markers. This is fundamentally changing what reviewers and editors observe [5]. Active pilot studies are already underway. Risk Analysis reports potential deployments of an AI ‘Screener’ for authors (live since Jan 2025) and an AIASTROBE Plus prereview prototype [6, 7]. Journals are testing process changes at scale. Elsevier has implemented structured peer-review prompts in over 300 journals after a preliminary study increased reviewer agreement (from about 31% to about 41%) [8]. This platform can easily support randomised or quasi-experimental evaluations of AI-assisted review templates and checklists. Beyond initial screening, AI can also enhance the depth and precision of peer reviews by supporting fact-checking, statistical consistency and image analysis. For instance, tools like StatReviewer are designed to evaluate statistical rigour within manuscripts, providing checks on methodological soundness that may sometimes be missed in traditional reviews due to human error or time constraints [9]. Additionally, AI-driven algorithms may help detect potential biases or ethical concerns in manuscripts, such as data fabrication or image manipulation, plagiarism, duplicate publication and undisclosed conflicts of interest. These are challenging for human reviewers to identify effectively [10]. Despite these benefits, AI-assisted peer review raises ethical questions, such as the risks of algorithmic bias and over-reliance on automated decisions. While AI can detect patterns, it may inadvertently replicate biases embedded within training data or produce inconsistent assessments in complex cases where nuanced judgement is required. Therefore, current research suggests a hybrid model of AI and human oversight, where AI augments but does not replace human reviewers. This can optimise the strengths of both approaches. This model preserves human judgement for high-level decision-making, while AI handles more routine, structured tasks [11-13]. With these capabilities, AI has the potential to make peer review more efficient and equitable, enabling broader access to publication opportunities while maintaining rigorous scientific standards. However, a balanced approach integrating both AI and human insights will likely be key to realising these advances fully and ethically [14, 15]. SWOT analyses can guide strategic decision-making by highlighting internal and external factors that may impact success [16]. It is a tool to identify and evaluate the Strengths, Weaknesses, Opportunities and Threats related to a particular organisation, project, or initiative. A SWOT analysis of the use of AI for peer review shows that AI can bring efficiency, consistency and scalability to peer review by automating repetitive tasks, standardising evaluations and enhancing quality control. However, it remains limited by biased training data, a lack of nuanced understanding and ethical concerns that risk over-reliance on automation. Its integration offers opportunities to promote fairness, develop hybrid review models and enable data-informed editorial decisions that evolve through continuous learning. Yet, challenges persist, including the perpetuation of bias, resistance from the scientific community and emerging legal or regulatory hurdles that could constrain its adoption (Table 1). Very little is known about the current knowledge and attitude of participants in the peer review process toward the use of AI. IOP Publishing surveyed its global reviewer community to understand how generative AI is entering peer-review practice in the physical sciences. In August 2025, invitations were sent to 30,570 researchers who had either reviewed for or co-authored in an IOP-owned journal within the prior 6 months, yielding 348 respondents. The instrument covered demographics, prior AI use in reviewing, perceived impacts and ethical concerns. At the time of the study, IOP's policy did not permit AI use in peer review. The findings demonstrate that polarisation is increasing, with 41% of respondents predicting a positive impact of generative AI on peer review, 37% a negative impact and 22% were neutral. About a third (32%) reported already using AI tools to support review work, most often to polish prose/flow, summarise manuscripts, or translate. Named tools were led by ChatGPT, followed by DeepSeek and Gemini. There were several concerns raised including 57% would be unhappy if a reviewer wrote a report with AI and 42% would be unhappy if AI only augmented the report. An additional 42% believed they could detect an AI-generated review. Overall, neutrality shrank compared with IOP's 2024 baseline, with both positive and negative views rising. Another large-scale, international cross-sectional survey assessed medical-journal peer reviewers' attitudes toward AI chatbots (AICs) in peer review and was published by Ng et al. [17]. Eligibility required completing ≥ 1 review for a MEDLINE-indexed journal in the prior 24 months. Email contacts were compiled from a frame of 72,847 corresponding authors. The SurveyMonkey questionnaire (pilot tested and CHERRIES/STROBE-informed) launched in April 2025 and remained open for 7 weeks, resulting in 1194 respondents and 1018 completions from 33,388 opened invitations (response rate 3%). Most respondents reported never having used an AI chatbot for peer-review tasks (707/999 [70.8%]), and nearly half did not anticipate doing so (460/1007 [45.7%]). Nonetheless, interest in capacity building was high (587/980 [59.9%] wanted training). The most endorsed potential benefit was workload reduction (601/958 [62.7%]), while the most endorsed risk was producing errors/inaccuracies (738/934 [79.0%]). Senior-career researchers constituted a slight majority (56.8%), suggesting current attitudes may reflect established reviewers' caution despite recognised efficiency gains. Integrating AI into the peer review process presents both transformative potential and critical challenges for academic publishing. It offers distinct advantages in streamlining initial manuscript triage, enhancing quality control through statistical checks and potentially reducing reviewer bias. This can allow human reviewers to focus on deeper scientific evaluations and clinical application of new evidence. However, using AI raises concerns regarding ethical implications, algorithmic biases and over-reliance on automated processes, which could compromise the nuanced judgement that complex studies often require. Moving forward, a hybrid approach combining AI's efficiency with human oversight may provide a balanced pathway, optimising the peer review process without diminishing its rigour. We must ensure that we use any new technology ethically and responsibly. By thoughtfully addressing the opportunities and challenges of AI, the academic emergency medicine community can leverage this new tool to enhance the quality, integrity and inclusivity of scientific publications. This would promote the trustworthiness of the scientific literature. During the preparation of this work, Dr. W. Kenneth Milne used ChatGPT (OpenAI, San Francisco, CA) to assist with drafting the initial content, language refinement and phrasing during manuscript preparation. The authors reviewed and edited the content as needed and take full responsibility for the publication's content. The authors declare no conflicts of interest. Data sharing not applicable to this article as no datasets were generated or analysed during the current study.

Ähnliche Arbeiten