Envisioning the Future of Peer Review: Investigating LLM-Assisted Reviewing Using ChatGPT as a Case Study

Abstract

Peer review ensures research quality but faces growing challenges. Large Language Models (LLMs) offer potential solutions, yet most research has focused on independent AI-generated reviews rather than human-AI collaboration. We conducted a within-subject experiment with 24 HCI reviewers to evaluate LLM-assisted peer review’s impact on review efficiency, quality, workload, and user perceptions. Analysing subjective ratings, objective completion data, qualitative interviews, and chat history, we found that ChatGPT significantly reduces workload but does not substantially shorten review time or improve review quality. Reviewers valued its support for summarisation, information retrieval, idea generation, and confidence-building but noted challenges in content verification, interaction inefficiencies, and the need for human oversight. Our findings contribute to the broader conversation on the role of technology in scholarly publishing, offering design insights for supportive review systems that complement rather than replace human expertise, and highlighting the importance of responsible integration in academic peer review.

Ähnliche Arbeiten