Evaluating AI toolchains for systematic literature reviews using a rubric-based workflow approach

Abstract

Artificial Intelligence (AI) tools are increasingly applied to support systematic literature reviews (SLRs), but researchers lack structured guidance for evaluating and integrating these tools into their workflows. This study proposes and applies a rubric-based framework for assessing AI toolchains across core SLR tasks, including search, screening, and synthesis. A mixed-methods evaluation was conducted with 63 postgraduate student researchers, representing early-stage users who are primary adopters of such workflows. Findings indicate that rubric-guided workflows improved participant-perceived usefulness of AI tools, supported structured criteria for comparison, and facilitated greater transparency in tool adoption. While formal inter-rater reliability statistics (e.g., Cohen’s or Fleiss’ Kappa) could not be computed because participants did not apply the rubric in parallel due to the absence of a common benchmark set, rubric consistency was qualitatively validated through pilot testing, iterative refinement, and structured participant discussions. This highlights interpretive alignment while acknowledging the need for future replication with independent evaluators. The study contributes a validated evaluation rubric, an illustrative workflow for AI-assisted SLRs, and reflections on implications for research training, reproducibility, and cross-domain adaptation.

Ähnliche Arbeiten