ChatGPT Performance on MCQ Exams in Higher Education. A Pragmatic Scoping Review

Abstract

Background. Media coverage suggests that ChatGPT can pass exams based on Multiple Choice Questions (MCQs), including those used to qualify doctors, lawyers, scientists etc. Objectives. We aimed to review current research evidence regarding the performance of ChatGPT on MCQ-based exams, along with recommendations for how educators might address the challenges and benefits of ChatGPT.Eligibility Criteria. Studies testing the performance of ChatGPT on MCQ-based exams in Higher Education. All disciplines and languages were included although the research itself was published in English.Results. 53 studies were included, covering 114 question sets, totalling 49014 MCQs. Free versions of ChatGPT based upon GPT-3/3.5 performed better than random guessing but failed most exams, performing significantly worse than the average human student. GPT-4 passed most exams with a performance that was on a par with human subjects. Conclusions. All summative MCQ-based assessments should be conducted under secure conditions with restricted access to ChatGPT and similar tools, particularly those which assess foundational knowledge. Authentic assessments could incorporate student use of ChatGPT since, due to its excellent performance, graduates will almost certainly be using it in practice.

Ähnliche Arbeiten