Do humans identify AI-generated text better than machines? Evidence based on excerpts from German theses

Abstract

We investigate whether human experts can identify AI-generated academic texts more accurately than current machine-based detectors. Conducted as a survey experiment at a German university of applied sciences, 63 lecturers in engineering, economics, and social sciences were asked to evaluate short excerpts (200–300 words) from both human-generated and AI-generated texts. These texts varied by discipline and writing level (student vs. professional) with the AI-generated content. The results show that both human evaluators and AI detectors correctly identified AI-generated texts only slightly better than chance, with humans achieving a recognition rate of 57 % for AI texts and 64 % for human-generated texts. There was no statistically significant difference between human and machine performance. Notably, professional-level AI texts were the most difficult to identify, with less than 20 % of respondents correctly classifying them. Regression analyses suggest that prior teaching experience slightly improves recognition accuracy, while subjective judgments of text quality were not influenced by actual or presumed authorship. These findings suggest that current written examination practices are increasingly vulnerable to undetected AI use. Both human judgment and existing AI detectors show high error rates, especially for high-quality AI-generated content. We conclude that a reconsideration of traditional assessment formats in academia is warranted. • A survey of 63 lecturers revealed that only half of the AI-generated texts could be recognized as such. • Humans recognize AI texts slightly better than AI detectors. • The higher the level of AI-generated texts, the more difficult it is to distinguish them from human texts. • Human assessment of text quality does not depend on whether the text is actually from an AI.

Ähnliche Arbeiten