Do Large Language Models Produce Texts With “Human‐Like” Lexical Diversity? Evidence From Four ChatGPT Models

Abstract

ABSTRACT The degree to which large language models produce writing that is truly human‐like remains unclear despite the extensive empirical attention that this question has received. The present study addresses this question from the perspective of lexical diversity. Specifically, the study investigates patterns of lexical diversity in texts generated by four ChatGPT models (ChatGPT‐3.5, ChatGPT‐4, ChatGPT‐o4 mini, and ChatGPT‐4.5) in comparison with texts written by L1 and L2 English participants ( n = 240) across four education levels. Six dimensions of lexical diversity were measured in each text: volume, abundance, variety‐repetition, evenness, disparity, and dispersion. Results from one‐way multivariate analyses of variance, one‐way analyses of variance, and support vector machines (SVM) revealed that the ChatGPT‐generated texts differed significantly from human‐written texts for each variable, with ChatGPT‐o4 mini and ChatGPT‐4.5 differing the most. Within these two groups, ChatGPT‐4.5 demonstrated higher levels of lexical diversity than older models despite producing fewer tokens. The human writers’ lexical diversity did not differ across subgroups (i.e., education, language status). Altogether, the results indicate that ChatGPT models do not produce human‐like responses to the target argumentative essay prompt in relation to lexical diversity, and the newer models produce less human‐like texts than older models. We discuss the implications of these results from ChatGPT for large language models in the context of language pedagogy and related applications.

Ähnliche Arbeiten