Diagnosing the Origin of Machine-Generated Arabic Texts through Self-Stylistic Analysis: Testing the Capability of Large Language Models

Abstract

With the increasing use of large language models in text generation, there arises a need to explore their intrinsic capability to diagnose the origin of texts generated by other models. This study investigates the diagnostic capability of four free versions of large language models—ChatGPT-4.5, Claude, Copilot, and Gemini—in identifying the origin (human or machine-generated) of 24 Arabic texts. It relies on a novel methodology based on self-stylistic analysis of Arabic texts generated by these models in the field of literature. This descriptive and exploratory study aims to uncover the stylistic features that large language models themselves associate with machine authorship, employing qualitative stylistic analysis. The significance of the study lies in its methodology, which offers a new perspective on the capability of artificial intelligence to diagnose Arabic machine-generated texts through stylistic analysis of language produced by other models. It also identifies Arabic stylistic features used as indicators of AI generation without comparing them to human-authored texts, and establishes a preliminary reference framework for researchers studying stylistic traits of Arabic machine writing. This framework may later inform the development of criteria for stylistic classification in Arabic AI-generated texts. The study found variation in the models’ diagnostic abilities. ChatGPT-4.5 and Gemini were the most accurate in identifying machine-generated texts, followed by Claude, while Copilot ranked last. This suggests that the models’ diagnosis of Arabic machine writing relies on stylistic criteria that are not universally agreed upon. The study also categorized the results of stylistic analysis into formal, syntactic and organizational structure, lexical, rhetorical, discursive, and cognitive levels. These are the features that large language models use to diagnose Arabic texts generated by AI.

Ähnliche Arbeiten