Can GPT-4o Reason about Language? A Syntax Challenge

Abstract

This study investigates the capacity of GPT-4o, a multimodal large language model, to engage in linguistic analysis through a syntax exam designed to probe foundational concepts such as constituency, ambiguity, and recursion in both English and Korean. While often providing accurate definitions and fluent responses, the model struggles to apply syntactic principles consistently, especially in tasks requiring structural reasoning and tree diagram generation. The model’s frequent misinterpretations and incoherent analyses within and across tasks reveal a reliance on pattern recognition and heuristics rather than a systematic grasp of hierarchical structures and fundamental linguistic reasoning. These findings point to the limitations of current large language models in performing metalinguistic analyses, exposing a gap between surface-level performance and genuine metalinguistic competence, which in turn presupposes linguistic competence. By examining GPT-4o’s responses across a range of syntactic challenges, this study emphasizes the need for more rigorous evaluation frameworks that go beyond surface-level fluency to assess models’ capacity for human-like linguistic reasoning and analysis.

Ähnliche Arbeiten