A Multi-Account Statistical Evaluation of ChatGPT Proficiency in the Kurdish Sorani Language

Abstract

This research analyzes the strengths and weaknesses of ChatGPT in responding to questions posed in the Kurdish language, specifically its Sorani dialect, by evaluating its responses to a structured dataset of 50 multiple-choice questions across multiple topics such as language, history, culture, and general knowledge. Using four independent user accounts, each subjected to ten repeated testing cycles, the research assesses accuracy, consistency, and variation influenced by account identity, session timing, and model behavior. This study evaluates the multilingual capabilities of ChatGPT by comparing its performance in Kurdish (Sorani) and Arabic languages. The research establishes a framework to examine how artificial intelligence chatbots, such as ChatGPT, function as applied tools for language understanding and educational use. The analysis demonstrates that ChatGPT achieved an overall average accuracy rate of approximately 70%, indicating satisfactory performance in multilingual contexts. However, significant variations were observed across different user accounts, suggesting that factors such as user profile and temporal dynamics can considerably influence output consistency. The comparative findings highlight the developmental challenges in Arabic and Kurdish language processing, emphasizing the need for further refinement of ChatGPT’s linguistic performance and its effective integration into academic and technological applications. While ChatGPT exhibited proficiency in answering general knowledge questions, it demonstrated a limited understanding of specialized topics in Kurdish, particularly classical literature and historical content. The research presents the strengths and limitations of ChatGPT for under-resourced languages and provides feedback to developers, educators, and researchers. Observing patterns in accuracy, question difficulty, and error behavior, this research also contributes to ongoing efforts toward improving the linguistic and cultural adequacy of AI models for under-resourced languages.

Ähnliche Arbeiten