Evaluating Large Language Models for Kazakh Question Generation: A Comparative Study of ChatGPT and Gemma

Abstract

This study compares two well-known big language models, ChatGPT and Gemma, to assess the quality of autonomously generated multiple-choice questions in Kazakh. We created questions using eight different configurations that varied by model, prompt language (English vs. Kazakh), source text language, and generation technique (direct vs. translation), all based on the historical book The Nomads by Ilyas Esenberlin. A 5-point Likert scale measuring fluency, clarity, and syntax was used to evaluate 48 questions from 577 Kazakh-speaking respondents in three surveys. The findings show that while ChatGPT slightly outperformed Gemma (3.95 vs. 3.86, <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$p=0.15$</tex>), Kazakh-prompted questions scored much higher than Englishprompted questions (4.00 vs. 3.81, <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathrm{p} \leq 0.01$</tex>). Translation was inferior to direct generation (3.96 vs. 3.85, <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$p=0.10$</tex>). These results have significant implications for the development of educational technologies in the Kazakh language as well as lowresource language NLP applications.

Ähnliche Arbeiten