Can AI Code? A Head-to-Head Showdown Between Large Language Models and Student Programmers in C

Abstract

The capabilities of large language models (LLMs) have raised questions about their potential to perform programming tasks at a level comparable to humans. This study, conducted with a unique dataset of C programming tasks and students’ solutions collected from an educational automated programming task assessment system, presents an evaluation of four state-of-the-art LLMs, i.e., ChatGPT, Claude, Gemini, and Llama, against first-year computer science students in C programming. We tested the LLMs on 14 progressively complex C programming tasks. We analyzed their performance based on success rate, number of trials, and specific programming skills required to solve a task. Our results indicate significant performance variations across LLMs. Claude consistently outperformed other models, requiring the fewest attempts to generate correct solutions. ChatGPT followed closely, excelling in structured problems but occasionally struggling with iterative refinement. Llama performed moderately well, often generating correct solutions but requiring more iterations, specifically on more complex tasks. Gemini, however, exhibited the highest variability and the lowest success rate, frequently failing to resolve errors even with multiple prompts. While LLMs handled programming concepts effectively, they struggled with tasks requiring logical reasoning and addressing specific constraints. In contrast, students showed resilience in iterative learning, whereas LLMs often repeated mistakes or failed to adapt to nuanced feedback. These findings reveal AI’s strengths and limitations in programming education and highlight areas for improving LLMs in human-AI collaboration.

Ähnliche Arbeiten