The Impact of a GenAI Integration on First-Year Engineering Students' Programming Outcomes

Abstract

This full-length research study compares first-year engineering student performance on a summative programming assignment before and after integrating Generative AI (GenAI) into the assignment. The code-writing capabilities of GenAI technology can potentially help introductory engineering students with programming. However, the limitations of GenAI, especially its tendency to produce hallucinations, might confuse and/or overwhelm novice programmers. Using a quasi-experimental design, we compared student performance on a final programming project from two consecutive years: one without GenAI support (2022) and one with a guided GenAI integration (2023). The assignment was such that GenAI tools were not likely to complete it correctly solely from the written instructions, and thus students were required to think critically about both programming and GenAI to develop a successful program. Student-submitted programs were evaluated in terms of program accuracy, communication, and functionality using a scoring rubric. It was anticipated that GenAI would improve program functionality, but unknown whether it would have the same positive impact on accuracy and communication based on its limitations. Results revealed a significant overall improvement in programming outcomes from 2022 to 2023, with significantly higher scores for accuracy and communication observed for the with-GenAI group than the without-GenAI group. Functionality scores remained consistently high (at ceiling). These findings suggest that a guided GenAI integration can enhance novice student programming outcomes despite its limitations. GenAI tools are expected to become increasingly valuable for introductory engineering programming instruction and for programming as a field.

Ähnliche Arbeiten