Not All Chatbots Teach: Evidence for Pedagogical Design in AI-Assisted Technical Education

Abstract

As generative AI tools like ChatGPT become embedded in technical education, a critical challenge emerges: how can we ensure these tools foster learning rather than bypass it? This study provides empirical evidence that pedagogical design, not merely model access, determines the educational value of AI assistants. We developed a freely available custom conversational AI tool that embeds metacognitive scaffolding through structured prompts grounded in the Feynman Technique and learning science literature. In a quasi-experimental study within an undergraduate data structures course (N = 36), students using this structured AI assistant significantly outperformed peers using the same interface configured as a minimally prompted ChatGPT wrapper (92.7 vs. 74.3, p <.001). Gains were especially strong in abstraction, technical justification, and documentation, which are skills critical across software engineering, IT, and cybersecurity. These findings underscore a key insight: AI-integrated learning environments must be intentionally designed to prompt reflection, prediction, and explanation. By aligning AI interactions with evidence-based pedagogy, our framework demonstrates how to develop conceptual understanding, reduce automation bias, and support equitable learning outcomes as AI reshapes computing education.

Ähnliche Arbeiten