Evaluating LLM Engines for TA Chatbots

Abstract

This full research paper investigates the application of AI-powered teaching assistant chatbots using seven prominent Large Language Models (LLMs). The aim is to assess their effectiveness in addressing course-specific academic queries with accuracy, depth, and pedagogical value. We developed a comprehensive evaluation framework comprising diverse questions spanning multiple subjects, cognitive levels, and complexity. The evaluation methodology combined an LLM-as-a-Judge approach, expert human reviews, and integrity stress testing. Furthermore, we propose a composite metric integrating explanation clarity, pedagogical relevance, and ethical consideration to holistically assess each model's performance. Our initial findings reveal notable performance variations among the models, with some excelling in clarity and adaptability while others display gaps in depth and ethical response handling. These insights are valuable for educators seeking to deploy AI-driven support tools effectively, helping to optimize student engagement and reduce instructional burdens. Looking ahead, this research provides a strong foundation for expanding LLM-based teaching assistant systems beyond Computer Science to broader educational contexts, aiming for enhanced scalability and learner-centered outcomes.

Ähnliche Arbeiten