Enhancing Academic Chatbot Accuracy With Retrieval-Augmented Generation in Higher Education

Abstract

Chatbot systems based on Natural Language Processing (NLP) are increasingly used to support information services, including in academic environments. In this context, students and prospective students often need quick access to information such as class schedules, registration procedures, and academic regulations. This study develops an academic chatbot using the Retrieval-Augmented Generation (RAG) method, which retrieves information directly from official institutional documents. The model combines a retriever for document search and a generator to produce responses. The dataset used includes internal academic documents such as course schedules and campus information. The system was evaluated using two metrics: Mean Reciprocal Rank (MRR) and Retrieval-Augmented Generation Assessment Score (RAGAS). The average RAGAS score, calculated from four main metrics (faithfulness, answer relevancy, context precision, and context recall), is 84%. For MRR-based evaluation, the average score from MRR, semantic similarity, answer relevancy, and faithfulness is 82%. These results show that the chatbot is capable of providing relevant responses that align well with institutional context.

Ähnliche Arbeiten