Manufacturing Domain QA with Integrated Term Enhanced RAG

Abstract

Large Language Models (LLMs) have demonstrated powerful capabilities, yet LLMs face issues like hallucination in certain domain-specific areas. Consequently, an increasing number of domain-specific models are emerging. The current paradigm for domain-specific models involves training with domain data, followed by the employment of Retrieval-Augmented Generation (RAG) to mitigate hallucination issues. However, in precision-critical domains such as manufacturing, if the knowledge documents are of low quality or contain noise, the context retrieved through simple semantic matching by RAG may not necessarily benefit model output. Additionally, there can be issues like getting "lost in the middle" due to irrelevant or excessive context. To overcome this, we introduce the Integrated Term Enhancement Methodology (ITEM). Inspired by Chinese educational methods focused on key term elucidation, ITEM extracts and explains critical terms precisely from knowledge documents to form a comprehensive Term Dictionary for retrieving terms and explanations to enhance query capabilities. This methodology refines query responses by providing more accurate and contextually relevant information. To assess ITEM's effectiveness, we utilize the Chinese Mould Manufacturing Dataset (CMMD) and Contextualized Adaptive Response Assessment (CARA) metric method. Our experiment demonstrates that ITEM significantly outperforms existing retrieval enhancement Dense Retrievers by over 17.0% in accuracy while requiring only 80% of their token length. Moreover, the accuracy of our method exceeded that of GPT-4 by 5.0%. This advancement represents a significant leap in context-specific retrieval in LLMs, especially beneficial for specialized domains. The results underscore ITEM's potential as a transformative method in the field, offering new perspectives on integrating domain-specific knowledge into LLMs.

Ähnliche Arbeiten