CATS-RAG: Contextual Augmented Triplet Synthesis for RAG in Technical QA

초록

Large language models (LLMs) are widely used for building question-answering (QA) systems, with retrieval augmented generation (RAG) commonly applied to improve the accuracy of the answers. However, in specific domains, achieving high performance often requires fine-tuning components within the RAG pipeline, such as the retriever and generator, because prompt or index engineering alone may not sufficiently capture domainspecific knowledge. Moreover, obtaining document-question-answer triplets for such tuning is particularly challenging in technical domains. This paper presents contextual augmented triplet synthesis for RAG in technical QA (CATS-RAG), a framework designed to expand domain-relevant data and improve RAG accuracy in technical domains with limited available data. CATS-RAG includes two core components: QA datasets generation and the fine-tuning of the retriever and generator components within the RAG. For data generation, a chain-ofthought prompting approach enables LLMs to generate triplets solely from provided documents. In the finetuning phase, by using highly similar documents from retrieved sets and probabilistic omission of golden documents, which act as hard distractor, improves answer robustness even though irrelevant documents are retrieved. Experiments on TechQA and Microsoft QA show that CATS-RAG consistently improves both retrieval and generation. On average, CATS-RAG increases retriever performance by 4.3% on TechQA and 6.5% on Microsoft QA, and generator performance improves by 1.7% and 11.9%, respectively. These results demonstrate that CATS-RAG provides reliable performance gains in specialized QA settings with limited supervision.

키워드