EP017: RAG Gives AI a Library Card

This paper introduces Retrieval-Augmented Generation (RAG), a method designed to enhance pre-trained language models by giving them access to explicit, non-parametric memory. While standard large language models store knowledge implicitly in their parameters, they often struggle with accessing precise information and can produce "hallucinations".To address this, the authors propose a hybrid architecture that combines two components trained end-to-end:• A Retriever (Non-Parametric Memory): A dense vector index of Wikipedia accessed by a neural retriever (DPR).• A Generator (Parametric Memory): A pre-trained sequence-to-sequence model (BART) that conditions its output on the retrieved documents.The paper presents two model variations: RAG-Sequence, which uses the same retrieved passage to generate a full sequence, and RAG-Token, which can utilize different passages for each token.Key Findings:• State-of-the-Art Performance: RAG models set new state-of-the-art results on open-domain question answering tasks (such as Natural Questions and WebQuestions), outperforming both parametric-only baselines and specialized extract-and-retrieve architectures.• Improved Generation: For knowledge-intensive generation tasks, such as Jeopardy question generation, RAG produces responses that are more factual, specific, and diverse than standard baseline models like BART.• Updatable Knowledge: A significant advantage of RAG is the ability to update the model's "world knowledge" simply by replacing the non-parametric document index, removing the need to re-train the entire model as facts change.