Concepts
Retrieval-augmented generation (RAG)
Intuition
Think of two ways to answer an exam question: from memory, using only what you memorized, or with a library beside you to consult before writing the answer. Someone who answers only from memory cannot say where the information came from and, to learn a new fact, has to study everything again. Someone who consults the library has the books they consulted in plain view and can swap an outdated book for a newer one without relearning anything.
How it works
The paper calls the knowledge stored in a language model's weights parametric memory, and an external index, which can be inspected and swapped, non-parametric memory; RAG combines the two. The retriever is a Dense Passage Retriever (DPR), a bi-encoder: one BERT-base encoder turns the query into a vector, another does the same for each document, and relevance is the inner product of the two, which reduces finding the top-k documents to a maximum inner product search, solved approximately with a FAISS index. The generator is BART-large, a sequence-to-sequence model that receives the input concatenated with a retrieved passage and produces the output text. Retrieval is treated as a latent variable: end-to-end training minimizes the marginal negative log-likelihood of the target output over the retrieved documents, with no labels saying which document was the correct one. In fine-tuning, the query encoder and the generator are updated, while the document encoder and the index stay fixed (the authors report that also updating the document encoder was not necessary); since part of the knowledge lives in the index, the paper states that it can be replaced without touching the rest of the model. There are two formulations: in RAG-Sequence the same retrieved document is used for the entire output sequence, and the per-document scores are combined, weighted by the retriever's probability; in RAG-Token a different document can be used for each token, so the answer can combine parts of different passages. In classification, where the output is a single-token label, the two are equivalent.
In practice
To test swapping the index, the authors compared a December 2016 Wikipedia index with the December 2018 one on questions about 82 world leaders who changed between the dates: with a consistent index and time period, accuracy was 70% (2016) and 68% (2018), and with a mismatched index it dropped to 12% and 4%, so the model's knowledge followed the index it was given. On Natural Questions, RAG-Sequence reached 44.5 exact match and RAG-Token 44.1, versus 41.5 for DPR and 34.5 for T5-11B. The answer also does not depend only on what was retrieved: on Natural Questions, RAG got 11.8% right when the answer was in none of the retrieved documents, a case in which an extractive model would score 0%. The results show that the best retriever depends on the task: BM25, which retrieves by word overlap, outperformed the learned retriever (DPR) on fact verification (FEVER), although it worsened performance on open-domain question answering.
Use it in a design
Practice the cases that rely on this concept in the System Design school.