Retrieval-Augmented Generation

01 · CONTEXT

Why a model needs a knowledge base

To be useful in a specific context, a language model usually needs background knowledge; legal analysis bots, for example, need to know a vast range of past cases. One way to do this is retrieval-augmented generation, or RAG, which retrieves relevant information from a knowledge base and appends it to the user's prompt.

If the knowledge base is smaller than 200,000 tokens (about 500 pages), you can include all of it in the prompt, without RAG, and prompt caching makes that approach faster and cheaper. For larger knowledge bases, which do not fit in the context window, RAG is the typical solution.

The topic appears as a chapter title on the public page of ByteByteGo's generative AI system design course.

02 · TRADITIONAL RAG

Chunk, embed, and search

Preprocessing splits the knowledge base into smaller chunks, usually at most a few hundred tokens; an embedding model converts the chunks into vectors that encode meaning; and the vectors are stored in a vector database that enables semantic similarity search. At query time, the most relevant chunks are added to the prompt sent to the generator model.

Embeddings capture semantic relationships well, but can miss important exact matches. BM25 is a lexical alternative: it is a ranking function that uses lexical matching to find exact words or phrases and is particularly effective for queries with unique identifiers or technical terms. For the query Error code TS-999 in a technical support knowledge base, an embedding model may find content about error codes in general and miss the exact match, whereas BM25 looks for that specific text sequence.

Hybrid search combines the two: BM25 finds the best chunks by exact match, embeddings find the best by semantic similarity, the results are combined and deduplicated with rank fusion, and the top K chunks go into the prompt.

Traditional RAG has a limitation: it destroys context when encoding information, which often makes the system fail to retrieve what is relevant. In the text's example, one chunk says only that revenue grew 3% over the previous quarter; on its own, it does not specify which company it is talking about, which makes it hard to retrieve the right information or use it well.

03 · INDEXING

Contextual Retrieval: giving each chunk context before indexing

Contextual Retrieval solves the problem by prepending to each chunk an explanatory context specific to it before generating the embedding. A model writes that context, receiving a prompt to give a concise, chunk-specific context, explaining it using the context of the whole document. The resulting text is usually 50-100 tokens and is prepended to the chunk before the embedding and before creating the BM25 index.

1. The diagram starts with the document corpus, the knowledge base that will be indexed.

2. The corpus is split into smaller text chunks, usually at most a few hundred tokens.

3. For each chunk, a model writes a short context that situates it within the whole document, and that context is prepended to the chunk before the embedding and the lexical index.

4. The contextualized chunks become embeddings, vectors that encode meaning, and are stored in a vector database that enables semantic similarity search.

5. The same contextualized text feeds the lexical index, which later finds exact words and phrases.

04 · QUERY

Hybrid search, fusion, and reranking

Reranking is a common filtering technique to ensure that only the most relevant chunks reach the model. It improves answers and reduces cost and latency because the model processes less information.

1. The diagram starts with the question the user sends to the API.

2. The same question goes to the embedding search, which finds chunks by semantic similarity, and to the lexical search, which finds chunks by exact match.

3. The results of the two searches are combined and deduplicated with rank fusion techniques.

4. In Anthropic's experiment, the initial retrieval brought back the top 150 candidate chunks, and a reranking model gave each one a relevance score to pick the top 20.

5. The top K chunks go into the generator model's prompt, which produces the final result.

6. The diagram ends with the answer returned to the user.

05 · RESULTS AND DECISIONS

What the combination delivered and what to tune

The text's metric is 1 minus recall@20: the percentage of relevant documents that are not retrieved among the top 20 chunks. The numbers below are averages across all the knowledge domains tested, with the best embedding configuration (Gemini Text 004) and retrieving the top 20 chunks.

Among the implementation considerations, the choice of chunk size, boundary, and overlap can affect retrieval performance. The authors tested 5, 10, and 20 chunks in context and 20 was the best-performing option among them; embeddings with BM25 were better than embeddings alone; and the text recommends always running evaluations (evals).