Anthropic Contextual Retrieval

01 · THE PROBLEM

Why traditional RAG fails to find the right information

To be useful in a specific context, an AI model usually needs access to background knowledge. The usual solution is RAG: retrieve relevant information from a knowledge base and append it to the user's prompt.

Anthropic's article points out the limit of this approach: traditional RAG strips out context when encoding information, and this often causes the system to fail to retrieve what was in the knowledge base.

Before building any pipeline, it is worth remembering the simplest alternative: if the knowledge base is under 200 thousand tokens, the article says you can put all of it in the prompt, without RAG.

02 · TRADITIONAL RAG

Embeddings, lexical search, and fusion

Traditional RAG starts by breaking the knowledge base into chunks of at most a few hundred tokens. An embedding model converts each chunk into a vector that encodes meaning, and the vectors are stored in a vector database that supports semantic similarity search.

Embeddings capture semantic relationships, but they can miss exact matches. That is why lexical search comes in: BM25 is a ranking function based on matching words and phrases, especially effective for queries with unique identifiers or technical terms. In a search for a specific error code, an embedding might find content about error codes in general and miss the exact chunk.

Next, the results of the two searches are combined and deduplicated with rank fusion techniques, and the top K chunks are added to the prompt to generate the answer.

03 · THE IDEA

Giving each chunk back the context it lost

The flaw is in the chunk itself. In an example from the article, a chunk on its own does not say which company or which period it refers to, which makes it harder to retrieve the right information and use it well.

The method, called Contextual Retrieval, has two sub-techniques: Contextual Embeddings and Contextual BM25. It prepends chunk-specific context to each chunk before generating the embeddings and before building the BM25 index.

In the article's example, the transformed chunk begins by saying it comes from an SEC filing on ACME's performance in the second quarter of 2023.

The article notes that manually annotating thousands or even millions of chunks would be too much work, so it uses a model. The prompt supplies the whole document and the chunk to be situated and asks for a concise context that explains the chunk in light of the document. The result, usually 50 to 100 tokens, is prepended to the chunk before the embeddings and the BM25 index.

The article also records what did not work: adding generic document summaries to the chunks gave very limited gains.

04 · COST AND CACHING

Indexing with prompt caching

Generating a context per chunk has a cost: the prompt supplies the whole document along with each chunk. The diagram shows the indexing path and where prompt caching comes in.

1. The knowledge base is broken into short chunks, and an isolated chunk may not say which company or which period it refers to.

2. For each chunk, the context generator sends a model the whole document and the chunk, and gets back a concise context that situates the chunk in the document.

3. The whole document is loaded into the prompt cache once and then only referenced, instead of being resent for every chunk.

4. The context is prepended to the chunk, and the resulting text feeds both the embedding model and the lexical index.

5. Both indexes now store the already contextualized chunk.

The article says prompt caching cuts latency by more than 2x and cost by up to 90%. With the text's assumptions (800-token chunks, 8k-token documents, 50 tokens of instruction, and 100 tokens of context per chunk), the one-time cost of generating the contextualized chunks is 1.02 dollars per million document tokens.

05 · RESULTS

What each layer gains

The article's metric is 1 minus recall@20: the fraction of relevant documents that do not appear among the top 20 retrieved chunks.

The article reports that contextualizing improved performance across every embedding and source combination evaluated, that Voyage and Gemini had the best embeddings among those tested, and that passing 20 chunks to the model was more effective than passing 10 or 5.

The article's summary is that the method can reduce retrieval failures by 49% and, combined with reranking, by 67%.

06 · RERANKING AND QUERY

From index to answer, with a final filter

The last step is reranking. Initial retrieval brings in the potentially relevant candidates (in the experiment, the top 150); a reranking model scores each chunk and selects the best K (20 in the experiment). The tests used Cohere's reranker.

1. The user's question reaches the search service.

2. The service queries the vector index, by semantic similarity, and the lexical index, by word matching.

3. The results of the two searches are combined and deduplicated with rank fusion techniques.

4. A reranking model scores each candidate; in the experiment, it started from 150 candidates and kept 20.

5. The selected chunks are added to the prompt of the generator model, which writes the answer.

Reranking is not free: since it adds a step at runtime, it adds a bit of latency. There is a trade-off between reranking more chunks, for better performance, and fewer, for lower latency and cost.

The authors themselves leave tuning to whoever deploys it: they recommend experimenting with configurations in your use case, suggest testing the number of chunks, and note that domain-specific contextualization prompts may give even better results.