LLM assistant with RAG (support and code)

01 · CONTEXT AND REQUIREMENTS

Why an assistant needs RAG

A support assistant needs to know the specific business it operates in. One technique for giving the model this knowledge is RAG: the system retrieves relevant information from the knowledge base and attaches it to the user's prompt.

In the abstract of the paper that named the technique, providing provenance for the model's decisions and updating its knowledge appear as open problems. Parametric-only models need more training to change behavior, whereas a model with non-parametric memory has its knowledge updated easily.

Before building the infrastructure, it is worth asking whether it is necessary. Anthropic notes that, for a knowledge base under 200,000 tokens (about 500 pages), you can include the entire base in the prompt, without RAG.

02 · OVERVIEW

The path of a question

The knowledge base is prepared before the questions: the corpus is broken into small chunks, an embedding model converts each chunk into a vector that encodes meaning, and the vectors are stored in a vector database that allows semantic similarity search.

1. In the diagram, the user reaches the API gateway, which forwards the request to the assistant orchestrator.

2. At query time, the vector database is used to find the most relevant chunks by semantic similarity to the question.

3. The most relevant chunks are added to the prompt sent to the generative model.

4. In the diagram, the safety filter sits between the orchestrator and the LLM service.

5. In the diagram, the orchestrator also sends the records of each interaction to metrics and evaluation collection.

03 · RETRIEVAL

Retrieving well is half the work

Vector search captures semantic relationships, but embedding models can miss exact matches. BM25 is a ranking function that uses lexical matching to find exact words or phrases and is especially effective on queries with unique identifiers or technical terms.

Hybrid retrieval uses both searches: BM25 finds the best chunks by exact match, embeddings find the best by semantic similarity, and the results are combined and deduplicated with rank fusion techniques.

Another problem is lost context: traditional RAG strips out context when encoding information. Anthropic's proposal, called Contextual Retrieval, prepends to each chunk an explanatory, chunk-specific context before the embedding.

Reranking is a filtering technique so that only the most relevant chunks reach the model; it improves answers and reduces cost and latency because the model processes less information. On the other hand, there is a trade-off between reranking more chunks, for better performance, and fewer, for lower latency and cost.

The same balance applies to the number of chunks in the prompt: putting more chunks in the context window increases the chance of including the relevant information, but too much information can distract the model.

04 · LLM SERVICE

The model is the expensive resource

A system design interview prep guide recommends understanding how batch inference works, how GPU clusters are managed, what RAG involves, and how rate limiting works for LLM APIs.

1. In the diagram, the orchestrator's requests go through the rate limiter and wait in the request queue.

2. Serving LLMs at high throughput requires batching enough requests together; in the diagram, the inference servers consume the queue.

3. The KV cache memory of each request is huge and grows and shrinks dynamically.

4. Poorly managed, this memory can be wasted through fragmentation and redundant duplication, which limits the batch size.

05 · UPDATES AND QUALITY

Keeping knowledge current and measuring quality

In the RAG paper, the maximum inner product search (MIPS) problem can be solved approximately in sublinear time, and the authors build the MIPS index with FAISS and a Hierarchical Navigable Small World-type approximation for fast retrieval.

1. In the diagram, changes in the sources enter the change queue.

2. In the diagram, the indexing worker consumes the queue and writes the chunks to the new index.

3. In the diagram, the retriever queries the active index.

4. In the RAG paper, the model's knowledge can be updated by swapping the non-parametric memory, the index, whereas parametric-only models need more training.

On quality, Anthropic's source lists among the implementation considerations always running evals, and recommends experimenting with configurations on your specific use case to find the balance.