Shortwave: email assistant with RAG

01 · CONTEXT

An email assistant with RAG

Shortwave, an email app, built an AI assistant with retrieval-augmented generation (RAG). Each user's inbox gathers correspondence, calendar invites, receipts, notifications, and newsletters. The company wanted a knowledge base built from it, but earlier attempts with traditional search had been disappointing, and LLMs could change that.

For users, the company describes the AI as a person sitting next to them, with perfect knowledge of their email history. This simple mental model helps predict what the assistant can do and how to use it better.

02 · ARCHITECTURE

One large prompt, four steps

The central principle is that how to answer a question should be handled by the LLM itself; the system's job is to find the right data. According to the team, long chains of LLM calls introduced data loss and errors at each stage, with low-quality answers. Instead, the answer comes from a single LLM that receives all the context in one large prompt, which depends on a very capable model and a large context window.

To organize this, Shortwave created the Tool abstraction. Each type of data, such as calendar, email history, or draft, comes from a different tool, and each tool decides internally which specific data goes into the final prompt.

When answering, the assistant follows four steps: tool selection, retrieval of the tools' data, answering the question, and post-processing.

1. The user's question goes immediately to an LLM. This call is not for answering, but for determining the types of data that need to be gathered.

2. The selection can return no tools, one, or several, and the prompt for this step carries details such as the app's current state.

3. The chosen tools retrieve their data in parallel. The most complex is EmailHistory, which gives the assistant knowledge of the entire email history.

4. With the data gathered, the system builds a prompt with the original question and the context brought by the tools and hands it to an LLM.

5. Here you need to decide how to allocate tokens among the tools, because EmailHistory usually produces far more content than fits in the prompt.

6. Finally, post-processing converts the output into rich text, adds source citations, and suggests actions to the user.

03 · CONTEXT BUDGET

More data than fits in the prompt

AI Search looks for relevant emails across the user's entire history and orders them by usefulness, so that smart trade-offs can be made when answering. The results often reach a thousand or more, and they must be cut down to a few dozen, and fast.

In the answer step, this becomes a context budget question: you must decide how to allocate tokens among the tools, because EmailHistory usually produces far more content than fits in the prompt. The post mentions heuristics for making these trade-offs.

04 · AI SEARCH

From the question to the email ranking

The first step is query reformulation. It uses an LLM to rewrite a query that lacks the necessary context so that it makes sense on its own; the post gives the example of a follow-up question in a conversation, which makes no sense as a standalone question. To do this, it considers what is visible on the screen that the user might be referring to, including the chat history, the visible conversation, and the current draft.

Then, features extracted from the query feed a traditional search and a vector search. At ingestion, each email becomes an embedding via an open model, computed on in-house servers with GPU acceleration, and is stored in a vector database with a separate space per user.

1. The first step of AI Search is to reformulate the query with an LLM, so that it makes sense on its own.

2. A set of fast LLMs extracts features from the reformulated query, each with a confidence score. With them, traditional full-text or metadata search may be the best path.

3. The reformulated query is used to search for semantically similar emails in the vector database, with a separate space per user. Features such as the date range restrict the search to a smaller subset.

4. The first re-ranking phase uses fast heuristics to cut the large list, ordering by semantic similarity and applying rules over the features, such as a Gaussian filter on dates.

5. In the second phase, an open cross-encoder on in-house GPUs receives the reformulated query and the pre-selected fragments. It is smarter than the heuristics, but slow, and so it only processes the top-ranked fragments.

6. The heuristics are applied again to the cross-encoder scores, and the final ordered list of fragments goes back to the assistant.

05 · TRADE-OFFS

Two re-ranking phases and the total cost

The team optimized the system to answer almost all questions in 3-5 seconds, and this end-to-end latency includes several LLM calls, ElasticSearch queries, and calls to open models. The system uses concurrency, streaming, and pipelining and a lot of computing power: robust servers and clusters of top-end GPUs.