Perplexity: answer engine with more than 20 models

01 · CONTEXT

One answer engine, many models

Perplexity is an AI search engine. According to the NVIDIA article that serves as the main source, the inference team serves more than 20 models at the same time, including Llama 3.1 variants such as 8B, 70B, and 405B. The engine handles more than 435 million queries per month, and each query generates several inference requests.

The case brings together three design questions: which model serves each request, how to batch requests to make use of the GPU without breaking the response deadline, and how much each query costs. The following sections follow that order.

02 · INTENT ROUTING

Classify the request and pick the pod

To connect each request to the right model, Perplexity uses smaller classifiers that help determine the user's intent. The detected tasks, such as text completion, are routed to specific models deployed on GPU pods.

Each pod has one or more H100 GPUs and is managed by a Triton Inference Server instance. The pods live in a Kubernetes cluster and operate under strict cost and interactivity SLAs. There is also an in-house front-end scheduler, which does load balancing routing traffic to the appropriate pod based on load and usage.

1. The user makes a query; each query generates several inference requests behind the scenes.

2. Smaller classifiers determine the user's intent, and the detected task defines which model the request goes to.

3. The front-end scheduler, built in-house, routes traffic to the appropriate pod based on load and usage.

4. The pods host models of different sizes; each pod has one or more GPUs and is managed by a Triton instance.

5. Triton provides GPU utilization metrics to the scheduler, which supports scaling deployments and GPUs up or down according to request volume.

The choice of scheduler algorithm matters: it can affect inter-token latency. The source shows a comparison between round-robin, least requests, and power of two random choices in an internal benchmark with a moderate load of 16 QPS.

03 · BATCHING BY SLA

Each use case has its own deadline

To define the SLA for each use case, the team runs comprehensive A/B tests, evaluating configurations and the impact on the user experience. The goal is to maximize GPU utilization while meeting the target SLA of each case, and improving batching within those SLAs is what optimizes the cost of serving.

Small embedding models, used in real-time retrieval, are usually hidden from the user, inside a larger flow; their focus is the lowest possible latency, which is why the configurations use small batches. Because they take up little memory, several run together on a single H100 GPU to keep utilization high. For the user-facing models, the team does a deeper analysis, with its own metrics.

04 · EMBEDDINGS: IVY, TULIP, AND ROSE

A dedicated stack for search

Besides LLMs, the busca depends on embedding and ranking models, which help identify the most relevant results for a query. Perplexity describes two traffic patterns: batch embedding, for building or reindexing the corpus while maximizing throughput and reducing cost, and online embedding, which converts a short query into a vector while minimizing latency.

Because the models are small, much of the implementation is shared with the LLM code: batch embedding resembles prefill, which is compute-bound, and the online one, which runs few tokens, resembles decode, which is memory-bound. According to Perplexity, this yields high throughput in batch with little additional engineering work and low latency online.

1. Perplexity's services call Ivy, an HTTP gateway written in Rust.

2. Ivy templates the input, splits it into batches, and translates the request into a gRPC protocol of its own for the downstream servers.

3. Because payloads vary, sending each request to a single replica can unbalance the load; that is why Ivy splits large requests into chunks and balances them across replicas.

4. Tulip receives the gRPC requests, handles scheduling and batching, sends the batches to ROSE, and returns the responses.

5. ROSE runs on the GPUs: it implements the forward passes and the management of CUDA graphs, reusing code from the LLMs and overlapping CPU and GPU work.

Tulip's scheduling is simple: requests accumulate while it dispatches work or waits for results, and sequences are picked in order of arrival. In these small models, latency is roughly proportional to the number of tokens, not the number of sequences. Once the batch saturates the GPU, about 512 tokens on a model under one billion parameters, gathering more sequences does not improve efficiency.

05 · PARALLELISM AND COST

How much it costs to serve a query

For user-facing LLMs, the strict SLAs led the team to increase tensor parallelism to four and eight GPUs, which reduces the cost for highly latency-sensitive requests within a fixed GPU budget. Data or pipeline parallelism was useful for maximizing throughput where latency is less critical. In an example from the source, splitting Llama 8B across four Hopper GPUs reduces the relative cost per million tokens by up to 3x for latency-sensitive requests.

The decision to host the models depends on serving them at a lower cost, still within the strict SLAs, than using third-party provider APIs. As an example, the team estimated saving about US$ 1 million per year by serving, on NVIDIA GPUs in the cloud, the models that power the Related-Questions feature.

On the evolution roadmap is disaggregated serving, which separates the prefill and decode phases of an LLM onto separate GPUs. According to the source, the technique increases system throughput within the SLAs, which translates into a lower cost per token, and Perplexity is working with the Triton engineering team to deploy it.