LLM inference batching on 1 GPU

01 · CONTEXT AND REQUIREMENTS

What the interview asks for

This is an interview problem that circulates in AI engineering question lists: design an inference batching system for a single GPU that accepts up to a hundred inputs per batch while users wait synchronously. Exponent's list records the topic as asked at Anthropic.

Exponent's model answer opens with requirements. Functional: the user submits an input and waits for the result, the system groups up to a hundred individual requests onto a single GPU and returns each result to whoever asked for it when it finishes. Non-functional: minimize idle time between batches, return an error when a batch fails, and scale to multiple GPUs. The core entities are Request, Batch, and Result.

02 · WHY THE BATCH MATTERS

Memory, not compute, limits throughput

According to Anyscale, LLM inference is limited by memory I/O, not by compute. That is why throughput is determined largely by the batch size that fits in the GPU's high-bandwidth memory.

The vLLM article shows where that memory goes on a 40 GB A100 serving a 13-billion-parameter model: roughly 65% for the weights, which stay static, and close to 30% for the requests' dynamic states. Those states are the attention key and value tensors, the KV cache. The rest of the memory holds other data, such as activations.

The KV cache is heavy. In the 13-billion-parameter OPT, the cache for a single token takes up 800 KB. Since OPT generates sequences of up to 2048 tokens, one request's cache can reach 1.6 GB, and even reserving all available memory for the KV cache, only a few dozen requests would fit.

Anyscale's back-of-the-envelope math points the same way: limiting sequences to 512 tokens, at most about 28 fit per batch; with 2048 tokens, the batch is limited to 7. This is an upper bound, since no room is left for intermediate computations.

03 · STATIC BATCHING

The batch that waits for the slowest

Static batching is the traditional approach: the batch size stays constant until inference finishes. It performs well when all outputs are the same size. Anyscale gives the example of a classification in which each output is a single token; with inputs of the same size too, each static batch achieves the best possible GPU utilization.

A chatbot cannot assume fixed lengths for either input or output, and variation in generation can cause huge GPU underutilization.

1. In the diagram, the client talks to the API, the API feeds the request queue, and a batch assembler hands the work to the GPU worker.

2. In static batching, the batch size stays constant until inference finishes.

3. As a result, the GPU is underutilized until the last sequence in the batch finishes generating.

4. Requests that finished before the others cannot go back to the client, and newcomers must wait for the current batch to finish entirely.

04 · CONTINUOUS BATCHING

Schedule per iteration, not per request

Continuous batching is also called dynamic batching or batching with iteration-level scheduling. According to Anyscale, the Orca paper is the first to address this problem, as far as it knows: instead of waiting for all the sequences in the batch to finish, the batch size is determined at each iteration. Orca also proposes selective batching, which applies batching only to a selected set of operations.

1. The endpoint puts newly arrived requests in the request pool, and the scheduler selects from the pool the requests that run next.

2. The scheduler calls the execution engine to run a single iteration of the model over the selected requests.

3. Because the scheduler gets a return after each iteration, it detects a request's completion and sends its tokens back to the client immediately.

4. A newly arrived request can be considered after just one model iteration, greatly reducing queueing delay.

5. When a sequence finishes, another can be inserted in its place, with higher GPU utilization than in static batching.

6. Prefill has a different computational pattern from generation and does not batch easily with it; frameworks handle this with the waiting_served_ratio hyperparameter.

In Anyscale's benchmarks, run on a single 40 GB A100 GPU with Meta's OPT-13B model, continuous batching reaches up to 23 times more throughput (with vLLM, which adds memory optimizations) and 8 times over naive batching, on both Ray Serve and Hugging Face's text-generation-inference. Continuous batching systems also improve median latency.

05 · MEMORY AND SCHEDULING

The KV cache decides how many requests fit

A request's KV cache grows and shrinks as the model generates tokens, and its length is not known in advance. Earlier systems pre-allocated a contiguous block of the request's maximum size. The vLLM paper's profile shows that only 20.4% to 38.2% of the KV cache memory held actual token states in those systems.

1. On a 40 GB A100 serving a 13-billion-parameter model, roughly 65% of memory goes to the weights and close to 30% to the requests' dynamic states.

2. PagedAttention allows the KV cache to be non-contiguous, allocated in fixed-size blocks, and allocation happens on demand rather than ahead of time.

3. According to Anyscale, the vLLM blog limits waste to less than 4%, only in the last block, and that saving turns into a larger batch.

4. When free physical blocks run out, vLLM selects sequences to evict and transfers their KV cache to the CPU.

5. In the recomputation alternative, the KV cache is recomputed when the preempted sequences are scheduled again.

6. vLLM adopts a first-come, first-served policy, and the most recent requests are the first to be preempted.

On the scheduler side, Orca defines max batch size as the largest possible number of requests in a batch. Raising it trades more throughput for more latency, with diminishing returns, and the operator can tune this knob to maximize throughput within the latency budget.

Orca's scheduler also respects memory: it selects at most max batch size requests in arrival order and reserves space for keys and values when the request is first scheduled. A naive implementation can fall into a deadlock, in which the scheduler cannot start an iteration for any request because there is no room for the next key and value.

06 · TRADE-OFFS AND WRAP-UP

What to say when closing the interview

Continuous batching is still limited by the space available on the GPU. As the system saturates, there are fewer opportunities to inject new requests immediately when they arrive. The vLLM paper, in turn, reports improving the throughput of popular LLMs by 2 to 4 times at the same latency level, compared with systems like FasterTransformer and Orca.

Anyscale treats the cost of serving as a direct consequence of how efficiently you serve at a given latency, which is why it measures throughput and latency together.