Real architecture · Staff
Character.AI: inference optimization
Real production architecture. Key points: MQA, local/global attention, KV shared across layers, cross-turn cache, and int8.
Short answer
Real production architecture. Key points: MQA, local/global attention, KV shared across layers, cross-turn cache, and int8. Character.AI designs the model architecture, the inference stack, and the product end to end. The June 2024 post that guides this case reports that the platform was serving more than 20,000 inference queries per second.
01 · CONTEXT
A conversation platform at scale
Character.AI designs the model architecture, the inference stack, and the product end to end. The June 2024 post that guides this case reports that the platform was serving more than 20,000 inference queries per second.
A case study published by AMD speaks of more than a billion queries per day and states that GPU infrastructure became the company's largest custo operating expense.
02 · THE BOTTLENECK
The key-value cache limits throughput
According to the post, the main bottleneck of LLM inference throughput is the size of the attention key-value (KV) cache. That size sets the maximum batch that fits on the GPU and also dominates the I/O cost of the attention layers.
In this reference design, with no cache between turns, the user sends the message to a load balancer, which picks an inference server.
The attention key-value cache lives on the GPU and, according to the post, determines the maximum batch size that fits on it.
Without reusing the cache between turns, the KV would have to be refilled on every message, something the company describes as prohibitively expensive as dialogues grow.
The company claims to have reduced the KV cache by more than 20X without regressing quality and that, with these techniques, GPU memory is no longer a bottleneck for large batches.
03 · MODEL ARCHITECTURE
Techniques to shrink the KV cache
The table summarizes the techniques described in the post, one per row.
| Technique | What the post describes | Stated effect |
|---|---|---|
| Multi-Query Attention | Adopted in all attention layers | Reduces the KV cache by 8X compared to Grouped-Query Attention |
| Hybrid attention | Local attention interleaved with global; only 1 in every 6 layers is global | Horizon of 1024 in most layers with no significant impact on metrics |
| KV shared across layers | KV cache shared across neighboring layers | Reduces the KV cache by a factor of 2-3x |
In the global attention layers, the KV cache of several global layers is shared across blocks, because those layers dominate the KV cache size in long-context uses. The post states that sharing KV across layers does not regress quality.
04 · CACHE BETWEEN TURNS
Reusing the KV across messages in the same dialogue
Most conversations are long: the average dialogue history per message is 180 messages. Refilling the KV cache every turn would be prohibitively expensive as dialogues grow.
The company's answer is a system of cache between turns. For each prefix already processed and each message generated, the KV values stay in host memory and are retrieved in future queries. They are organized in an LRU cache with a tree structure, similar to RadixAttention, and indexed by a rolling hash of the prefix tokens: on each query the hash of each context prefix is computed and the cache of the longest match is retrieved, which allows reusing even partially identical messages.
Queries from the same dialogue are routed to the same server through sticky sessions.
For each new query, the server computes a rolling hash of each context prefix.
The cache in host memory is queried and the KV of the longest prefix match is retrieved, even with partially identical messages.
For each processed prefix and each generated message, the KV is stored in host memory for future queries.
With this system, the company reports a 95% cache hit rate.
At the fleet level, queries from the same dialogue go to the same server via sticky sessions. Because the KV cache is small, each server can keep thousands of dialogues at the same time, and the company reports a 95% cache hit rate.
05 · QUANTIZATION AND COST
End-to-end int8 and the cost result
The company uses int8 quantization on weights, activations, and the attention KV cache. Instead of quantizing after training, it trains the models natively in int8, which eliminates the risk of a mismatch between training and serving.
The post states that serving cost has fallen by a factor of 33 since late 2022 and that serving the same traffic with leading commercial APIs would cost at least 13.5X more. Both are numbers reported by the company itself.
06 · HARDWARE AND ORCHESTRATION
Another lever: GPUs, vLLM, and parallelism
According to the AMD study, the company worked with DigitalOcean to deploy a new inference environment on AMD Instinct GPUs. It uses the open vLLM framework for serving, and a production deployment described in the study was built around the Qwen3-235B Instruct FP8 model.
The managed Kubernetes platform handles networking, gateway, and internal cluster routing.
Serving uses the open vLLM framework.
The GPUs combine data, tensor, and expert parallelism.
A company executive states that AMD Instinct GPUs enabled a 2X improvement in inference throughput in production and cut the cost per token by 50%.
Frequently asked questions
What is Character.AI: inference optimization?
Real production architecture. Key points: MQA, local/global attention, KV shared across layers, cross-turn cache, and int8.
What does the Character.AI: inference optimization design cover?
It is organized in these sections: A conversation platform at scale, The key-value cache limits throughput, Techniques to shrink the KV cache, Reusing the KV across messages in the same dialogue, End-to-end int8 and the cost result, and Another lever: GPUs, vLLM, and parallelism.
How long is Character.AI: inference optimization and what level is it?
It takes about 18 minutes to read and is rated Staff level.
Now design it yourself
Run this case as an interview in the System Design school: you make the decisions, the tutor challenges them and you see how your design scores.