LLM service (ChatGPT)

01 · CONTEXT

A chat on top of a model

The exercise asks for the system that serves a chat assistant: the user sends natural-language prompts and receives streamed responses from a large language model. Conversations are saved, and the user can return to an old chat and continue where they left off.

The prompt treats the model as a black box called by the system, without training it or operating its internals. The scope is text input and output only. Editing or branching existing messages is out of scope.

The two functional requirements are sending a prompt and receiving an AI-generated response, and viewing past chats to resume the conversation with the earlier context in the prompt.

02 · REQUIREMENTS

Latency, GPU, and cost

The interesting parts of the problem, according to the prompt, are streaming tokens fast, scheduling limited GPU capacity, and keeping cost under control as conversations grow.

For scale, the prompt starts from a little over two hundred million daily active users. Its estimate yields about 20 thousand prompts per second at peak and about 120 thousand responses being generated at the same time.

03 · DESIGN

From prompt to token

1. In the design, the chat client sends the prompt through the API entry point.

2. In the design, the conversation service reads and updates the conversation store, which is where the chat context comes from.

3. In the design, the conversation service puts the request on a generation queue, which sits between it and the GPUs.

4. In the design, the GPU workers pull requests from the queue and call the model as a black box.

5. In the design, the generated tokens return to the conversation service and from there go on to the client.

The design separates the conversation service from the GPU workers with a generation queue. The tokens return in streaming to the client. The prompt says that latency to the first token matters more than total completion time.

The deep-dive questions in the next section cover how this path delivers tokens, survives reconnections, and uses the GPUs.

04 · DEEP DIVES

The remaining questions

The prompt lists five deep-dive questions, which serve as a study roadmap for this case.

According to the guide, the patterns behind these LLM prompts are the same as in familiar distributed systems: queues, batching, load balancing, and asynchronous processing. The guide also recommends understanding how rate limiting works for LLM APIs.

1. In the design, a user's request arrives through the API entry point.

2. In the design, the entry point forwards the request to admission control before any queue.

3. In the design, admission control checks the user's usage counters.

4. In the design, only the accepted request enters the generation queue.

5. In the design, the GPU workers consume the queue and send metrics to monitoring.

The second design puts admission control before the queue, so that each user's consumption is measured before the request competes for GPU. This is a starting point for the question about heavy users and paid plans, not a ready-made answer.

The question about cost in long conversations connects to the discipline of cost and FinOps, and the one about scheduling the GPU, to the ML service.