ChatGPT: personal assistant chatbot

01 · CONTEXT

A conversational assistant in two halves

ByteByteGo includes in its Generative AI System Design Interview course a chapter called “ChatGPT: Personal Assistant Chatbot”. Educative has, in its generative AI system design course, a text-to-text generation chapter, described as the training and deployment design of an efficient conversational AI system.

The problem can be read in two halves: how the model is trained to help the user and how it is served to many users at the same time. Both belong to the system design of machine learning.

02 · TRAINING

From the pretrained model to the aligned model

The InstructGPT paper starts from an observation: making models bigger does not, by itself, make them better at following the user's intent, and large models can generate outputs that are false, toxic, or unhelpful.

1. The starting point is an already pretrained language model.

2. Labelers demonstrate the desired behavior on the prompts, and the pretrained model is fine-tuned with supervised learning on this data.

3. Labelers compare model outputs and indicate which they prefer; these comparisons train a reward model that predicts the preferred output.

4. The reward model's output becomes a scalar reward, and the supervised policy is fine-tuned with PPO to optimize it.

5. The comparison and optimization steps can be repeated continuously; most of the comparison data comes from the supervised policies, and some comes from the PPO policies.

6. In the paper's test, outputs from the 1.3B-parameter InstructGPT were preferred over those of the 175B GPT-3, which has over 100x more parameters.

The training prompts sent to earlier versions of InstructGPT on the API Playground go through a filter that removes those containing PII.

03 · SERVING

Serving the model to many users

After training, the model needs to be served to many users at the same time. The reference design puts a balanceador in front of the inference servers and a guardrails layer between the input and the model.

1. The diagram starts with the user, who sends the message to the API entry point.

2. The diagram shows a guardrails layer between the API entry point and the balancer.

3. The diagram puts a balancer in front of the inference server.

4. The diagram connects the inference server to a store holding the conversation history.

5. In Transformers, the request states are the attention key and value tensors, the KV cache, and managing this cache is decisive for the maximum batch size.

6. The diagram ends at a metrics destination connected to the inference server.

Token-by-token generation makes the workload memory-bound, underuses the GPU's compute, and limits throughput. In the vLLM paper's example, an LLM with 13B parameters on an NVIDIA A100 GPU with 40GB of memory, roughly 65% of the memory goes to the model weights and close to 30% holds the requests' dynamic states.

04 · BATCHING

Grouping requests without making anyone wait

Throughput improves by grouping several requests into a batch. Since they share the same weights, the cost of moving weights is amortized across the requests in the batch.

1. User requests can arrive at different times.

2. A naive batching strategy makes old requests wait for new ones, or delays new ones until the old ones finish, with significant queueing delays.

3. Requests can have very different input and output lengths, and a straightforward batching technique pads inputs and outputs to equalize lengths, wasting GPU compute and memory.

4. With fine-grained batching, after each iteration the completed requests leave the batch and new ones join, and a new request waits a single iteration, not the whole batch.

5. KV cache management is decisive for the maximum batch size.

05 · DECISIONS

Points to discuss in practice

Input and output guardrails, conversation memory, and the cost per response, tied to custo of GPU, are left as discussion points for practice. This article does not claim how ChatGPT implements them.