Classic design · Senior
Code assistant (Copilot-style)
Classic interview design. Key points: FIM, repository context under a token budget, autocomplete latency, and acceptance rate.
Short answer
Classic interview design. Key points: FIM, repository context under a token budget, autocomplete latency, and acceptance rate. GitHub Copilot is the example used in this case. According to the source, Codex, a descendant of GPT-3, was the model chosen to power it, and the product went from technical preview in June 2021 to general availability in June 2022.
01 · CONTEXT AND REQUIREMENTS
What a code assistant needs to do
GitHub Copilot is the example used in this case. According to the source, Codex, a descendant of GPT-3, was the model chosen to power it, and the product went from technical preview in June 2021 to general availability in June 2022. Designing an assistant like this is an exercise in ML system design.
The goal stated by the team is to deliver context-relevant responses with low latency. The challenge, in its words, is deciding not only which data to send to the model, but also in what order and in what form to insert it.
02 · SERVICE DESIGN
From the keyboard to the model and back to telemetry
The design below is a teaching reading for interviews, built from the points the source describes.
The IDE is what triggers the process: according to the source, prompts are generated in the background at any point while typing.
In this design, the gateway is the service's entry point and receives the requests coming from the IDE.
Context assembly selects snippets from the current file and other sources and then prioritizes, filters, and assembles them into the final prompt.
The cache comes in so that assembly runs in the background with no added latency.
The model server is where the suggestion is generated; in Copilot, the model that would power it was Codex, a descendant of GPT-3.
Telemetry receives from the IDE the accepted or ignored suggestions; the source evaluates the context techniques precisely by the acceptance of the completions shown, in A/B tests.
The source says prompts are generated in the background at any point while typing and that, thanks to optimized cache use, neighboring tabs and FIM work with no added latency. It is a typical case of taking care of latency on the request path.
03 · CONTEXT ON A BUDGET
Not all the code fits in the prompt
At the time of the text, transformers fast enough for Copilot processed about 6,000 characters at a time. This limit means that not all of the person's code can become context.
To choose what goes in, a series of algorithms selects code snippets and comments from the current file and other sources; these snippets are then prioritized, filtered, and assembled into the final prompt.
The first version considered only the file open in the IDE as context.
Neighboring tabs start searching the other open files for snippets similar to the code around the cursor and add them to the prompt.
Retrieval across the repository is described as an experiment: embeddings of the snippets would sit in a vector database, with approximate matching in real time.
Assembly prioritizes, filters, and assembles the final prompt with what the sources deliver.
With FIM, the prompt tells the model which part is the prefix and which is the suffix, and the suggestion covers the span that should sit between the two.
| Technique | What the source reports |
|---|---|
| Current file | The first version considered only the file open in the IDE. |
| Neighboring tabs | Finds similar snippets in open files and adds them to the prompt; 5% relative increase in acceptance. |
| FIM | Tells the model the prefix and the suffix; 10% more completions accepted in the A/B tests. |
| Embeddings | In experimentation: repository embeddings in a vector database, with approximate matching. |
When developing neighboring tabs, the team concluded that a very low bar for including a match yielded the best suggestions. Including the best match found, even an imperfect one, was better than including nothing.
04 · FIM AND ACCEPTANCE
Prefix, suffix, and the metric that matters
Before FIM, only the code before the cursor went into the prompt. GitHub calls the code before the cursor the prefix and the code after it the suffix; with FIM, it is possible to tell the model which part of the prompt is which.
The source measures these gains by suggestion acceptance: neighboring tabs raised relative acceptance by 5%, and FIM yielded 10% more accepted completions in the A/B tests.
05 · REPOSITORY RETRIEVAL
The next step: embeddings and a vector database
The source describes, as an experiment, the use of vector databases for a personalized experience in private repositories. The algorithms would create embeddings of all the snippets in the repository, potentially billions, and perform approximate matching in real time against snippets from the IDE.
The feature was designed for enterprise customers who explicitly opted in to use it.
06 · POINTS TO DISCUSS
Trade-offs the interviewer tends to explore
Points to discuss: context size versus latency, the cost of querying more sources versus suggestion quality, and how to measure acceptance without delaying typing.
It is also worth asking what happens when the model is slow: a late suggestion loses its value, and the client may prefer to show nothing.
Frequently asked questions
What is Code assistant (Copilot-style)?
Classic interview design. Key points: FIM, repository context under a token budget, autocomplete latency, and acceptance rate.
What does the Code assistant (Copilot-style) design cover?
It is organized in these sections: What a code assistant needs to do, From the keyboard to the model and back to telemetry, Not all the code fits in the prompt, Prefix, suffix, and the metric that matters, The next step: embeddings and a vector database, and Trade-offs the interviewer tends to explore.
How long is Code assistant (Copilot-style) and what level is it?
It takes about 16 minutes to read and is rated Senior level.
Now design it yourself
Run this case as an interview in the System Design school: you make the decisions, the tutor challenges them and you see how your design scores.