GitHub Copilot: context assembly

01 · CONTEXT

An assistant that has to guess the context

The GitHub blog describes GitHub Copilot as an AI pair programmer. It entered technical preview in June 2021 and became generally available in June 2022. The model behind it is Codex, a descendant of GPT-3.

The central point of the case is context. The models fast enough for Copilot processed, at the time of the text, about 6,000 characters at a time, so not all of a person's code fits as context. The challenge, in the source's words, is deciding which data to send to the model and in what order to send it.

02 · PROMPT ASSEMBLY

The prompt is an assembly under a budget

According to the source, prompts are compilations of IDE code and relevant context sent to the model, generated by algorithms in the background at any point while typing.

1. At any point while typing, the IDE triggers the assembly of a new prompt.

2. The prompt library selects snippets of code and comments from the current file and from other sources.

3. The snippets are prioritized, filtered, and assembled into the final prompt, which must fit in the model's limited window.

4. Optimized use of caching lets the assembly run in the background without extra latency.

5. The assembled prompt goes to the model, which generates the suggestion.

Most of this work happens in what GitHub calls the prompt library, where ML experts and algorithms extract and prioritize the sources of information about the person's context. The first version considered only the file the person was working on.

03 · NEIGHBORING TABS AND FIM

Two techniques that expand the context

Neighboring tabs lets the system process all the files open in the IDE, not just the current file: it searches those files for snippets that match the code around the cursor and adds them to the prompt.

In A/B tests, a very low threshold for including a match gave the best suggestions. According to engineer Albert Ziegler, quoted in the text, including the best match found, even if not perfect, was better than including nothing.

Before FIM, only the code before the cursor, the prefix, went into the prompt. The code after the cursor, the suffix, was left out. With FIM, the prompt tells the model which part is the prefix and which is the suffix, and the suggestion becomes the code that should go between the two.

According to the source, thanks to the optimized use of caching, both techniques run in the background without extra latency.

04 · SEMANTIC RETRIEVAL

The next step, still experimental: repository embeddings

The text describes experiments with vector databases for a personalized experience in private repositories or with proprietary code. Generative code tools use embeddings to retrieve information of this kind from such a database.

1. In the proposed design, the indexer generates embeddings of all the repository's snippets, potentially billions.

2. The vectors are stored in the vector database.

3. While programming, the IDE snippets also become embeddings.

4. The vector database looks for approximate matches, not just exact ones, even with billions of stored snippets.

5. In the diagram, the search result goes on to the prompt library.

The feature is designed for enterprise customers with private repositories who would explicitly opt in to use it. The source uses the conditional, and ZenML's analysis notes that it is described as experimental, with no performance data.

05 · LESSONS

What to take to a similar design

The source shows recurring practices: accepting weak matches rather than nothing, validating each technique by A/B testing suggestion acceptance, and sustaining low latency with cache.