Design an AI Code Assistant

The ask

Code completion is a suggestion of the next code, shown while a developer types.

The assistant also answers chat questions about code. Chat covers explanations, fixes and larger edits.

The two jobs have very different speed needs. A completion must feel instant. A chat answer may take a few seconds.

Private code raises a second need. Code sent for help must stay protected.

Estimate the scale

The service has 10,000,000 developers, as a design figure.

Each developer asks for 100 completions a day, as a design figure.

10,000,000 x 100 = 1,000,000,000 completions a day.

1,000,000,000 / 86,400 gives 11,574 completions per second, rounded.

Work follows working hours across regions. At three times the average rate, peak load is about 34,722 completions per second.

Each completion carries about 4,000 tokens of context, as a design figure.

1,000,000,000 x 4,000 = 4,000,000,000,000 context tokens processed a day. Context handling is a first-class cost.

Data and features

Training data comes from code shared for training. Licences decide what may be used.

Private customer code is not training data by default. A customer can choose to share code for training.

The next lines a developer writes form a natural label. A completion example pairs earlier code with those lines.

Accepted and dismissed suggestions are serving signals. An accepted suggestion is a weak label, not proof of correctness.

A request carries facts about the editing moment. The facts are the language, the file name, the imports and the recent edits.

High-level design

A small fast model serves completions. A large model serves chat and edits.

The editor sends a completion request to a nearby service. Short distance protects the budget.

A context builder gathers the open file and its neighbours. Neighbours are imported files, nearby project files and recent files.

Chat requests follow a separate path to the large model. The chat path can take longer and return longer answers.

Both paths share account checks and logging. Both paths record latency for every request.

A slow completion is dropped. A late suggestion is worse than no suggestion.

Training the models

The small model trains on the same code data. Fewer parameters make each answer faster and cheaper.

Fine-tuning teaches the small model the completion format. The answer is code only, with no explanation.

Fine-tuning teaches the large model to follow edit instructions. Training examples pair an instruction with the changed code.

Held-out projects test both models before release. Training projects never appear among the test projects.

New language versions enter the training data slowly. A sudden style change in suggestions signals drift.

Serving completions and chat

The completion budget is about 200 milliseconds, end to end.

Context building and sending take about 50 milliseconds, as a planning figure.

200 - 50 = 150 milliseconds remain for the model and the return trip.

The small model therefore stays small, even when a larger model scores better offline.

The large model has a looser budget on the chat path. Chat answers stream back token by token.

App servers on both paths hold no request state. Any server can take the next request.

Deep dive: building the context

Context is built from the open file and its neighbours. The open file matters most, because typing happens there.

A token is about 4 characters, by the standard estimate.

4,000 x 4 = about 16,000 characters of code in one request.

Text near the cursor comes first. Imports and definitions used by the open file come next.

Older and unrelated files are left out. The context window is limited, so gathering is a choice, not a dump.

Files marked as secrets are never sent. File names that suggest keys or passwords are excluded by rule.

The editor sends a fresh copy of the code with each request. A copy taken after a pause in typing avoids stale text.

Deep dive: edits that must still work

A chat edit changes working code. The assistant shows the change before the change is applied.

The editor can run the project tests after an edit. A failing test marks the edit as unsafe.

An unsafe edit is easy to undo. The original code stays one step away.

Explanations stay short on the edit path. The developer asked for a change, not an essay.

A bigger model would complete better and miss the budget. The budget, not model pride, drives the two-speed split.

Evaluation and trade-offs

Models are scored on held-out code before release. Scores cover correctness of completions and quality of edits.

Serving reports track the acceptance rate for suggestions. Serving reports also track the share of completions shown inside the budget.

Acceptance alone can mislead. Short safe suggestions are easy to accept and add little value.

Longer context improves relevance. Longer context also costs time inside a fixed budget.

The small model protects speed. The large model protects depth on hard questions.

The code assistant in numbers