Design ChatGPT

The ask

The service is a general chat assistant. A user types a question or a request. The service returns a written answer.

A prompt is the text sent to the model for one answer. In a conversation, the prompt holds the earlier messages plus the new message.

A token is the unit the model reads and writes. Text is split into tokens before the model sees it. A token is about 4 characters of English text.

A language model is a model trained on text to predict the next token. Predicting one token at a time, over and over, produces whole sentences and paragraphs.

The context window is the most text the model can consider for one answer. The window holds the conversation so far plus the new message. Its size in the design is 8,000 tokens.

Estimate the scale

The design target is 500,000,000 prompts a day. The average prompt brings 500 tokens in and takes 500 tokens out.

500,000,000 / 86,400 = 5,787 prompts per second on average.

500,000,000 prompts × 500 output tokens = 250,000,000,000 output tokens a day.

250,000,000,000 / 86,400 = 2,893,519 output tokens per second.

One serving GPU, working on grouped requests, generates 2,000 tokens per second.

2,893,519 / 2,000 = 1,447 GPUs for output alone.

Reading the prompts costs compute too, and peaks sit above the average. The fleet plan therefore starts at 1,447 GPUs and adds headroom.

Data and features

Training data is text at very large scale. Books, articles, manuals and web pages supply the raw material, gathered under the service's data rules.

Raw text is cleaned before training. Duplicates are removed. Broken and low-quality text is dropped, because the model learns whatever patterns the data contains.

The main training needs no human labels. The text supplies its own answer, since the task is to predict the next token and the real next token is already there.

The input for each example is a stretch of tokens. No hand-built features are added. The model learns patterns directly from the sequence.

A held-out set of text is kept aside for checking quality. It never enters training, so the check stays honest.

High-level design

The client sends a message to an app server. The app server identifies the user and loads the conversation.

The app server builds the prompt from the stored conversation and the new message. It sends the prompt to a model fleet, a pool of GPU machines that each hold a copy of the model.

The model generates the answer token by token. Tokens stream back through the app server to the client, so reading starts while generation continues.

The finished exchange is stored in a conversation store. The store is a database keyed by conversation, and it grows with messages, not with model work.

Checks sit on both sides of the model. Requests that break the service rules are refused before generation. Answers are checked before delivery.

Training the model

The model family is the transformer. A transformer reads the whole prompt at once and weighs which earlier tokens matter for each next token.

That design is good at using long stretches of context. Its cost is heavy compute, and the cost grows as the context grows.

Training teaches the model to predict the next token across the training text. Each pass adjusts the model a little, and passes repeat until quality stops improving.

The design's model has 20,000,000,000 parameters. A parameter is one learned number inside the model.

Stored at 2 bytes each, the model takes 20,000,000,000 × 2 = 40,000,000,000 bytes, which is 40 GB.

Fine-tuning is a second, smaller training stage on curated examples. The examples show the behaviour the service wants, such as following instructions and answering helpfully.

Human raters compare pairs of answers during fine-tuning. Their preferences become training examples, so the model learns which answer people judge better.

Training runs on a large GPU cluster for weeks. Serving then copies the finished model to every machine in the fleet.

Serving predictions

Serving a prompt means running the trained model to produce its answer. That run is called inference, the model's prediction at serving time.

Requests are grouped so each GPU works on many conversations at once. A GPU idling on one short answer wastes most of its capacity.

The first token matters most for how fast the service feels. Streaming keeps the wait honest, since the user watches the answer grow instead of facing a blank screen.

Long answers cost more than short ones. Cost is counted per output token, so pricing and limits are set in tokens, not in messages.

When the fleet is full, new prompts wait in a queue. The queue protects running answers from being slowed by late arrivals.

Deep dive: long conversations

A conversation grows with every exchange. After many turns, the stored history passes 8,000 tokens and no longer fits the window.

Only the window is sent to the model. Anything outside it cannot shape the answer, however important it was earlier.

One fix trims the oldest messages. Trimming is simple and cheap, and it forgets early facts without warning.

A second fix summarises the older part of the conversation. The summary takes few tokens, and the recent messages stay in full.

Cost grows with the window too. Every new token is weighed against the whole context in the window, so long conversations are the expensive ones.

Evaluation and trade-offs

Offline evaluation uses fixed sets of prompts with known good answers. Scores cover correctness, helpfulness and refusal of harmful requests.

Human raters grade samples of real answers. Their grades catch failures that automatic scores miss, such as confident and fluent mistakes.

Online experiments release a new model to a small share of users first. Answer quality, latency and cost per answer decide whether the release grows.

A larger model answers better and costs more per token. A smaller model is cheaper and faster, and weaker on hard questions.

The fleet can route easy prompts to the small model and hard prompts to the large one. Routing keeps the average cost down without giving up quality where it matters.

The design therefore spends its money where generation happens. Storage and app servers are ordinary. The GPU fleet is the system.

The scale in one view