Design LLM Serving

The ask

Serving takes a finished, trained model as given. Nothing about the model changes at serving time. The problem is purely operational.

The job is to answer many prompts, quickly, on expensive hardware.

Any client can call the serving layer. A chat app, a support tool or an internal program all send prompts and receive streamed tokens.

Two goals pull against each other. Each user wants a fast stream. The operator wants every GPU busy, since idle GPU time is wasted money.

Estimate the scale

Demand from the chat service alone runs to 2,893,519 output tokens per second. Serving must hit that rate without wasting the fleet.

One stream alone generates about 20 output tokens per second.

A 500-token answer therefore takes 500 / 20 = 25 seconds for that stream.

Twenty tokens a second is a comfortable reading pace. The problem is not the pace. The problem is that one stream leaves the GPU almost idle.

Grouped generation is what makes the fleet affordable. The memory budget comes first, because memory decides how much grouping is possible.

Two phases of an answer

An answer is generated in two phases. Prefill is the first phase, where the model reads the whole prompt at once.

Prefill processes all prompt tokens together, in parallel. Its cost grows with prompt length, and it ends by producing the first output token.

The decode phase follows, producing one token at a time. Each step computes a single new token against everything written so far.

Prefill happens once per answer. Decode repeats for every output token, so it dominates the time of long answers.

Time to first token is set mostly by prefill. The pace after that is set by decode. The two phases need different handling, so the design treats them separately.

Memory and the key-value cache

The canon model holds 20,000,000,000 parameters at 2 bytes each.

20,000,000,000 × 2 = 40,000,000,000 bytes, which is 40 GB of weights.

The serving GPU has 80 GB of memory. The weights take 40 GB, so 80 - 40 = 40 GB stays free.

The key-value cache (KV cache) stores the values the model computed for each token it has read or written. A new token is computed against those stored values instead of against the raw text again.

Without the cache, every decode step would recompute the whole conversation. With it, each step computes one new token and appends its values.

The cache takes 0.5 MB per token of context.

A conversation of 8,000 tokens needs 8,000 × 0.5 MB = 4,000 MB, which is 4 GB.

40 / 4 = 10 conversations of that size share one GPU.

High-level design

Clients reach a load balancer, which spreads prompts across the serving fleet.

Every GPU holds a full copy of the model weights. A prompt can therefore run on any GPU, and no GPU waits on another for the model itself.

A model too large for one GPU can be split across several, at the cost of constant traffic between them.

A scheduler on each machine decides which streams share the GPU. It watches memory first, because the key-value cache sets the true capacity.

Tokens stream back to the client as they are produced.

If a GPU fails mid-answer, the conversation restarts on another GPU from the stored prompt. The user sees a pause, not a lost conversation.

Serving predictions

Batching means running many streams through one GPU together. The GPU's cost is shared across all of them, because one pass can advance many conversations at once.

A fixed group waits for a set of requests, runs them together, and waits again at the end for the slowest. Short answers sit finished while long ones complete.

Continuous batching is grouping that never closes. Streams join the running group as they arrive and leave the moment they finish, so the group is re-formed at every token step.

One GPU running 32 streams together generates 2,000 tokens per second.

2,000 / 32 = 62.5 tokens per second for each stream on average.

The same GPU now serves 32 readers at a good pace instead of one.

Admission follows memory, not stream count. A new stream joins only when its key-value cache fits in the free memory.

Prompt caching

Prompt caching stores the computation for a repeated prompt opening, so a later prompt with the same opening reuses it. The opening is usually a long shared instruction text placed before the user's own words.

Suppose 1,000 prompts share an opening of 2,000 tokens.

Without the cache, prefill computes 1,000 × 2,000 = 2,000,000 token computations for the openings alone.

With the cache, the opening is computed once and read from the cache 999 times.

A cached opening is valid only while the opening text is unchanged. Any edit makes a new cache entry, and the old one expires unused.

Prompt caching saves prefill work. It does not shrink the key-value cache a live conversation needs while decoding.

Deep dive: the memory wall

Memory, not compute, caps how many conversations one GPU serves. The weights are fixed, and every live conversation adds its key-value cache on top.

Short conversations are cheap in memory. A 2,000-token conversation needs 2,000 × 0.5 MB = 1,000 MB, which is 1 GB, so 40 of them fit in the same 40 GB.

Doubling the context window doubles each conversation's cache. At 16,000 tokens a conversation takes 8 GB, and 40 / 8 = 5 such conversations share the GPU.

When memory runs out, the scheduler can pause a stream and keep its cache, or drop the cache and recompute the conversation later. Keeping costs memory. Recomputing costs time.

Storing cache values in fewer bytes shrinks the cache per token. The saving trades a little answer quality for a lot of capacity.

Evaluation and trade-offs

Serving is measured on four numbers. Time to first token, tokens per second per stream, tokens per second per GPU, and cost per token.

Load tests replay recorded prompt traffic against a new serving setup. The replay shows the real mix of prompt lengths and answer lengths, which averages hide.

A bigger group raises throughput and can slow each stream. The right group size is the largest one that still meets the pace promised to users.

A smaller model frees memory for more conversations and answers worse. Serving choices and model choices cannot be separated, because the same 80 GB pays for both.

The serving design is a memory budget with a scheduler attached. The weights take their fixed share first, and every other choice spends what remains.

One GPU, in numbers