Design a Deployment System

The ask

New software must reach every server in the fleet.

Changing all servers at once puts the whole service at risk.

Progressive deployment is the release of new software in small steps, with a check after each step before more servers change.

The system rolls the change out step by step.

It can pause the rollout, and it can reverse a finished step.

Estimate the scale

The fleet is the same 10,000 servers the monitoring system watches.

A normal wave is 10% of the fleet.

10,000 servers x 10% = 1,000 servers in a normal wave.

10,000 servers / 1,000 servers per wave = 10 waves for a full rollout.

One wave takes 5 minutes.

10 waves x 5 minutes = 50 minutes for a full rollout.

The canary wave is 1% of the fleet.

10,000 servers x 1% = 100 servers in the canary wave.

The canary wave is watched for 10 minutes before the rollout continues.

API and data

A release request names the new version and its owner.

The request also names the wave plan and the checks that gate it.

Each server reports its current version.

Each server also reports the result of its health check after a change.

The system keeps one record per release.

The record holds the state of every wave, step by step.

High-level design

A builder turns the new software into one package.

The package is stored once in a package store.

A controller holds the wave plan and orders each wave.

An agent on each server takes the orders for its server.

The agent fetches the package, installs it and restarts the service.

The agent reports the result back to the controller.

Before the rollout

The new version first runs in a safe test area.

The safe test area holds copies of real requests, never real traffic.

A package that fails there never reaches the fleet.

Each package carries a checksum from the builder.

An agent checks the checksum before it installs anything.

A package with a wrong checksum is refused.

Deep dive: the canary wave

The canary wave is the first and smallest wave.

Its 100 servers take real traffic on the new version.

Their metrics are compared with the same metrics on old servers.

Error counts get the closest watch.

An alert from the canary servers pauses the rollout at once.

The watch lasts 10 minutes before the rollout continues.

A clean canary wave earns the next and larger wave.

A bad canary wave harms only a small share of traffic.

The waves after the canary

Each wave changes its servers in the same way.

The controller waits for the wave to finish and report.

The gating checks run again after each wave.

A wave that fails its checks stops all later waves.

Servers not yet reached keep the old version and keep serving.

The fleet can therefore run two versions for a while.

New software must work beside the old version during that window.

Stopping and reversing

A rollback is a return to the version before the change.

Each server keeps the old version until the rollout completes.

A rollback therefore needs no fresh download.

The agent restarts the service on the kept old version.

Rollbacks run in waves too, starting with the newest changes.

Changes to stored data are the hard case for a rollback.

A release that reshapes stored data must keep the old shape readable.

Evaluation and trade-offs

Small steps trade speed for safety.

A full rollout takes 50 minutes, not one risky moment.

The canary wave turns a fleet-wide failure into a small one.

The cost is machinery. Waves, gates and kept old versions all need care.

The design pays that cost because a bad release is far costlier.

Scale at a glance