The November 2020 Outage

The case

The service was a streaming service run by a large cloud provider, and other services depended on it. Only the publicly reported facts are used here.

The strangest fact comes first. This outage was caused by adding capacity, the standard cure for overload.

What happened

The front-end fleet is the set of servers that first receive work and hand it onward. The provider added servers to that fleet, in one capacity addition.

The fleet's servers keep connections to each other. Each server therefore holds more connections as the fleet grows, even if user traffic stays flat.

Work on a server runs in threads. A thread is one path of execution the operating system schedules, and the operating system caps how many threads a program may have.

The addition pushed every server in the fleet past that thread limit at once. Servers could not start the threads their work needed.

The service's capacity dropped by about 90 percent at the worst point. The failure then cascaded to dependent services in the same region, and the trouble lasted several hours.

Estimate the damage

Capacity fell by about 90 percent, and 100 − 90 = 10, so about 10 percent of capacity remained at the worst point.

Nine tenths of the service vanished without any server dying. The hardware was healthy. A limit, not a failure of machines, removed the capacity.

The damage stayed inside one region, which limited the blast. It also concentrated there, because the dependent services had nowhere else to go.

Why it cascaded

Dependent services treated the streaming service as a sure thing. When it faltered, their own work faltered with it, across the region.

Retries make such moments worse when they are careless. A dependent service that retries hard against a service at a tenth of its capacity adds load exactly where load is the problem.

Graceful degradation is the counterweight. A dependent that can shed the streaming feature and keep its core work alive turns a cascade into a partial outage.

The design lessons

First, limits are part of capacity. A fleet is not ready to grow until somebody has checked the ceilings that growth approaches, including the quiet ones inside the operating system.

Second, adding machines changes the load on the machines already there. Any cost that grows with fleet size, like connections held per server, must be estimated at the new size, not the old one.

Third, test at the size you intend to run. The fleet behaved at its old size for a long time. The limit only appeared at the size the addition created.

Fourth, change fleets gradually. A few servers first, watched closely, would have crossed the limit in miniature, where the lesson was cheap.

Wrap-up

Nothing in this outage was worn out or attacked. A planned improvement crossed a hidden ceiling, everywhere in the fleet at the same time.

The main idea is that scale changes the system, not just its size. Every fleet has limits that only growth can reach, and a good design goes looking for them before growth does.

How adding capacity caused an outage