Lessons from Real Failures

The October 2021 case

The company ran a global social platform. Its data centers were linked by a backbone network, the private high-capacity network that carries traffic between them.

Only the publicly reported facts are stated for this case.

What happened in October 2021

A configuration change was made on the backbone network. Configuration is the set of settings that tells network equipment how to route traffic.

Networks find each other through routing announcements. BGP is the protocol that carries those announcements, telling every network which paths lead where.

The change caused the routes to the company's data centers to be withdrawn. A withdrawn route is an announcement taken back, so other networks no longer knew any path in.

The data centers were cut off from each other and from the internet at once. The machines were running. Almost nothing could reach them.

Why the tools failed too

The company's services went down, which was expected once the routes vanished. The deeper problem was that its internal tools went down with them.

Those tools were how engineers inspected and repaired the network. They ran on the same network they existed to manage.

A repair tool that needs the network cannot repair the network. The people fixing the failure had to work around the loss of their own instruments.

Recovery took about six hours in total. The services, and the tools, came back as the routing was restored.

Estimate the damage in 2021

The damage is measured against a standard budget. A service that promises 99.9 percent availability allows itself about 8.8 hours of downtime a year.

This single outage lasted about six hours. That is 6 ÷ 8.8 × 100 = about 68 percent of a whole year's downtime budget, spent in one event.

The arithmetic is the lesson. One shared failure can consume most of a year's allowance at once, however good the rest of the year is.

Design lessons from 2021

First, the repair path must not depend on the thing it repairs. Management tools need a separate way in, one that still works when the main network does not.

Second, a change that can reach the whole backbone at once has no redundancy protecting against itself. Redundant hardware cannot help when one instruction reaches every copy of it.

Third, the riskiest changes deserve the slowest rollouts. A backbone configuration change should spread in stages, with an automatic way back when early stages misbehave.

Fourth, plan for the tools being down. The runbook that assumes working dashboards and working internal chat fails exactly when it is needed most.

One configuration change reached the whole backbone at once.

Routes withdrawn cut the data centers off from everything.

Tools on the same network went down with the services.

Wrap-up for 2021

The network itself was redundant. Data centers, links and machines were duplicated many times over.

The control over the network was not redundant. One configuration change, and one set of tools riding the network it managed, formed the real single point of failure.

That is the main idea. Redundancy in the system means little when the system that repairs it shares its fate.

The November 2020 case

The service was a streaming service run by a large cloud provider, and other services depended on it. Only the publicly reported facts are stated for this case.

This outage was caused by adding capacity, the standard cure for overload.

What happened in November 2020

The front-end fleet is the set of servers that first receive work and hand it onward. The provider added servers to that fleet, in one capacity addition.

The fleet's servers keep connections to each other. Each server therefore holds more connections as the fleet grows, even if user traffic stays flat.

Work on a server runs in threads. A thread is one path of execution the operating system schedules, and the operating system caps how many threads a program may have.

The addition pushed every server in the fleet past that thread limit at once. Servers could not start the threads their work needed.

The service's capacity dropped by about 90 percent at the worst point. The failure then cascaded to dependent services in the same region, and the trouble lasted several hours.

Capacity addition. The front-end fleet grew.

More fleet members meant more connections per server.

The thread limit was crossed, and servers could not take the work.

Dependent services spread the failure within one region.

Estimate the damage in 2020

Capacity fell by about 90 percent, and 100 − 90 = 10, so about 10 percent of capacity remained at the worst point.

Nine tenths of the service vanished without any server dying. The hardware was healthy. A limit, not a failure of machines, removed the capacity.

The damage stayed inside one region, which limited the blast. It also concentrated there, because the dependent services had nowhere else to go.

Why it cascaded

Dependent services treated the streaming service as a sure thing. When it faltered, their own work faltered with it, across the region.

Retries make such moments worse when they are careless. A dependent service that retries hard against a service at a tenth of its capacity adds load exactly where load is the problem.

Graceful degradation is the counterweight. A dependent that can shed the streaming feature and keep its core work alive turns a cascade into a partial outage.

Design lessons from 2020

First, limits are part of capacity. A fleet is not ready to grow until somebody has checked the ceilings that growth approaches, including the quiet ones inside the operating system.

Second, adding machines changes the load on the machines already there. Any cost that grows with fleet size, like connections held per server, must be estimated at the new size, not the old one.

Third, test at the size you intend to run. The fleet behaved at its old size for a long time. The limit only appeared at the size the addition created.

Fourth, change fleets gradually. A few servers first, watched closely, would have crossed the limit in miniature, where the lesson was cheap.

Wrap-up for 2020

Nothing in this outage was worn out or attacked. A planned improvement crossed a hidden ceiling, everywhere in the fleet at the same time.

The main idea is that scale changes the system, not just its size. Every fleet has limits that only growth can reach, and a good design goes looking for them before growth does.

Evaluate the design

The failure checklist is a short set of questions that the two outages teach, asked of a design before it ships. Each question hunts for a weakness that ordinary testing missed in a real system.

Two real outages become questions a design must answer before it ships.

Neither outage was caused by user load overwhelming a healthy system. Both were caused by the system's own operations, a configuration change and a capacity addition.

That is why the checklist exists. Load testing would not have caught either failure, because neither failure needed load.

Check the blast radius

The blast radius of a failure is how much of the system that one failure can take down with it. The first check asks how far a single mistake can reach.

Can one configuration change reach every machine at once? If it can, the change itself is a single point of failure, however redundant the machines are.

Changes should spread in stages, with a pause to watch each stage. An automatic way back matters as much as the change, because the way back is what gets used at the worst moment.

Regions and data centers should fail separately by design. A weakness shared by all of them is not divided by having many of them.

Check the repair path

The second check asks whether the tools survive the failure. Monitoring, dashboards and internal tools must not depend on the thing they monitor.

In the 2021 outage, the internal tools rode the backbone they managed. When the backbone failed, diagnosis and repair lost their instruments.

A separate management path is the fix. It can be small and plain. It only has to work when the main system does not.

Runbooks belong on the same checklist. A recovery plan stored inside the failing system, or written for tools that are down, is not a recovery plan.

Check the hidden limits

The third check hunts for ceilings. Every system has limits that everyday running never approaches, like thread counts, open connections and file handles.

A file handle is the operating system's token for one open file or connection. Like threads, handles are capped, and a program that runs out stops working in confusing ways.

The dangerous limits are the ones that grow with the fleet. In the 2020 outage, connections per server grew with every server added, until the thread limit stopped the whole fleet.

The check is arithmetic, not hope. Estimate each growing cost at the fleet size planned for next year, and compare it with the ceiling, in writing.

Capacity additions are changes, and changes are risk. Adding servers deserves the same staged rollout and watching as any code release.

Check the neighbours

The fourth check looks sideways. Which other services depend on this one, and what happens to them when it slows rather than stops?

Slow is often worse than down. A down service fails fast and triggers failover. A slow one holds its callers waiting, and the waiting spreads.

Timeouts bound the waiting, and retries must be stingy against a struggling service. Graceful degradation lets a neighbour shed the failing feature and keep its core work alive.

Dependencies inside one region deserve special suspicion. The 2020 cascade stayed regional because the service was regional, and everything near it shared its fate.

Use the checklist

Run the four checks out loud in every design review. Blast radius, repair path, hidden limits, neighbours.

Each check is cheap. Each one is a question, a short estimate and an honest answer, long before any of them becomes an outage.

Both systems were well built for the failures they expected. They fell to the failures their own operations created, and those are the failures a checklist can catch.

The checklist