The Failure Checklist

Evaluate the design

Two real outages become questions a design must answer before it ships.

Neither outage was caused by user load overwhelming a healthy system. Both were caused by the system's own operations, a configuration change and a capacity addition.

That is why the checklist exists. Load testing would not have caught either failure, because neither failure needed load.

Check the blast radius

The blast radius of a failure is how much of the system that one failure can take down with it. The first check asks how far a single mistake can reach.

Can one configuration change reach every machine at once? If it can, the change itself is a single point of failure, however redundant the machines are.

Changes should spread in stages, with a pause to watch each stage. An automatic way back matters as much as the change, because the way back is what gets used at the worst moment.

Regions and data centers should fail separately by design. A weakness shared by all of them is not divided by having many of them.

Check the repair path

The second check asks whether the tools survive the failure. Monitoring, dashboards and internal tools must not depend on the thing they monitor.

In the 2021 outage, the internal tools rode the backbone they managed. When the backbone failed, diagnosis and repair lost their instruments.

A separate management path is the fix. It can be small and plain. It only has to work when the main system does not.

Runbooks belong on the same checklist. A recovery plan stored inside the failing system, or written for tools that are down, is not a recovery plan.

Check the hidden limits

The third check hunts for ceilings. Every system has limits that everyday running never approaches, like thread counts, open connections and file handles.

A file handle is the operating system's token for one open file or connection. Like threads, handles are capped, and a program that runs out stops working in confusing ways.

The dangerous limits are the ones that grow with the fleet. In the 2020 outage, connections per server grew with every server added, until the thread limit stopped the whole fleet.

The check is arithmetic, not hope. Estimate each growing cost at the fleet size planned for next year, and compare it with the ceiling, in writing.

Capacity additions are changes, and changes are risk. Adding servers deserves the same staged rollout and watching as any code release.

Check the neighbours

The fourth check looks sideways. Which other services depend on this one, and what happens to them when it slows rather than stops?

Slow is often worse than down. A down service fails fast and triggers failover. A slow one holds its callers waiting, and the waiting spreads.

Timeouts bound the waiting, and retries must be stingy against a struggling service. Graceful degradation lets a neighbour shed the failing feature and keep its core work alive.

Dependencies inside one region deserve special suspicion. The 2020 cascade stayed regional because the service was regional, and everything near it shared its fate.

Use the checklist

Run the four checks out loud in every design review. Blast radius, repair path, hidden limits, neighbours.

Each check is cheap. Each one is a question, a short estimate and an honest answer, long before any of them becomes an outage.

Both systems were well built for the failures they expected. They fell to the failures their own operations created, and those are the failures a checklist can catch.

The checklist