The October 2021 Outage

The case

The company ran a global social platform. Its data centers were linked by a backbone network, the private high-capacity network that carries traffic between them.

Only the publicly reported facts are stated for this case.

What happened

A configuration change was made on the backbone network. Configuration is the set of settings that tells network equipment how to route traffic.

Networks find each other through routing announcements. BGP is the protocol that carries those announcements, telling every network which paths lead where.

The change caused the routes to the company's data centers to be withdrawn. A withdrawn route is an announcement taken back, so other networks no longer knew any path in.

The data centers were cut off from each other and from the internet at once. The machines were running. Almost nothing could reach them.

Why the tools failed too

The company's services went down, which was expected once the routes vanished. The deeper problem was that its internal tools went down with them.

Those tools were how engineers inspected and repaired the network. They ran on the same network they existed to manage.

A repair tool that needs the network cannot repair the network. The people fixing the failure had to work around the loss of their own instruments.

Recovery took about six hours in total. The services, and the tools, came back as the routing was restored.

Estimate the damage

The damage is measured against a standard budget. A service that promises 99.9 percent availability allows itself about 8.8 hours of downtime a year.

This single outage lasted about six hours. That is 6 ÷ 8.8 × 100 = about 68 percent of a whole year's downtime budget, spent in one event.

The arithmetic is the lesson. One shared failure can consume most of a year's allowance at once, however good the rest of the year is.

The design lessons

First, the repair path must not depend on the thing it repairs. Management tools need a separate way in, one that still works when the main network does not.

Second, a change that can reach the whole backbone at once has no redundancy protecting against itself. Redundant hardware cannot help when one instruction reaches every copy of it.

Third, the riskiest changes deserve the slowest rollouts. A backbone configuration change should spread in stages, with an automatic way back when early stages misbehave.

Fourth, plan for the tools being down. The runbook that assumes working dashboards and working internal chat fails exactly when it is needed most.

Wrap-up

The network itself was redundant. Data centers, links and machines were duplicated many times over.

The control over the network was not redundant. One configuration change, and one set of tools riding the network it managed, formed the real single point of failure.

That is the main idea. Redundancy in the system means little when the system that repairs it shares its fate.

Cause and effect