Incident · Staff
Cloudflare Jun 2022: BGP policy out of order in 19 data centers
Real incident. Key points: network: DNS and BGP, and progressive deployment.
Short answer
Real incident. Key points: network: DNS and BGP, and progressive deployment. On June 21, 2022, a network configuration change caused an outage that affected traffic in 19 Cloudflare data centers; it began at 06:27 UTC. These locations account for a significant share of global traffic, and the company says the error was its own and not an attack.
01 · CONTEXT
A network incident in the busiest locations
On June 21, 2022, a network configuration change caused an outage that affected traffic in 19 Cloudflare data centers; it began at 06:27 UTC. These locations account for a significant share of global traffic, and the company says the error was its own and not an attack.
The change was part of a long-running project to increase resilience in the busiest locations. They were converted to an architecture called internally Multi-Colo PoP (MCP), designed as a Clos network with an extra routing layer, the spines, which forms a mesh of connections.
The mesh makes it possible to disable and re-enable parts of the internal network for maintenance or to deal with a problem. Because these locations carry a significant share of the traffic, a problem there can have a very wide impact.
Traffic reaches the spine layer, the extra routing layer that forms a mesh of connections inside the data center.
This mesh makes it possible to disable and re-enable parts of the internal network for maintenance or to deal with a problem.
The internal load balancer Multimog forwards requests between the MCP servers.
The servers communicate with each other and with customer origins through the site-local prefixes.
02 · BGP
How BGP decides what to advertise
To be reachable on the internet, a network like Cloudflare's uses the BGP protocol. In it, operators define policies that decide which prefixes (sets of adjacent IP addresses) are advertised to peers or accepted from them.
The components of a policy, called terms, are evaluated in sequence. A change can make a previously advertised prefix stop being advertised, which the post calls withdrawn.
03 · THE CHANGE
A change that looked identical
The goal was to standardize the BGP communities attached to a subset of the advertised prefixes, adding informational communities to the site-local prefixes. These prefixes let machines communicate with each other and connect to customer origins.
The change followed the company's procedure: a Change Request ticket with a dry-run and staged rollout, and peer review by several engineers. Even so, the stages were not small enough to catch the error before it hit all the spines.
On the routers, the change was harmless and only added information to the advertisements. On the spines, the diff showed reordered terms: 4-ADV-SITE-LOCALS and 6-ADV-SITE-LOCALS moved from the top to the bottom of the policy, after the REJECT-THE-REST term, an explicit reject. The reordering made the company withdraw a critical subset of prefixes.
On the spines, the reject term now comes before the terms that advertise the site-local prefixes.
Cloudflare stops advertising the site-local prefixes, loses direct access to the affected locations, and its servers lose the ability to reach the origins.
Without these prefixes, Multimog stops working, because it can no longer forward requests between servers.
The smaller clusters receive the same amount of traffic as the larger ones and become overloaded.
04 · IMPACT
From the first location to the last rollback
The rollout was staged. The first locations were not affected because they used the old architecture, and the incident began when the change reached the spines of the MCP locations.
| Time (UTC) | What happened |
|---|---|
| 03:56 | The change is deployed to the first location, affecting none, since they used the old architecture. |
| 06:17 | The change reaches the busiest locations, but not the MCP locations. |
| 06:27 | The rollout reaches the MCP locations and the change is applied to the spines; the incident begins. |
| 06:58 | The root cause is found and the rollback begins; the first data center comes back up. |
| 07:42 | The last rollback is completed, delayed by engineers undoing each other's changes. |
| 08:00 | The incident is closed. |
The affected locations make up only 4% of the network, but the incident affected 50% of total requests.
05 · REMEDIATION
What the company intends to change
The post points to a process gap: the staging policy included only one MCP data center in the final stage. Because of this, change procedures and automation need to include MCP-specific tests and deploys.
| Frente | What the post proposes |
|---|---|
| Processo | Procedures and automation with MCP-specific tests and deploys. |
| Arquitetura | Redesign the policy that caused the incorrect advertisement to avoid unintended reordering. |
| Automation | Enforce a better staging policy for network changes and offer automatic commit-confirm rollback. |
Frequently asked questions
What is Cloudflare Jun 2022: BGP policy out of order in 19 data centers?
Real incident. Key points: network: DNS and BGP, and progressive deployment.
What does the Cloudflare Jun 2022: BGP policy out of order in 19 data centers design cover?
It is organized in these sections: A network incident in the busiest locations, How BGP decides what to advertise, A change that looked identical, From the first location to the last rollback, and What the company intends to change.
How long is Cloudflare Jun 2022: BGP policy out of order in 19 data centers and what level is it?
It takes about 15 minutes to read and is rated Staff level.
Now design it yourself
Run this case as an interview in the System Design school: you make the decisions, the tutor challenges them and you see how your design scores.