Cloudflare Jul 2020: backbone sends all traffic to Atlanta

01 · CONTEXT

A private backbone and a change to relieve congestion

Cloudflare operates a backbone: private lines between many of its data centers, used as faster and more reliable paths than those of the public Internet. The post-mortem explains that it provides better quality of service because it allows avoiding points of Internet congestion. This arrangement is part of the design of CDN and edge computing.

1. The diagram starts with the visitor and the data center that serves them.

2. When needed, traffic crosses the private backbone to another data center instead of going over the public Internet.

3. The distant data center fetches the content from the customer's origin server and responds along the same path.

On the day of the outage, according to the post, a configuration error on the backbone caused a 27-minute outage, with network traffic dropping by about 50%. The network team was handling an unrelated problem, on the stretch between Newark and Chicago, and updated the configuration of a router in Atlanta to relieve congestion. The configuration had an error that sent all backbone traffic to Atlanta, overloaded the router, and took down the locations connected to the backbone. Because of the backbone's architecture, the outage was localized to certain geographies.

02 · TIMELINE

From the lost link to full recovery

The sequence begins with a problem on the backbone link between Newark and Chicago, which led to congestion on the stretch between Atlanta and Washington. Because the backbone was congested at Atlanta, according to the post, the team decided to pull some of the backbone traffic away from Atlanta and updated the router configuration there. The times below are in UTC and come from the post's timeline.

03 · THE CONFIGURATION

A BGP policy line that advertised everything

According to the post, instead of removing Atlanta's routes from the backbone, a one-line change started leaking all BGP routes onto the backbone. The correct change would have been to disable the policy term, not the prefix list. By removing the condition from the list, the router was instructed to send all of its BGP routes to the other backbone routers, with an increased local-preference. It is an example of load balancing decided by the routing layer: the route with the highest preference wins and attracts the traffic.

1. Atlanta is congested and the team decides to pull some of the traffic away from it.

2. The one-line change makes the router advertise all of its BGP routes to the other backbone routers.

3. The routes arrive with a higher local-preference than the servers' local routes.

4. Because the higher preference wins, traffic that used to go to local servers now goes to the Atlanta servers.

04 · IMPACT

Who went down, who didn't, and how it recovered

The Atlanta router became overloaded and the locations connected to the backbone failed. The post's internal traffic dashboard shows Atlanta's CPU overloaded and the CPU of the affected data centers near zero, because they stopped handling traffic. Locations not listed as affected kept operating normally, which the post attributes to the backbone's architecture.

1. Backbone traffic is diverted to Atlanta.

2. The data centers connected to the backbone fail and stop handling traffic.

3. The locations that did not depend on the backbone keep serving normally.

4. With the Atlanta router disabled, traffic returns to normal.

5. Shortly after, a core data center that processes logs and metrics becomes congested and some logs are dropped, but the edge keeps operating.

According to the CEO, quoted by The Register, the company isolated the Atlanta router and shut down the backbone, routing traffic through transit providers. The company's own API and recursive DNS had degraded performance, and the press listed affected third parties such as Authy, Digital Ocean, Discord, Downdetector, GitLab, Medium, Patron, and Riot, among others.

05 · LESSONS

Limits so a router does not attract everything

The post announces two changes. One is a maximum prefix limit on the backbone's BGP sessions, which would have shut down the backbone at Atlanta, something the network can tolerate because it was built to work without the backbone. The other changes the local-preference of local server routes, so that a single location does not attract the traffic of the others. The post also says the company had never had a backbone outage. Limiting the reach of a failure is the subject of availability patterns.