Facebook: BGP and DNS outage

01 · CONTEXT

One backbone, many data centers

In the engineering post about Facebook's global outage, the origin is described as a routine maintenance job. A command was issued with the intent of assessing the availability of global backbone capacity, and it ended up taking down all the connections in that network, which disconnected the data centers from each other. The company's systems were designed to audit commands like this, but a bug in the audit tool kept the command from being stopped in time.

1. In normal operation, the data centers talk to each other over the backbone.

2. The maintenance command takes down the backbone's connections, and the data centers are left disconnected.

02 · DNS AND BGP

The health check that erased the server from the map

The authoritative name servers sit in smaller facilities and have their addresses advertised to the rest of the internet via BGP. To operate reliably, these servers disable the BGP advertisements when they cannot talk to the data centers, since that indicates an unhealthy network connection. With the entire backbone out of service, all of them declared themselves unhealthy and withdrew the advertisements. The result was that the DNS servers became unreachable even though they were running, and the rest of the internet could no longer find the services. Here global load balancing and routing failed because of a protection rule: a health check that was correct for each node produced a collective failure.

1. The name server considers itself healthy only while it can talk to the data centers over the backbone.

2. Without the backbone, the name server declares itself unhealthy and withdraws its BGP advertisements.

3. The rest of the internet can no longer find the name server, even though it is still running.

03 · OPERATIONS

No network and no tools

While the engineers tried to understand what was happening, they ran into two big obstacles. First, it was not possible to access the data centers by the normal means, because their networks were down. Second, the total loss of DNS broke many of the internal tools normally used to investigate and resolve outages. In terms of monitoring and observability, the diagnostic path and the emergency access path shared the same failure domain as the system that needed to be fixed.

1. Normally, teams use internal tools to reach the data centers and investigate problems.

2. With DNS and the network down, the tools stop working and normal access to the data centers disappears.

04 · RECOVERY

Physical security, time, and a gradual power-up

With normal and emergency access both down, the team sent engineers to the data centers. This took time, since the facilities have a high level of physical and systems security, and secure access protocols had to be activated before putting people to work on the servers. Once the backbone was restored, the post says that turning everything on at once could cause a new round of failures from a traffic spike. Individual data centers were reporting drops in power consumption in the range of tens of megawatts, and reversing that suddenly would put everything from electrical systems to caches at risk. This is a case of disaster recovery, in which turning things back on is a risky step in itself.

1. Engineers go physically to the data center, because the remote paths are down.

2. The backbone is back in operation and the data centers reconnect.

3. Load is returned gradually, to avoid a spike in traffic and power consumption.

05 · LESSONS

What to take to your project

The company says the preparation came from exercises called storms, in which a major failure is simulated by taking a service, a data center, or an entire region offline. According to the post, the experience from these exercises gave confidence to bring everything back carefully, but they had never simulated the global backbone being down and intended to look for ways to simulate events like this. This is the practice of chaos engineering applied to the network. The author also acknowledges a cost: the day-to-day security hardening slowed down recovery, and he considers the trade-off worthwhile, since security increases greatly day to day in exchange for a slower recovery in a rare event.