Cloudflare Nov 2023: power outage takes down the control plane

01 · CONTEXT

An outage that, according to the company, should not have happened

On November 2, 2023, Cloudflare's control plane and analytics services suffered an outage. According to the post-mortem, the incident lasted from 11:44 UTC on November 2 until 04:25 UTC on November 4.

The post-mortem defines the control plane as the customer-facing interface for all services, including the website and the APIs; the analytics services, in turn, include logging and analysis reports.

The company states that it believed it had high availability systems capable of preventing an outage like this, even with the catastrophic failure of a data center provider. Many systems stayed up as designed, but some critical systems had non-obvious dependencies that made them unavailable.

02 · THE INTENDED DESIGN

Three independent data centers and a separate network

The control plane and analytics run mainly on servers in three data centers in the Hillsboro region of Oregon. They are independent of each other, each with multiple power feeds and multiple redundant, independent network connections.

The distance between them was chosen to reduce the chance of a natural disaster hitting all three, but short enough to operate active-active data clusters. Data is synchronized continuously, so that, by design, losing one of them does not stop the others. See also disaster recovery.

1. The customer uses the control plane dashboard and APIs, which run on servers in three data centers near Hillsboro, Oregon.

2. The data centers are independent and synchronize data continuously; by design, if one goes offline, the others keep operating.

3. The largest of the three, operated by Flexential, houses the largest analytics cluster and more than a third of the machines in the high availability cluster.

4. The logging systems sit outside the high availability cluster on purpose: logs would be queued at the network edge, and delaying analytics was considered acceptable.

5. Customer traffic goes through the global network, which kept working as expected even with the control plane down.

The migration to this high availability cluster was not yet complete: some services, mostly from newer products, had not been added to it. The logging systems were deliberately kept outside the cluster, with logs queued at the edge while the logging site was offline.

03 · THE CHAIN OF FAILURES

From the utility's maintenance to the total blackout

The affected data center, the largest of the three, is operated by Flexential. It houses Cloudflare's largest analytics cluster and more than a third of the machines in the high availability cluster.

The power utility, PGE, performed unplanned maintenance that affected one of the building's power feeds. Flexential did not tell Cloudflare that it had switched to generators, and Cloudflare's observability tools did not detect the change in power source.

Around 11:40 UTC there was a ground fault in a PGE transformer at the data center. The electrical protection measure also shut down all of the building's generators, and the UPS batteries, which should have sustained it for about 10 minutes, began to fail after 4 minutes; Flexential took well over 10 minutes to restore the generators.

Cloudflare reports, without official confirmation, that employees told it there were three factors that delayed the generators coming back: on-site manual restart, access control with no backup battery, and an overnight team without an experienced operations or electrical specialist.

04 · HIDDEN DEPENDENCIES

What should have been in the cluster but wasn't

Cloudflare's expectation was that analytics would go offline, logs would be queued at the edge and delayed, and lower-priority services outside the high availability cluster would be temporarily offline until restored at another site. The other two data centers would take over the cluster and keep the critical services running, which in general worked as planned.

1. The high availability cluster's services run in all three data centers; if Flexential's failed, the other two would take over the critical services.

2. Kafka and ClickHouse, which process logs and power analytics, were available only inside the Flexential data center.

3. Even so, services in the high availability cluster depended on these two components.

4. With the data center without power, these components went down, and critical systems that depended on them became unavailable.

According to Cloudflare, services in the high availability cluster depended on components that existed only in the affected data center, and it was non-obvious dependencies like this that made critical systems unavailable. The company itself says these dependencies should not have been so rigid, should have failed more gracefully, and should have been detected.

The company's failure tests had taken each of the other two data centers and the high availability part of the affected one offline, but never the entire data center. Cloudflare acknowledges that, because of this, it did not notice the importance of some dependencies. It also admits that it demanded little of new products in integrating with the cluster and did not formally require it before they were declared GA, which made the redundancy protections work unevenly depending on the product.

05 · THE RECOVERY

Failover to Europe, burst of calls, and a restart from scratch

1. The affected data center loses power, and most services keep running on the high availability cluster, in the two active data centers.

2. At 12:48 UTC Flexential restarted the generators; when Cloudflare's circuits were turned back on, the circuit breakers turned out to be faulty. At 13:40 UTC, Cloudflare decided to fail over to its disaster recovery sites in Europe, and only a small percentage of the control plane needed to be transferred.

3. The first services came up at the recovery site at 13:43 UTC, and the API calls that had been failing overloaded the services (thundering herd); Cloudflare applied rate limits.

4. At 17:57 UTC what had been moved to the recovery site was stable and most customers were no longer directly affected; log processing and some custom APIs remained unavailable until the data center was restored.

5. Generally newer products did not come up properly at the recovery site, for lack of a fully implemented and tested procedure; the team pursued two fronts, redeploying them there and migrating them to the high availability cluster.

6. Flexential confirmed clean power at 22:48 UTC; returning to the data center required a full bootstrap of the facility, and services were fully restored on November 4, at 04:25 UTC.

With clean power confirmed, the CEO decided that most of the team should rest and start the return to the data center the next morning. According to him, the decision delayed full recovery but reduced the chance of new errors. Since the state of the services was unknown, the only safe process was a full bootstrap of the building.

The process required bringing up the configuration management servers first, which took 3 hours. After that, each server took from 10 minutes to 2 hours to be rebuilt. This could be done in parallel across several servers, but dependencies between services required some to come back in sequence.

The impact on data was uneven: because part of the analytics also lives in the European data centers, most customers should not have lost analytics data, but datasets not replicated in the EU will have permanent gaps. The raw log services were unavailable to most customers throughout the incident, and those who use log push will not get back the logs they did not receive.

06 · LESSONS

What Cloudflare will change and what to take into your design

The company decided to create what it called Code Orange, its own version of Google's Code Yellow and Code Red processes, and had all non-critical engineering functions focus on control plane reliability.

Cloudflare says that many services depend on the availability of the core and fail if it becomes unavailable, and that it needs to use for all its services the distributed products it offers customers, so that they keep working even with the core interrupted.

Useful questions when reviewing a similar design: where do the supporting components, such as queues and analytics storage, of each service marked as redundant live? Does the failure test (chaos engineering) remove the entire failure domain or only the redundant part of it? Is there a disaster recovery plan tested before a product is released to everyone? These precautions come close to cell-based architecture, which limits the blast radius of each failure.