OpenAI Dec 2024: telemetry service takes down the Kubernetes control plane and DNS

01 · CONTEXT

Hundreds of clusters and two layers with different roles

On December 11, 2024, all of OpenAI's services were down or severely degraded. The cause was an internal change to roll out new telemetry across the fleet: the collection service overloaded the Kubernetes control plane and, with it, service discovery. There was no security incident. The company's own public post-mortem is the source for this case.

OpenAI operates hundreds of Kubernetes clusters around the world. Each cluster has two layers: the control plane, which administers the cluster, and the data plane, where the workloads, such as model inference, actually run. The data plane operates almost without the control plane, which gives a feeling of isolation, but there are exceptions.

02 · THE TRIGGER

An observability agent that charged every node dearly

As part of an effort to improve fleet observability, the team rolled out a service to collect detailed metrics from the control plane. Telemetry services have a wide footprint, and this one's configuration made every node in every cluster perform expensive operations against the Kubernetes API, at a cost that grows with the size of the cluster.

With thousands of nodes doing this at the same time, the API servers became overloaded and the control plane went down in most of the large clusters. The irony is that the observabilidade tool built to provide more visibility was the one that took down what it was observing.

1. In the normal state, services resolve names through the cluster DNS, which behind the scenes queries the API server.

2. The new telemetry service lands on every node in the cluster.

3. Each node starts performing expensive operations against the API server, and the cost grows with the size of the cluster.

4. Under this load, the API server stops responding and the control plane goes down.

5. DNS depends on the control plane, so service discovery breaks and services cannot locate each other.

03 · WHY IT WAS NOT CAUGHT

A test that passed and a cache that delayed the symptom

The change was tested in a staging cluster and nothing showed up. The effect depended on cluster size, and the problem was stronger in the largest ones. Before the deploy, the main concern was the new service's CPU and memory consumption, and the load on the API server was not evaluated. The rollout tracked the health of the services, but not the health of the clusters.

There was also a built-in delay: the DNS cache on each node served stale but working records. They expired over about 20 minutes, and only then did services start to fail. This gap let the rollout move forward before the problem fully appeared: the change was applied to all clusters in under half an hour.

When the caches emptied, the load on the DNS servers multiplied and added more pressure to the control plane, which made mitigation even harder.

04 · LOCKED OUT

The fix required the system that was broken

Rolling back is normally simple, and here detection worked: the alerts fired before customers felt any impact. The problem is that the fix was to remove the telemetry service, and doing that required access to the control plane, precisely what the load on the API servers prevented. The company itself points to this lockout effect as the reason for the slowness.

1. The pipeline applies the configuration change, which goes through each cluster's API server.

2. The alerts work and call engineering before the impact reaches customers.

3. To remove the service, the team has to go through the API server, which is overloaded and not responding.

4. Meanwhile, traffic is diverted to clusters that are still working.

Three fronts were opened in parallel: shrinking the clusters to reduce the aggregate load on the API, blocking network access to the administration APIs to prevent new expensive requests and give the servers time to recover, and scaling up the API servers to absorb the pending requests. Together, they restored enough control to remove the service.

Recovery still had a side effect: some clusters stayed degraded because many services tried to download resources at the same time, saturating limits and requiring manual intervention. The clusters came back gradually, and the impact lasted from 3:16 PM to 7:38 PM Pacific time.

05 · WHAT THE DESIGN MUST PROVE

Wave rollout, autonomous data plane, and emergency access

OpenAI listed measures that address each link in the chain. The first is to make every infrastructure configuration rollout a robust phased deployment, with continuous monitoring of both services and clusters, control plane included.

The others deal with isolation and recovery. They include fault injection tests so that the data plane survives longer without the control plane, emergency mechanisms to access the API under any circumstances, and fully decoupling the two planes so that the control plane is not essential to critical workloads. To recover faster, the company plans better caches, dynamic rate limiters, and regular exercises that replace an entire cluster.

1. The change goes first to a large canary cluster, which behaves like the largest ones in the fleet.

2. The health of the cluster and the control plane is measured, not just that of the services, and for longer than the DNS cache.

3. Without a healthy signal, the pipeline blocks the next wave and the change does not leave the canary.

4. With a healthy signal, the change advances wave by wave to the rest of the fleet.

5. If the control plane locks up anyway, emergency access still lets you remove the change.