Reddit Pi Day 2023: Kubernetes upgrade breaks Calico

01 · CONTEXT

A legacy cluster on the path of almost everything

On Pi Day, March 14, Reddit was down for 314 minutes. The engineering team's public post-mortem is the source of this case. The trigger was a Kubernetes upgrade, an operation the team itself recognizes as a point of availability risk.

Upgrades go first through clusters dedicated to testing and then reach production, from the least critical environment to the most critical. The cluster upgraded that day was one of the company's most important: it ran the legacy part of the stack.

Not everything went down. The modern API layers stayed up, but the impact hit the most critical legacy node in the dependency graph, and so most user flows were affected.

1. In the diagram, the CDN edge sits in front of the site; it is deliberately separate from the cluster, and the traffic graph cited in the post comes from it.

2. The modern API layers stayed up and resilient.

3. The impact hit the most critical legacy node in the dependency graph, and the affected area still covered most user flows.

02 · THE OUTAGE

Everything looked fine, until the site stopped all at once

The engineer started the upgrade shortly after 19:00 UTC and everything looked normal for about two minutes. The upgrade was from Kubernetes 1.23 to 1.24, and the team says the problem had never appeared in earlier upgrades or in tests in non-production environments.

The affected cluster lost all metrics. One symptom stood out: DNS was not working. Consul and internal cluster DNS records did not resolve, while public records resolved normally.

03 · TRYING TO FIX OR ROLL BACK

Why rolling back was not simple

Rolling back is plan A for any deploy failure, and the team considered it immediately. The problem is that Kubernetes has no supported downgrade procedure. Because the upgrade runs schema and data migrations automatically, there is no defined reverse path, and going back requires restoring a backup and reloading the state.

The upgrade already included a backup by default. The restore had been tested on pilot clusters, but it had not kept up with environment changes and had never been used on a production cluster. Initial estimates were hours of guaranteed downtime, and the initial decision was to keep investigating.

During the investigation, the team noticed timeouts on the API server when calling the admission controllers. The only widespread admission controller was OPA; deleting its webhook configurations made the timeouts go away, but the cluster did not recover.

04 · THE CAUSE

Routes that vanished: Calico and the route reflectors

Calico, the cluster's container network interface, was not working well and its pods were not healthy. Three components matter. calico-kube-controllers assigns IP ranges to nodes. calico-typha is an aggregating proxy with a cache between Calico and the control plane, to reduce load on the Kubernetes API. calico-node runs on every node and generates and registers the pods' network interfaces.

After the outage ended, the team still did not know the reason, and the investigation continued through the logs. The logs were preserved because the logging agent was not affected. In them, two seconds before the chaos, calico-node across the whole cluster started dropping routes to the first upgraded control plane node, which is normal and expected behavior. What was unexpected was that the routes to all nodes also started being dropped.

By default, in Calico every node peers directly with every other node, in a mesh. In larger clusters this becomes heavy, and the alternative is route reflectors: a few nodes peer with all the others and the rest peer only with the reflectors.

1. calico-node runs on every node and generates and registers the pods' network interfaces.

2. calico-typha sits between the Calico components and the control plane, to reduce load on the Kubernetes API.

3. In this cluster, the control plane nodes were the route reflectors and all the other nodes used them.

4. With the first control plane node offline for the upgrade, calico-node started dropping routes to it, which is normal and expected behavior.

5. But calico-node also started dropping the routes to all nodes, which was not expected.

Nobody else knew the reflectors existed: everyone who knew had changed roles or left the company. Part of the configuration required downloading Calico-specific data, managed only through the CLI, editing it by hand and uploading it back, and the team had not written its own tooling to treat this as code. As a result, the configuration was not recorded anywhere.

The reflectors' node and peer selectors pointed to the label node-role.kubernetes.io/master. In version 1.24 Kubernetes removed references to master, even in running clusters, and the team points to this as the cause of the outage.

05 · RESTORATION

Coming back from the backup, step by step

With the outage passing the two-hour mark, the team decided to restore from backup. Since most worker nodes would be invalidated by the restore, they were terminated, which took about 20 minutes of API calls. No one involved had performed a restore before, and it was written for a Kubernetes version already out of support and predating CRI-O, with instructions designed for Docker.

1. Before the restore, the worker nodes were terminated, because most would be invalidated by it.

2. The procedure terminated the other control plane nodes, restored the backup on the first one, and brought it up as a single-node control plane.

3. After the first node's restore, worker nodes started coming back with the autoscaler, a sign that the network was working again.

4. AWS capacity for the control plane instance type was exhausted, which delayed the new nodes from coming up.

5. The new nodes did not join: they could not connect to the first node's etcd. The backup could be run on any control plane node, but the restore had to be done on the same node, and it was not.

6. With new certificates generated because of the hostname difference, the new nodes joined and the high-availability control plane came back.

The difference was between the two procedures: the backup could be run on any control plane node, but the restore had to be done on the same node, and it was not. Because of the hostname difference, new certificates had to be generated, despite the lack of documentation.

Traffic came back conservatively, in steps from 1% to 100%. Reddit depends on many caches and there are points where a thundering herd can form if traffic returns to 100% all at once. Some of the more sensitive legacy services were stopped manually and carefully started again.

06 · LESSONS

What the design needs to prove

The team says it will use the outage to change architecture and process decisions and make cluster upgrades safe. Among the actions described: standardize and share more components across environments, keep more pre-production clusters to test with confidence, and codify whatever possible, to apply it consistently and keep a historical record of the choices.