Datadog Mar 2023: automatic systemd update wipes out routes

01 · CONTEXT

Isolated regions and changes in sequence

On March 8, 2023, Datadog had an outage that affected all services across several regions. The company's engineering post is the source for this case. It concludes that new systemd behavior on Ubuntu 22.04 disconnected more than 60% of instances from the network.

The company considered this kind of outage rare: the regions have no direct connections to each other and changes go out one region at a time. Node replacement follows a gradual process, orchestrated across regions and clusters, with the change under observation for one week to one month between stages.

1. Node replacement is orchestrated to advance progressively across regions and clusters.

2. Between stages, the change stays under observation to avoid a regression.

3. The servers also used the Ubuntu default for automatic updates, and all hosts ran that update in the same daily window.

02 · THE PODS' NETWORK

Pods with cloud IPs depend on rules on the host

The pods received IPs from the cloud's own network (VPC), and Cilium managed their networking. The Cilium agent on each node takes care of routes and rules, because the pods need to rely on the host's routing rules to talk to the rest of the network.

1. The pod gets an IP from the cloud's own network (VPC).

2. The Cilium agent on each node maintains the host's routes and rules.

3. To talk to the rest of the network, the pod depends on the host's routing rules.

03 · THE TRIGGER

A security update that deletes rules

systemd-networkd, the systemd component that manages hosts' network configuration, deletes all IP rules it does not know about when it starts. When a systemd patch is installed, all of its components are restarted, including systemd-networkd.

Ubuntu 22.04 uses systemd version 249, with the default configuration that manages external routing rules. The rule-deleting behavior only shows up when systemd-networkd starts, which in the company's infrastructure only happened when a new host was created, when there were no specific rules yet.

On March 7, 2023, a systemd CVE patch became available in the Ubuntu repositories, and the hosts ran apt-upgrade every day between 06:00 and 07:00 UTC. The new behavior affects only the 22.04 nodes, which were more than 90% of the fleet on March 8.

1. The systemd patch is installed and restarts all of its components, including systemd-networkd.

2. On start, systemd-networkd deletes the IP rules it does not know about, and Cilium's rules are among them.

3. On AWS and Azure, without the source rules, the pods' traffic stops using the correct interface.

4. On all providers, without the original local rule, traffic to the node's IP is no longer treated as local and the node loses its network.

04 · THE CLOUDS REACTED DIFFERENTLY

Restart on one cloud, replace on the other

On Google Cloud and Azure, instances that lose connectivity keep running without a network. Those providers repair instances with a hardware or hypervisor problem, but by default not those with a problem at the guest operating system level. Since the company does not enable HTTP health checks, nothing was replaced, and the instances were recovered by restarting them through the APIs.

On AWS, an instance in an Auto Scaling group that fails status checks is terminated and replaced, and AWS detected the instances as unhealthy. By 08:00 UTC, it had replaced about 60% of the instances in the region, tens of thousands.

The company relies heavily on local disks for data stores, coordination, and caches. On AWS, the instances were terminated and all the data on the local disks was lost.

1. On Google Cloud and Azure, the instance stays up without a network, nobody replaces it, and the local disk is still there.

2. On AWS, the status check flags the instance as having no network and Auto Scaling terminates and replaces it.

3. With the instance terminated, what was on the local disk disappears with it.

4. The situation that looked better, AWS, was the hardest to recover.

05 · LESSONS

What the design needs to isolate

The post shows two change paths on the nodes: gradual replacement, orchestrated across regions and clusters, and the operating system's automatic update, which ran on all hosts in the same daily window.

The post's conclusion is that the new systemd behavior on Ubuntu 22.04 disconnected more than 60% of instances, across several regions and providers, and that the outage also affected the regions that host the CI/CD and automation tools.