Roblox Oct 2021: 73 h outage from Consul contention

01 · CONTEXT

An entire platform resting on one service registry

Between October 28 and 31, 2021, Roblox was down for 73 hours. The platform serves tens of millions of players per day across hundreds of internal services. The official post-mortem states there was no loss of user data and no unauthorized access; the duration, not the damage to data, is what makes the case a classic of incident response.

The infrastructure is self-owned, with more than 18 thousand servers and 170 thousand containers. Nomad schedules the containers, Vault delivers secrets, and Consul does service discovery, health checks, session locks, and also serves as a KV store. Services only find each other through this registry.

Consul runs as a cluster with 5 voters and 5 non-voters. Voters hold the authoritative state and non-voters are read replicas. A leader is elected using the algorithm Raft and it is the one that replicates the writes.

1. Player traffic arrives through DNS and goes to the backend services.

2. Each service queries Consul to learn where the services it depends on are.

3. The scheduler and the secrets vault also depend on Consul, so it sits on the path of almost everything.

4. Below the services are the caches and databases. The databases were not affected by the outage, but the caching system became unhealthy.

02 · THE INVESTIGATION

Three reasonable hypotheses, three failures

On the afternoon of October 28, Vault became slow and a single Consul server showed high CPU, with no impact on players. Metrics pointed to elevated write latency on the KV, and the first hypothesis was degraded hardware, with one node replaced. Even with new hardware, performance did not return, and at 16:35 online players dropped to 50% of normal.

The reason for the total outage is structural: without a healthy Consul, services cannot locate each other, the scheduler does not create containers, and the vault does not deliver secrets. The post itself concludes that Consul was a single point of failure. At that point came reasoning about disponibilidade of a layer that all the others assume is up.

03 · THE ROOT CAUSE

A new feature and a storage defect

Looking inside Consul, instead of looking from the services that depend on it, the team saw KV writes blocked for long periods: contention.

Months earlier, Roblox had migrated from Consul 1.9 to 1.10 to use a streaming feature, designed to reduce CPU and bandwidth when distributing updates. One day before the outage, on October 27 at 14:00, the feature was enabled on a traffic-routing service, and that service's node count rose 50% for the year-end peak.

1. The routing service starts subscribing to updates via streaming on the read nodes.

2. At the same time, the other services write health checks and data to the leader, in a high write load.

3. Under very high read and write, contention on a single channel blocks the writes and the cluster loses efficiency.

4. Even after turning off streaming, some leaders remained slow, and the team had to prevent them from staying elected.

HashiCorp's analysis showed the effect: the streaming design concentrates contention on a single Go channel when read and write are both very high, and two-socket servers with NUMA memory make the problem worse, which explains why more cores did not help. Turning off streaming, propagated at 15:51 on October 30, brought median write latency down to about 300 ms.

The second problem stayed hidden longer. The Raft stores its log in BoltDB, which never returns space to disk after deleting old entries, and instead reuses free pages tracked by a freelist. Under Roblox's load, a 4.2 GB log held only 489 MB of real data, and a 7.8 MB freelist was rewritten on every append, even for writes of 16 kB or less.

04 · RECOVERY

No visibility and no cold start

The cause also took time to surface because the monitoramento depended on what was broken. Critical monitoring systems depended on affected systems, such as Consul, and there was a circular dependency between telemetry and the registry.

1. In normal operation, services send metrics to telemetry.

2. Telemetry discovers what to collect by querying the same Consul.

3. With Consul unhealthy, telemetry loses visibility exactly when the team needs it most.

With Consul stable after 54 hours, what remained was rebuilding the cache. The cache served about 1 billion requests per second and, because it holds transient data, the simplest way out was to redeploy it. The cache's scheduling data in the KV was incorrect, probably because of the snapshot reset, and an unhealthy node appeared to the scheduler as free and received jobs that failed. The deployment tool was meant for incremental adjustments, not for bringing up a large cluster from scratch.

At 05:00 on October 31, 61 hours after the start, Consul and the cache were healthy. With cold caches, Roblox used DNS steering to limit how many players got in, raising access in steps of about 10% and checking database load, cache performance, and stability at each step. At 16:45 on Sunday, 73 hours after the start, all players had access.

1. Players try to get in again and DNS decides who passes.

2. A randomly chosen fraction goes to the services and the rest stays on the maintenance page.

3. With empty caches, reads fall through to the database until the caches warm up.

4. The team checks the metrics before raising the fraction to the next step.

05 · LESSONS

What changed and what the case teaches

The failure points lay in how the platform depended on a single Consul cluster for several workloads. Roblox built an additional datacenter and began migrating to multiple zones, separated critical services into dedicated clusters, started moving data out of the KV into more appropriate storage, removed the circular telemetry dependency, and added alerts targeted at Consul and BoltDB. Part of the disaster recovery plan became bringing up caches from scratch.