AWS us-east-1 Oct 2025: race condition in DynamoDB DNS automation

01 · CONTEXT

An empty DNS name takes down an entire region

On October 19-20, 2025, AWS's us-east-1 region had a long outage that started in DynamoDB and spread to EC2, load balancers, and other services. AWS's own post-event summary is the source for this case, and it splits the event into three impact periods.

The trigger was a failure in the automation that maintains the DNS records for the regional endpoint. The root cause was a latent race condition that left the endpoint with no addresses at all, and the automation could not repair itself.

DynamoDB depends on a very large fleet of load balancers in each region and on hundreds of thousands of DNS records to operate it. When the regional endpoint stopped resolving, customers and other AWS services could not open new connections, which turned a single-component defect into a failure of regional availability.

02 · THE RACE CONDITION

Two correct automations and an old plan in the wrong place

To understand the failure, you need to see how DynamoDB's DNS was maintained. The system is split into two independent components for availability reasons. The Planner monitors load balancer health and capacity and generates DNS plans, and the Enactors apply those plans to the managed DNS service.

There is a single regional plan, which simplifies capacity management when several endpoints share load balancers. The Enactor is redundant and independent across three availability zones, and each instance tries to swap the current plan for a new one in a DNS service transaction.

1. The Planner keeps generating new plans, and each Enactor receives the most recent plan it knows about.

2. One Enactor hits unusual delays: its transactions are blocked by another Enactor and it retries across several endpoints, still holding a plan that has gone stale.

3. Meanwhile, the other Enactor quickly applies a much newer plan to all endpoints.

4. The check that the plan was newer was done only at the start and had already expired. The delayed Enactor applies its old plan to the regional endpoint and overwrites the new one.

5. The fast Enactor's cleanup deletes the old plan because it is many generations behind. Since that was the active plan, the endpoint is left with no addresses and no Enactor can apply later plans.

6. Customers and AWS services cannot resolve the endpoint and cannot reach the database, which was healthy. The fix required manual intervention by operators.

03 · THE CASCADE

The database is back, but the rest of the region is still wary

After DNS was restored, DynamoDB started responding again. Customers resumed connecting as cached records expired. Those using global tables could reach replicas in other regions during the outage, but saw replication lag with the affected region's replica. Even so, EC2 and the load balancers stayed degraded for hours.

1. The host manager's state checks depend on the database. With the endpoint down, they fail and the leases on the physical servers begin to expire, even though running instances remain healthy.

2. With the database back, the manager tries to re-establish the leases for all servers. The work takes longer than the leases' own deadline, retries pile up in the queue, and it falls into congestive collapse.

3. Servers without a lease are not candidates for new launches, and the launch API responds with insufficient-capacity errors even though hardware is available.

4. When the leases return, the Network Manager has to process the accumulated backlog of network configurations. Instances come up but have no connectivity while the queue drains.

5. The load balancer's health check starts seeing new instances whose network has not propagated yet. Healthy nodes flip between failing and succeeding and are removed from DNS and returned on every check.

6. The health check itself degrades under load, and the automatic per-zone failover removes capacity from the load balancer. Applications with little remaining capacity start getting connection errors.

The origin of the second and third periods was the recovery, not the original failure: a subsystem trying to resume all leases at once and a propagation queue that grew during the outage. AWS describes the host manager's collapse as a situation with no established operating procedure.

04 · RECOVERY

Limiting load so you can recover

The timeline shows how diagnosis and recovery moved at different paces. The origin in DynamoDB's DNS was identified at 12:38 AM, and the DNS information was restored at 2:25 AM, but the remaining recoveries depended on separate interventions.

On the host manager, engineers throttled the incoming work and selectively restarted hosts, which emptied the queues and allowed the leases to be re-established. On the load balancer, they disabled the automatic health-check failover to return healthy nodes to service. EC2 only returned to normal at 1:50 PM, after request limits were gradually relaxed.

05 · WHAT TO TAKE TO YOUR PROJECT

Blast radius of automation and the control plane

The changes announced by AWS point to three kinds of protection. First, AWS disabled the DynamoDB DNS automation worldwide, promising to fix the race condition and to add protections against applying incorrect plans before re-enabling it.

For the load balancer, there will be a velocity control that limits how much capacity a single NLB can remove when health check failures trigger zone failover. For EC2, load throttling in the propagation systems will now take the size of the wait queue into account.

Beyond what AWS announced, the design of the case suggests useful questions for any critical regional service: what happens when the automation that maintains DNS writes an empty value, which internal dependencies only work with one specific database, and how the monitoramento tells a source failure apart from a recovery failure. A disaster recovery strategy must cover the case where the whole region loses a foundational service, and multi-region architecture only helps if the control planes also stop depending on a single region.