Azure Front Door Oct 2025: global config passes the safety checks

01 · CONTEXT

One configuration, every edge site in the world

Azure Front Door (AFD) and Azure CDN forward traffic through edge sites distributed globally, which serve customers and Microsoft's own services, including management portals. This case uses Microsoft's official post-incident report (PIR) on the October 29, 2025 outage, which affected customers and services with timeout and DNS resolution errors.

The AFD control plane generates customers' configuration metadata, and the data plane consumes it in operations such as purge and web application firewall (WAF) rules. Because applications can be accessed from anywhere in the world, these changes are deployed to all edge sites.

1. The customer changes their application's configuration, such as routes, WAF rules, or origins, and the change reaches the control plane.

2. The control plane generates configuration metadata, and the data plane, at the edge sites, consumes it.

3. Users from anywhere reach an edge site, which applies the configuration and fetches the content from the origin when needed.

4. The platform's internal DNS, used to resolve names, is hosted on the edge sites themselves.

02 · THE PROTECTION

What should have blocked a bad configuration

AFD uses multiple deployment stages and a configuration protection system. It validates the configuration at each stage and only advances after receiving positive health signals from the data plane.

When a deployment finishes well, the system updates a "last known good configuration" (LKG) snapshot, a periodic picture of healthy configurations that allows automatic rollback in case of problems. The system waits about a minute between stages and finishes, on average, in 5 to 10 minutes worldwide.

03 · THE CHAIN OF EVENTS

A failure that only shows up after the configuration has already passed

A specific sequence of customer configuration changes, made on two different build versions of the control plane, generated incompatible metadata. The changes were valid and non-malicious, but the metadata exposed a latent bug in the data plane, which crashed during asynchronous processing.

The defect escaped pre-production validation because not all resources are validated across different control plane versions.

1. The control plane generates incompatible metadata from valid changes, and the incompatible change is made at 15:35 UTC.

2. The configuration is applied in pre-production at 15:36 UTC and monitoring keeps receiving healthy signals, because the problematic metadata had not yet caused a problem.

3. Because the crash only shows up asynchronously, after about five minutes, the configuration passes through the safeguards and moves on to later stages.

4. By 15:39 UTC most edge sites already had the configuration, and the last known good configuration snapshot was also updated with it.

5. Impact starts in pre-production at 15:41 UTC; at 15:43 UTC the protection system halts new and in-flight changes; by 15:45 UTC every edge site in the world is affected.

The crashes also affected AFD's internal DNS, hosted on the edge sites, which caused intermittent name resolution errors for some customer requests.

The report says this incident was not directly related to the October 9 one: the failure mode was similar, but the underlying defects were different. On October 9 the protection system worked as expected, but was bypassed by engineering during a manual cleanup; on October 29 the configuration passed through the protection system before the delayed asynchronous task caused the crash.

04 · THE RESPONSE

Repairing the last known good configuration by hand

Because the most recent snapshot already contained the conflicting metadata, the team decided not to roll back to it and also not to go back to earlier versions. Instead, it edited the most recent one, manually removing the problematic configurations.

At 17:30 UTC all customer configuration changes were blocked, so as not to reintroduce the problem during mitigation, and at 17:40 UTC the deployment of the updated snapshot to the global fleet began. Recovery required reloading the configurations at each edge site and rebalancing traffic gradually, to avoid overload.

1. The most recent snapshot already contained the conflicting metadata, and the team decided not to roll back to it.

2. Instead of going back to an earlier version, engineering edits the most recent snapshot and manually removes the problematic configurations.

3. At 17:30 UTC customer configuration changes stop being propagated to the data plane, so as not to reintroduce the problem.

4. At 17:40 UTC the deployment of the updated snapshot begins, and each edge site reloads all customer configurations.

5. At 18:30 UTC DNS recovers and traffic is manually rebalanced to a few healthy sites; at 20:20 UTC management becomes automatic; at 00:05 UTC on October 30 the impact is confirmed as mitigated.

Many services that use AFD were able to do failover, such as the Entra and Intune portals and Azure AD B2C. The Azure Portal used its standard recovery process to move off AFD, but some services inside the Portal had no fallback strategy, and parts of the experience kept failing even after the Portal recovered, such as the Marketplace.

1. On the normal path, access to the portal goes through the edge platform.

2. During the incident, the edge platform is failing.

3. The Portal uses its standard recovery process and starts being accessed outside the edge platform; the transition appears in the report at 17:26 UTC.

4. Services inside the Portal with no fallback strategy, such as the Marketplace, keep failing even after the Portal recovers.

05 · REPAIRS AND LESSONS

What changed in the pipeline

Among the deployment safety measures, Microsoft lists full synchronous processing of each configuration before advancing to production stages, removal of asynchronous processing from the data plane, an additional pre-canary stage, and more wait time (bake time) at each propagation stage.

This has a cost: the new stages and wait time increased the propagation time of all operations, including create, update, delete, WAF operations, and cache purges. The report cites a goal of reducing that time from 45 minutes to about 15 minutes.

To reduce the blast radius, Microsoft migrated its own critical infrastructure, such as the Portal, the Marketplace, and support ticket creation, to an active-active solution with fail away, and plans to segment the data plane into micro cells to isolate each customer's configuration and traffic. To recover faster, it cites improvements in data plane recovery time, from about 4.5 hours to about one hour, and decoupling configuration processing from the instances that serve traffic.