Azure AD Mar 2021: rotation removes signing key

01 · CONTEXT

When the identity provider is everyone's dependency

From around 19:00 UTC on March 15, 2021 until 09:25 UTC, customers may have encountered errors in authentication operations. The impact reached Microsoft and third-party applications that depend on Azure AD for authentication.

Microsoft's preliminary post attributes the incident to an error in the rotation of the keys that Azure AD uses in OpenID and other identity protocols, for cryptographic signing operations. The case is therefore a problem of disponibilidade that originates in a key rotation operation.

Before the failure, the expected behavior is this: Azure AD publishes the signing key metadata at a global location, and applications that use these protocols fetch that metadata and keep it cached.

1. The user authenticates with the identity service, which issues a token signed with one of its keys.

2. The identity service publishes the signing key metadata at a global location.

3. The application fetches this metadata and keeps a cached copy.

02 · THE CHAIN OF EVENTS

A key removed by automation

As a security hygiene measure, an automated system removes, on a time schedule, keys that are no longer in use. In recent weeks, one specific key had been marked as retain for longer than normal, to support a complex cross-cloud migration.

This situation exposed a bug: the automation ignored the retain state and removed precisely that key.

1. The automation removes the key marked as retain, because it ignores that state.

2. The public key metadata changes at 19:00 UTC.

3. Applications fetch the new metadata and stop trusting tokens signed with the removed key.

4. End users can no longer access these applications.

5. The service telemetry identifies the problem and the engineering team is paged automatically.

6. The key removal is identified as the cause, and the key metadata is returned to its previous state at 21:05 UTC.

03 · THE RECOVERY

Reverting the metadata does not revert the caches

Microsoft rolled the metadata back to its previous state at 21:05 UTC. However, applications needed to fetch the reverted metadata and renew their caches.

The post states that time to mitigation varies by application, because of server implementations that handle the cache in different ways.

A subset of Storage resources had residual impact because of cached metadata. Microsoft pushed an update to invalidate those entries and force a refresh, and mitigation for the still-affected customers was declared at 09:25 UTC.

1. The key metadata returns to its previous state.

2. For a subset of Storage resources, the cached metadata kept the residual impact.

3. An update invalidates the cached entries and forces a refresh from the source.

04 · WHAT WAS MISSING

Backup authentication and safe deployment

In the earlier September incident, Microsoft had mentioned rolling out a backup Azure AD authentication. In this case it did not help: it covered token issuance, but not validation, which depended on the affected metadata endpoint.

1. The backup authentication covered token issuance.

2. Token validation depended on the affected metadata endpoint.

3. With no coverage for validation, the backup authentication did not help in this case.

Azure AD had a multi-phase effort under way to protect the backend's safe deployment system (SDP). The first phase protects adding a new key; the key removal component was in the second phase, planned for mid-year.

05 · LESSONS

What the design needs to prove

In any system with a central identity provider, it is worth separating two paths and asking which one each failure goes through: issuing credentials and validating them. A contingency plan only covers what it was designed to cover.

Removing a shared artifact deserves the same rigor of progressive rollout as creating one: the change reaches a small part of the consumers first and only advances if the health signals stay normal.

Reverting at the source only helps those who re-read the source. Metadata consumers need a predictable path to renew or invalidate the cache, and whoever operates the system needs to know how to use it under pressure. Alerts for observabilidade about validation failures shorten the time to detection.