Incident · Junior
Microsoft Teams 2020: expired certificate
Real incident. Key points: time and clocks.
Short answer
Real incident. Key points: time and clocks. On the morning of Monday, February 3, 2020, Microsoft Teams went down. Microsoft said an expired authentication certificate caused the outage, and Dark Reading reports that millions of people were unable to log in.
01 · CONTEXT
A login that depends on a certificate
On the morning of Monday, February 3, 2020, Microsoft Teams went down. Microsoft said an expired authentication certificate caused the outage, and Dark Reading reports that millions of people were unable to log in.
To understand the effect, a simple model of how login works is useful. The following diagram is a didactic simplification, not Microsoft's actual architecture.
The user opens the app and the login goes through the authentication service.
Once authenticated, the user starts using the chat and collaboration service.
The authentication service depends on an authentication certificate to grant access.
02 · THE INCIDENT
From outage to fix
According to GeekWire, Microsoft acknowledged the problem shortly after 6 a.m. Pacific time, in a tweet. A little over an hour later, it said an expired authentication certificate had caused the outage.
The authentication certificate expired.
Without that valid certificate, millions of users could not log in.
Microsoft prepared a new certificate and began rolling it out to bring the service back.
Hours later, Microsoft said the fix had been applied.
| GeekWire | Dark Reading | |
|---|---|---|
| Reconhecimento | Shortly after 6 a.m. Pacific time, in a tweet | Around 9:15 a.m. Eastern time, the outage was acknowledged |
| Causa | A little over an hour later, Microsoft pointed to an expired authentication certificate | Around 10:30 a.m. Eastern time, the problem was acknowledged as certificate-related |
| Fix | Microsoft was preparing a new certificate and began rolling it out | Near 5 p.m. Eastern time, the message came that the fix had been applied |
Dark Reading sums up the effect: the expired certificate left the Office 365 version of Teams down for almost a whole day.
03 · TIME AS A DEPENDENCY
What a clock can break
According to GeekWire, The Verge called the outage an embarrassing mistake and noted that Microsoft builds tools that monitor problems like certificate expiration, so it should not let them lapse in core programs like Teams. That is the point of monitoring and observability: a known deadline can be watched.
The design question is: where else does your system depend on a date? Certificates, tokens, keys, and licenses have expiration dates. If nobody owns the renewal and nothing warns when the deadline approaches, the expiration only shows up when the service stops. The following diagram illustrates one possible control; it does not describe what Microsoft was doing.
The monitor reads the certificate's expiration date and compares it with the current date.
When the deadline approaches, the monitor notifies the responsible people and triggers the renewal routine.
The routine installs the new certificate before expiration, and the service continues without interruption.
| Controle | What it does |
|---|---|
| Inventory with an owner | Lists each certificate and credential with its deadline and a responsible person or team. |
| Expiration alert | Warns in advance, on a channel someone actually reads, before the deadline arrives. |
| Automatic renewal | Swaps the certificate by routine, without relying on a manual task someone remembers. |
| Rotation drill | Practices rotation outside a critical moment, so the procedure is familiar. |
04 · WRAP-UP
What to take into the interview
In a conversation about disponibilidade, remember that not every failure comes from load or hardware. Cite the dependencies with deadlines, say how you would inventory, monitor, and renew them, and how you would reduce the blast radius if one of them failed.
Frequently asked questions
What is Microsoft Teams 2020: expired certificate?
Real incident. Key points: time and clocks.
What does the Microsoft Teams 2020: expired certificate design cover?
It is organized in these sections: A login that depends on a certificate, From outage to fix, What a clock can break, and What to take into the interview.
How long is Microsoft Teams 2020: expired certificate and what level is it?
It takes about 9 minutes to read and is rated Junior level.
Now design it yourself
Run this case as an interview in the System Design school: you make the decisions, the tutor challenges them and you see how your design scores.