Incident · Mid
Fastly 2021: a valid customer configuration triggers a latent bug
Real incident. Key points: progressive deployment.
Short answer
Real incident. Key points: progressive deployment. On June 8, 2021, Fastly published a summary of a global outage that occurred the same day. The post is the source for this case. According to the company, a previously undiscovered software bug surfaced when it was triggered by a valid customer configuration change.
01 · CONTEXT
A valid configuration and a bug nobody had seen
On June 8, 2021, Fastly published a summary of a global outage that occurred the same day. The post is the source for this case. According to the company, a previously undiscovered software bug surfaced when it was triggered by a valid customer configuration change.
Um CDN is a shared edge network, positioned between users and customers' sites. This case shows what it means to split the same layer among many customers.
02 · THE TRIGGER
A May deployment and a June configuration
On May 12, Fastly began a software deployment that introduced a bug. The bug could be triggered by a specific customer configuration under specific circumstances. Early on June 8, a customer pushed a valid configuration change that included those circumstances, and 85% of the network began returning errors.
On May 12, Fastly began a software deployment that introduced a bug.
Early on June 8, a customer pushed a valid configuration change.
That change included the specific circumstances that triggered the bug.
The network returned errors across 85% of its footprint.
The outage hit Fastly's customers and the people who depend on them.
03 · THE RESPONSE
From the first minute to the permanent fix
According to Fastly, the outage was detectada within a minute; then the cause was identified and isolated and the configuration was disabled. Within 49 minutes, 95% of the network was operating normally.
| Time (UTC) | Evento |
|---|---|
| 09:47 | Global outage begins |
| 09:48 | Outage identified by Fastly's monitoring |
| 09:58 | Status post published |
| 10:27 | Fastly engineering identifies the customer configuration |
| 10:36 | Affected services begin to recover |
| 11:00 | Most services recovered |
| 12:35 | Incident mitigated |
| 12:44 | Status post resolved |
| 17:25 | Bug fix deployment begins |
At 09:48 UTC, Fastly's monitoring identified the global outage.
At 10:27 UTC, Fastly engineering identified the customer configuration.
The cause was isolated and the configuration was disabled.
At 10:36 UTC, affected services began to recover.
At 12:35 UTC, the incident was considered mitigated.
At 17:25 UTC, the bug fix deployment began.
04 · COMMITMENTS
What the post promises to do
In its conclusion, Fastly states that, although the conditions that triggered the outage were specific, the company should have anticipated it.
| Stated commitment | |
|---|---|
| Bug fix | Deploy the fix to the network as quickly and as safely as possible |
| Post-mortem | Conduct a full post-mortem of the processes and practices followed in the incident |
| Testing and quality | Find out why the bug was not detected during the quality assurance and software testing processes |
| Remediation | Evaluate ways to improve remediation time |
| Plataforma | Make full use of the isolation capabilities of WebAssembly and Compute@Edge to build more resilience from the ground up |
05 · LESSONS
Questions for your design
This case invites us to look at disponibilidade in terms of blast radius: how much of the system can a single change reach?
It also invites us to look at the operations routine: how quickly does monitoring notice a problem, how quickly does the team find the trigger, and what does it take to disable it?
Frequently asked questions
What is Fastly 2021: a valid customer configuration triggers a latent bug?
Real incident. Key points: progressive deployment.
What does the Fastly 2021: a valid customer configuration triggers a latent bug design cover?
It is organized in these sections: A valid configuration and a bug nobody had seen, A May deployment and a June configuration, From the first minute to the permanent fix, What the post promises to do, and Questions for your design.
How long is Fastly 2021: a valid customer configuration triggers a latent bug and what level is it?
It takes about 12 minutes to read and is rated Mid level.
Now design it yourself
Run this case as an interview in the System Design school: you make the decisions, the tutor challenges them and you see how your design scores.