Incident · Senior
VTEX 07/13/2023: rollback doesn't stop the cascade and disaster recovery is needed
The rollback came in 12 min, but the cascade had already spread. Almost half of the sessions and orders worldwide were affected, part of the infrastructure was rebuilt, and one of the actions is a centralized cache flush.
Short answer
The rollback came in 12 min, but the cascade had already spread. Almost half of the sessions and orders worldwide were affected, part of the infrastructure was rebuilt, and one of the actions is a centralized cache flush. On July 13, 2023, at 15:02 UTC, VTEX's alarms indicated a possible global platform outage.
01 · CONTEXT
A global outage alarm
On July 13, 2023, at 15:02 UTC, VTEX's alarms indicated a possible global platform outage. The incident response team convened, identified a problem in the License Manager module, and began remediating it.
As you read, look for three things: what rolling back a deploy fixed and what it did not; how part of the degraded infrastructure is rebuilt, with a disaster recovery done under pressure; and what the company decided to change afterward, especially in the strategy for cache.
02 · TIMELINE
From the alarm to the degraded infrastructure
The official report records a sequence of milestones, all in UTC. The diagram shows only the components the text mentions and does not aim to reproduce the platform's internal dependencies.
The team identified a problem in the License Manager module.
At 15:14 UTC the team rolled back a deployment and observed gradual recovery of the License Manager.
The team also identified a cascading failure across several modules.
At 15:36 UTC most of the cascade was mitigated, but part of the underlying infrastructure remained degraded and affected almost half of global sessions and orders.
| What the report records | |
|---|---|
| 15:02 UTC | Alarms fire for a possible global platform outage. |
| 15:14 UTC | A deployment is rolled back, with gradual recovery of the License Manager. |
| 15:36 UTC | Most of the cascading failure is mitigated; part of the underlying infrastructure remains degraded. |
| 16:47 UTC | A disaster recovery protocol begins to rebuild part of the degraded infrastructure. |
| 19:26 UTC | More than 90% of the global sales flow is recovered. |
| 23:20 UTC | The minor instability tied to above-normal order cancellations is resolved. |
03 · THE ROLLED-BACK DEPLOY AND THE CASCADE
Rolling back helped the module, but did not end the incident
At 15:14 UTC the team rolled back a deployment and saw the License Manager gradually recover. The same report also says that a cascading failure across several modules was identified.
At 15:36 UTC most of the cascade was mitigated, but part of the underlying infrastructure remained degraded and affected almost half of global sessions and orders. The report also states that several actions increased platform stability, but not consistently up to 100%.
04 · DISASTER RECOVERY
Rebuilding part of the infrastructure
At 16:47 UTC, after several actions that did not consistently bring stability to 100%, VTEX initiated a disaster recovery protocol to fully rebuild part of the degraded infrastructure. A status update published during the incident said the process was taking longer than initially expected.
Part of the underlying infrastructure was degraded and affected almost half of global sessions and orders.
Disaster recovery began rebuilding part of the degraded infrastructure.
VTEX shifted traffic to the healthy infrastructure, taking care to avoid regression in the mitigations.
Over the next two hours, the sales flow went from almost half to almost 100%.
At 19:26 UTC more than 90% of the global sales flow was recovered, and soon afterward recovery reached 100%.
| Rolling back the deployment | Disaster recovery | |
|---|---|---|
| What was done | A deployment was rolled back, with gradual recovery of the License Manager. | Part of the degraded underlying infrastructure was rebuilt. |
| What the report records next | A cascading failure across several modules was identified. | The sales flow went from almost half to almost 100% over the next two hours. |
Even after recovery reached 100%, there was minor instability for some customers, with an above-normal order cancellation rate of about 2-5%, which was resolved at 23:20 UTC.
05 · ACTIONS AND THE DAY BEFORE
What VTEX decided to change
The report lists actions to avoid similar incidents. One of them is to monitor the system's deploy metrics earlier and identify services with no update for a certain period, as a way to ensure preventive change management.
Another is to redesign the strategy for cache across several services, with flush buttons that can be adjusted centrally. According to the report, this reduces the time to mitigate incidents, because it avoids having to provision capacity to invalidate certain caches.
The action addresses caches used by several services.
The redesigned strategy includes cache flush buttons that can be adjusted centrally.
The report says this reduces mitigation time, because it avoids provisioning capacity to invalidate certain caches.
The two reports are separate publications, and the global report does not mention the other; for that reason this text does not link one incident to the other as a cause. Compare the two as a pair, paying attention to what each learned about disponibilidade and about how much of the platform a failure can reach.
Frequently asked questions
What is VTEX 07/13/2023: rollback doesn't stop the cascade and disaster recovery is needed?
The rollback came in 12 min, but the cascade had already spread. Almost half of the sessions and orders worldwide were affected, part of the infrastructure was rebuilt, and one of the actions is a centralized cache flush.
What does the VTEX 07/13/2023: rollback doesn't stop the cascade and disaster recovery is needed design cover?
It is organized in these sections: A global outage alarm, From the alarm to the degraded infrastructure, Rolling back helped the module, but did not end the incident, Rebuilding part of the infrastructure, and What VTEX decided to change.
How long is VTEX 07/13/2023: rollback doesn't stop the cascade and disaster recovery is needed and what level is it?
It takes about 14 minutes to read and is rated Senior level.
Now design it yourself
Run this case as an interview in the System Design school: you make the decisions, the tutor challenges them and you see how your design scores.