Incident · Mid
The 2020 Valentine's Day outage and the turn to chaos engineering
The company lays out the sequence: the outage, then fallbacks and circuit breakers, then manual game days, and finally LitmusChaos. It teaches how to prove a fallback works before the next peak.
Short answer
The company lays out the sequence: the outage, then fallbacks and circuit breakers, then manual game days, and finally LitmusChaos. It teaches how to prove a fallback works before the next peak. iFood is a meal delivery company that, according to the engineering post used in this case, delivers more than 60 million orders per month.
01 · CONTEXT
A platform that had already stopped being a monolith
iFood is a meal delivery company that, according to the engineering post used in this case, delivers more than 60 million orders per month. Growth over recent years led the company to break up the monolith into several microservices, which brought scalability but, in the post's words, also a cost: complexity and additional points of failure.
The same post starts from a general idea: it is fallacious to trust infrastructure components taken for granted. Database servers go down, message brokers crash, an entire region of a cloud provider can become unavailable due to a power outage, and network bandwidth can plummet without warning.
In the illustrative design, the application calls an API gateway.
The gateway forwards the call to the orders service.
The orders service is connected to a database and a message queue.
A consumer reads the queue and performs the asynchronous work.
02 · THE OUTAGE
What is known about Valentine's Day night
According to Canaltech, Valentine's Day (Dia dos Namorados) is a date when the app is in especially high demand, mainly because restaurants were closed due to the COVID-19 quarantine. That day, users could not complete meal purchase and delivery transactions.
| What Canaltech recorded | |
|---|---|
| Around 6:06 PM | Start of the instability |
| 19h06 | Peak, with 5856 malfunction reports on Downdetector |
| 19h46 | Complaints falling, to 3256 |
| 20h31 | New surge, with 4420 complaints |
Among the most reported problems on Downdetector were online purchases, app functioning, and general failure. On Twitter, the topic became a trending topic, with more than 16 thousand mentions.
In a statement to Canaltech, iFood said the platform experienced instability for two hours, affecting consumers, restaurants, and couriers, and apologized. In the same statement, it said it had made a mistake and was frustrated and dissatisfied.
03 · THE RESPONSE
Fallbacks and circuit breakers for crucial services
The engineering post says that, on Brazil's Valentine's Day in 2020, the platform collapsed like a house of cards. Right after the outage, the company decided to implement fallbacks and circuit breakers in all services crucial to delivering an order to the customer.
The illustrative diagram shows a client calling the orders service.
The orders service calls a dependent service.
The orders service also has a path that leads to a fallback, an alternative response.
In this illustration, the dependent service is marked as unavailable, while the orders service and the fallback have no failure marking.
iFood reports having implemented fallbacks and circuit breakers in all services crucial to delivering an order; the diagram shows the pattern in a single service.
The question that remained was how to ensure these mechanisms would work properly in critical events. Waiting to see what the future would bring was no longer an option.
A fallback only becomes a guarantee after it is exercised: without testing, it remains a hypothesis about the system's behavior.
04 · FROM MANUAL TO AUTOMATED
Manual tests in production and the adoption of LitmusChaos
The first step was simple: during low-traffic hours, the company gathered most of the engineering teams to take down parts of the production environment, mainly services and databases. The post says the approach helped tune the fallbacks and understand side effects of component outages.
The cost was high: the process involved many people and the tests were error-prone. The post illustrates the problem with questions such as how many pods a service had before the experiment and whether the service user's database permissions were restored after the tests, and concludes that this did not scale the way the company wanted.
Taking the practice further, the team wanted to know what would happen with a CPU spike, how to drop a persistent database connection, and how to simulate an AWS SNS outage. According to the post, the other chaos engineering tools were not ready for these questions, and LitmusChaos promised a wider range of experiments in a simple, controlled way. The company decided to adopt it as the de facto tool across all teams.
| Manual tests | LitmusChaos | |
|---|---|---|
| Who participates | Gathered most of the engineering teams | Adopted as the de facto tool across all teams |
| Point of attention | Involved many people and was error-prone | The post highlights authenticating users and restricting the services where chaos is injected |
| Scope of experiments | Taking down services and databases in production | Promised a wider range of experiments, in a simple and controlled way |
The post points to authenticating users in LitmusChaos and restricting the services where developers can inject chaos as an important aspect.
Each LitmusChaos project corresponds to a Kubernetes cluster, which limits a developer's experiments to the clusters where their services are.
LitmusChaos's audit details are important for knowing who ran the experiments.
05 · WHAT TO TAKE AWAY
Prove the mechanism works before the next peak
The reported sequence goes from the outage to fallbacks and circuit breakers, then to manual tests in production, and finally to a dedicated chaos engineering tool.
For system design, the lesson is that a resilience mechanism only deserves trust after being provoked in a controlled way. This calls for rules about who can inject failure, where it is allowed, and a record of who ran each experiment.
Frequently asked questions
What is The 2020 Valentine's Day outage and the turn to chaos engineering?
The company lays out the sequence: the outage, then fallbacks and circuit breakers, then manual game days, and finally LitmusChaos. It teaches how to prove a fallback works before the next peak.
What does the The 2020 Valentine's Day outage and the turn to chaos engineering design cover?
It is organized in these sections: A platform that had already stopped being a monolith, What is known about Valentine's Day night, Fallbacks and circuit breakers for crucial services, Manual tests in production and the adoption of LitmusChaos, and Prove the mechanism works before the next peak.
How long is The 2020 Valentine's Day outage and the turn to chaos engineering and what level is it?
It takes about 12 minutes to read and is rated Mid level.
Now design it yourself
Run this case as an interview in the System Design school: you make the decisions, the tutor challenges them and you see how your design scores.