Netflix: chaos engineering

01 · CONTEXT

From database corruption to the cloud

Netflix's journey to the cloud began after a major database corruption that, for three days, prevented DVDs from being shipped to members. The lesson was to abandon vertically scaled single points of failure, such as relational databases in the datacenter, and move to distributed, horizontally scalable systems.

AWS was chosen as the provider, and the architecture went from a monolithic app to hundreds of microservices. The migration only finished in early January 2016, after seven years of effort.

02 · WHY BREAK THINGS

Break on purpose so it does not break by accident

If instance failures are certain, waiting for them to happen by accident lets the team discover the weakness at the worst possible time. Chaos Monkey does the opposite: it randomly selects instances that host production services and terminates them, so that engineers design services that withstand the loss of an instance.

It is only active during business hours, so that people are on hand to respond if something fails. That matters because the company changes code and runtime configuration hundreds of times a day, so fragility can reappear with any change. It was a cultural success: today, inside the company, everyone designs their services to tolerate instance failure as a routine matter.

1. Clients reach the load balancer, which splits requests among the service's instances.

2. The chaos agent picks instance B at random and terminates it during working hours.

3. The load balancer stops sending traffic to instance B and instances A and C absorb the load. The user does not notice, and that is what the design must guarantee.

4. Monitoring records the loss and the engineers, present because it is business hours, can act if the service degrades.

Note what the diagram assumes: the terminated instance held nothing that could not be recreated, and the load balancer knows to stop using it. Services without irreplaceable local state and health checks are what make the death of an instance a mundane event. The source does not detail these mechanisms, so this is the author's reading of what Chaos Monkey requires.

03 · SIMIAN ARMY

From one instance to the army of failures

Chaos Monkey became part of a larger suite, the Simian Army, described as services (the Monkeys) in the cloud that generate various kinds of failure. The public repository lists Chaos Monkey, Janitor Monkey and Conformity Monkey among the members, and it is now archived and read-only.

The scale of failure rises in steps. Chaos Monkey terminates instances; Chaos Gorilla takes down an entire AWS availability zone; and Chaos Kong simulates an outage that affects an entire region.

04 · FROM INSTANCE-KILL TO REGION

Chaos Kong and region evacuation

Netflix's control plane runs on AWS, replicated across three geographic regions: Northern Virginia, Oregon and Ireland. If an outage is limited to one region, it is possible to redirect that region's clients to a healthy region.

1. In normal operation, global routing splits clients among the three regions, all reading replicated data.

2. In the exercise, the Virginia region is treated as lost. The source clarifies that the region is not actually shut down: the team acts under the assumption that it went down.

3. Global routing now sends the lost region's clients to the other two.

4. Oregon and Ireland receive the extra load. This only works if reserved capacity exists: Netflix states that it maintains capacity for a full shift of traffic from one region to another.

5. During the exercise, the hypothesis is that the steady-state metric, stream starts per second, suffers minimal impact.

Chaos Kong runs as a scheduled exercise, once a month, while Chaos Monkey runs continuously on weekdays. This cadence reflects the risk: the regional exercise is rare and planned, while instance death is routine.

From an architecture standpoint, evacuating a region requires that the other regions already have the data and the capacity to serve the user. That is why the regional test is both proof and design pressure for multi-region architecture and for availability patterns.

05 · PRINCIPLES AND EXPERIMENTS

From vandalism to the scientific method

Over time, the team realized the practice was subtler than breaking things in production, and named it chaos engineering: the discipline of experimenting on a distributed system to build confidence in its ability to withstand turbulent conditions in production.

The first step is to define a steady state: a measurable output that indicates normal behavior. At Netflix, the main health indicator is stream starts per second (SPS), which measures whether users can find and watch content. The hypothesis is always about a metric at the system's boundary, not about an internal detail.

1. Two small groups of users enter the experiment: control and experimental. The hypothesis is that the steady state will be the same in both.

2. The gateway fails only the calls to the bookmark service made on behalf of the experimental group. The rest of the users never see the failure, which limits the blast radius.

3. The playback service uses a fallback: without the bookmark, the video starts from the beginning instead of resuming where the user left off. It degrades, but does not break.

4. The steady-state metric of the two groups is compared. If there is no relevant difference, the hypothesis was not refuted and confidence increases; if there is, a weakness was found.

The principles complete the method: vary real-world events, experiment in production and automate to run continuously. In production because the system behaves differently by environment and traffic, and because at Netflix the whole architecture cannot be reproduced in an end-to-end test. Automate because confidence in old results falls as the system changes.

There is also a realism cost: whoever designs the experiment must balance the realism of the event against the risk of harm, which is why Netflix simulates the loss of the region instead of shutting it down. The monitoramento of the boundary metric, in turn, is what makes it possible to abort early.

06 · WHAT CHANGES IN THE ARCHITECTURE

Designing for failure as the normal case

The long-term effect of chaos is architectural. Because instance death became routine, services are designed to tolerate machine loss. Because failures between services are injected, every non-critical dependency gets a fallback and the system learns to degrade gracefully. And because the region can go out, data and capacity are replicated across regions.

07 · LESSONS AND LIMITS

When not to do chaos

Netflix's source already builds in the safety conditions. Chaos Monkey only runs during business hours so that people are on standby; the blast radius must be contained; and new experiments start manually and are only automated afterward. A real integration failure they cite, an unbounded local queue that consumes all of the client's memory, shows why traditional tests are not enough.

Finally, the change is as much cultural as technical: Chaos Monkey's gain was to make fault tolerance a requirement for every engineer, not a project for a specialist team. Replicating the tools without the same discipline is copying the form without the effect.