Papers
Chaos Engineering
Ali Basiri, Niosha Behnam, Ruud de Rooij, Lorin Hochstein, Luke Kosewski, Justin Reynolds, Casey Rosenthal · IEEE Software 33(3):35-41 (maio-junho de 2016), 2016
The problem: leaving the system alone is not an option
The authors, from Netflix's Traffic and Chaos team, begin by recalling a maxim from Jim Gray: to improve availability, install proven software and hardware and then leave it alone. For companies that provide services over the Internet, this does not work, because they need to change constantly to add features and improve performance. At Netflix, engineers push new code and change configuration parameters in production hundreds of times a day, and availability still matters: a customer who cannot watch a video because of an outage may stop being a customer. The company's historical answer was Chaos Monkey, an internal service that randomly picks production virtual machine instances and terminates them, only during business hours, so that engineers can react quickly. The goal was to force the design of services that survive the loss of individual instances. It worked: today all engineers design services assuming this failure. Next came the Chaos Kong exercises, which simulate the failure of an entire Amazon EC2 region, and Failure Injection Testing (FIT), which makes requests between services fail to verify graceful degradation. The paper then defines Chaos Engineering as the discipline of experimenting on a distributed system to build confidence in its ability to withstand turbulent conditions in production, from a hardware failure to a malformed value in a configuration parameter.
The shift to the system perspective
The core of the proposal is to treat a set of services in production as a single system and understand it by injecting real-world inputs, such as transient network failures, request spikes, or malformed data, and observing what happens at the system's boundary. The traditional approach uses a functional specification: given a program f, inputs x produce outputs y = f(x), and if the system has several programs, the specification also defines each of them. The authors argue that, in real distributed systems, this specification is incomplete. The scale and complexity of user and infrastructure behavior make it hard to enumerate all inputs, and composing each service's specifications to reason about the whole is impractical. From the system perspective, the engineer starts from the assumption that the software has already been deployed, has active users, and observes metrics at the system boundary, being interested in the typical behavior of these metrics over time, the so-called steady-state behavior. This behavior can be altered by user interactions, by changes made by engineers (configuration, code), and by events such as the transient failure of a third-party service. The questions become 'does everything seem to work?' and 'are users complaining?', instead of 'does the implementation match the specification?'.
Principle 1: a hypothesis about the steady state and the SPS metric
An experiment requires specifying hypotheses, independent variables, dependent variables and context. The authors distill four principles: build a hypothesis around steady state, vary real-world events, run experiments in production, and automate them to run continuously. For the first, "works properly" is too vague, so the chosen quality attribute is availability. Netflix uses fallbacks for graceful degradation: if the cache in front of a service B fails, service A can query B directly, which increases latency and load but does not affect the user in the short term. If the bookmarks service fails, the interface simply starts videos from the beginning. The question that matters is whether users can find and watch some content, and this is operationalized by the SPS (stream starts per second) metric, how many playbacks start each second. SPS varies slowly and predictably over the course of the day, which reflects the premise that complex systems behave regularly enough to be predicted. The hypotheses focus on metrics visible at the system boundary, such as SPS, and not on fine-grained metrics like CPU load or database query time, which serve only to check that the system is actually healthy during the experiment and can end the test early. An example hypothesis: switching traffic from one region to another will have minimal impact on SPS.
Principle 2: vary real-world events
The happy path is the execution in which no input leads to errors or corner cases, and the temptation when testing your own code is to concentrate on it. In the real world, however, clients send malformed requests, dependent services send malformed responses, servers die, disks fill up, memory runs out, network latency spikes by orders of magnitude, and traffic surges without warning. The paper cites a study according to which 92% of catastrophic system failures resulted from incorrect handling of non-fatal errors. Hence the second principle: when designing experiments, sample the space of inputs that can actually occur. One obvious source is the historical data from past outages, an added reason to do post-mortems; but any input that can disturb steady state is a candidate. Examples used at Netflix: terminating VM instances, injecting latency between services, making requests between services fail, taking down an internal service, and making an entire Amazon region unavailable. Sometimes the event is simulated rather than injected: Netflix does not take an Amazon region offline, but acts as if it had gone down, redirecting requests. In other cases the event is applied to only a subset of users, as a risk-mitigation strategy. It is up to the engineer to balance the realism of the event against the risk of damage to the system.
Principles 3 and 4: run in production and automate
No single server has the computing power, memory, storage, bandwidth or reliability to host Netflix, so the service has to be a distributed system, and the authors expect that, with the popularity of microservices, more and more engineers will build systems like this. Traditional testing is not enough to find their failures. Two examples observed at Netflix: an overloaded server takes longer and longer to respond and a client queues outgoing requests in an unbounded local queue, which consumes memory until the client crashes; or a service behind a cache returns a transient error that is cached by mistake, and other clients receive that error. These are failure modes that require integration testing, and in some contexts that is only possible in production, because Netflix cannot reproduce the entire architecture for an end-to-end test. Even when it would be possible, you never reproduce everything: synthetic clients behave differently from real ones, and there are problems such as DNS configuration. The fourth principle comes from the fact that the system changes continuously (service behavior, new services, parameters, catalog metadata), which erodes confidence in old results. An experiment starts out manual, but it needs to be automated; the cadence depends on the context. At Netflix, Chaos Monkey runs continuously on business days and the Chaos Kong exercises happen once a month.
How to run an experiment and the future of the discipline
The paper summarizes an experiment in four steps. First, define the steady state as some measurable output that indicates normal behavior. Second, hypothesize that it holds in both the control group and the experimental group. Third, introduce variables that reflect real-world events, such as servers that crash, disks that fail, or network connections that are severed. Fourth, try to disprove the hypothesis by looking for a steady-state difference between the two groups. The hypothetical example is the bookmarks service, useful but not critical: SPS is used as the output, a small percentage of users is chosen and split into control and experimental; for the experimental group, all requests to the bookmarks service associated with those users are selectively failed, via the fault injection service (FIT); SPS is expected to be approximately equal in both groups. In the end, either there is more confidence in the system's ability to maintain its behavior, or a weakness has been revealed and points to a path for improvement. On the future, the authors say the concepts are not fundamentally new, but the practice is still in its infancy, and they list open fronts: case studies from other domains, organizational adoption, reusable tooling (Netflix's is internal and coupled to its infrastructure), and event injection models, since many failures result from combinations of events and not from isolated events.
See the ideas at work
Practice the designs that use this paper in the System Design school.