Concepts
Steady-state hypothesis and chaos experiments
Intuition
It is like a fire drill in a building that is already full of people: instead of trusting the floor plan and hoping the emergency exits work, you cause a real problem, in a controlled way, and check whether day-to-day life stays normal for the people inside. Before the test you decide what 'normal' means (for example, how many people come in and out per minute) and, afterward, you compare. If nothing changed, you gain confidence; if something did, you discovered a weakness before a real fire revealed it.
How it works
The paper by Basiri and colleagues (Netflix) starts from the idea that the functional specification of a distributed system in production is incomplete, so the system is treated as a whole and observed at its boundary, through the typical behavior of metrics over time, the steady state. The experiment follows four steps: define the steady state as a measurable output that indicates normal behavior; hypothesize that it holds in both the control group and the experimental group; introduce variables that reflect real events (a server that goes down, a disk that fails, a severed network connection); and try to disprove the hypothesis by looking for a difference between the two groups. At Netflix, the metric is SPS (stream starts per second), which varies slowly and predictably throughout the day; hypotheses rely on metrics visible at the boundary such as SPS, while fine-grained metrics (CPU, database query time) serve only to check the health of the system and can end the test early. Four principles come out of this: build the hypothesis around the steady state, vary real-world events (not just the happy path), run in production, and automate to run continuously, since the system changes all the time and old results lose validity. To limit risk, the event can be simulated instead of actually injected, or applied only to a subset of users; the engineer balances the realism of the event against the risk of damage. In the bookmarks service example, a small percentage of users is split into control and experimental, FIT makes requests to the bookmarks service fail for the experimental group, and SPS is expected to be approximately equal in both groups.
In practice
At Netflix, Chaos Monkey randomly terminates production VM instances during business hours, continuously on weekdays, which led every engineer to design services assuming that loss; FIT makes requests between services fail to verify graceful degradation; and the Chaos Kong exercises, once a month, simulate the failure of an entire Amazon EC2 region. You would choose this approach when the system is too distributed to be reproduced in an end-to-end test outside production and there are fallbacks that need to be proven, starting with useful but non-critical services and with a few users. The authors themselves stress that the practice is still in its infancy and that Netflix's tools are internal and coupled to its infrastructure.
Use it in a design
Practice the cases that rely on this concept in the System Design school.