Concepts
Retries, overload, and metastable failures
Intuition
Imagine a packed restaurant where the kitchen falls behind. Impatient customers give up without telling the kitchen and place another order, and the kitchen starts preparing dishes that no one is waiting for anymore; the more it falls behind, the more repeated orders arrive, and the delay gets worse. Even if the original cause of the delay has disappeared, the restaurant stays swamped, because the customers' own reaction sustains the chaos. Repeating an order helps when the problem is small and transient, but makes everything worse when the house is already overloaded.
How it works
A metastable failure happens in open systems, where clients keep arriving without the system controlling the load: a trigger (load spike, brief failure, deploy) pushes the system into a bad state that persists after the trigger goes away. What sustains that state is a sustaining effect, a feedback loop that amplifies work and usually exhausts some resource. For Bronson et al., the root cause is that loop, not the trigger: different triggers lead to the same state, so treating only the trigger prevents little. Retries are the central example: queues grow, latency rises, the client times out and resends, and the server spends capacity on requests nobody is waiting for anymore, staying up but delivering little useful work. Many systems run permanently in a vulnerable state, because operating close to the limit is more efficient than having slack, and a temporary overload is enough to trigger the loop. Leaving the state usually requires strong action, such as drastically reducing load or restarting, and not just waiting. The Tail at Scale shows the other side: in systems with high fan-out, a small fraction of slow requests is inevitable and dominates tail latency, which is why redundancy is useful, but with techniques designed to limit the extra load: in hedged requests, the second copy is sent to another replica only after a time close to the expected 95th percentile, the first response wins and the other is canceled, which keeps the additional load at a few percent; in tied requests, the copies cancel each other in the queues. The difference between a retry that helps and one that takes the system down is therefore whether the redundancy is bounded and cancelable or grows uncontrolled when the system is already slow.
In practice
When designing calls between services, treat the retry policy as something that needs explicit limits, along with admission control and load shedding under overload, and use hedging only with a calibrated delay and cancellation of the redundant copy. The papers also suggest watching early signals, such as latency outliers and high percentiles (p99, p999), at the points where parallel calls converge. It is the right lens in interviews and design reviews whenever the question is what happens when the system is already slow and all the clients retry.
Use it in a design
Practice the cases that rely on this concept in the System Design school.