Papers
Metastable Failures in Distributed Systems
Nathan Bronson, Abutalib Aghayev, Aleksey Charapko, Timothy Zhu · HotOS 2021 (Workshop on Hot Topics in Operating Systems), 2021
The problem: systems that break with no apparent defect
Drawing on about a decade of experience operating large-scale distributed systems, the authors describe a failure pattern that can bring down a service even without a hardware failure, a misconfiguration, or a software bug. These incidents resemble black swans: nothing in the history warns that they are going to happen, the impact is severe, and they are much easier to explain after the fact than to predict. On the surface, each incident seems unique, with a different trigger and different symptoms, but the paper argues that they all share a common structure, and therefore proposes a framework to describe them. The text applies this framework to examples lived by the authors themselves and surveys ad hoc fixes that teams created after the damage was done. The conclusion is honest: a systematic approach to building systems that are robust against still-unknown metastable failures remains an open problem. This is a workshop (HotOS) paper, that is, a position and conceptualization paper, not a new system evaluated with benchmarks. Its value lies in giving a name and a vocabulary to a type of outage that on-call engineers already knew but had difficulty classifying.
The core idea: trigger, vulnerable state, and sustaining effect
A metastable failure occurs in open systems, that is, systems with a load source the system does not control (clients that keep arriving). A trigger pushes the system into a bad state that persists even after the trigger goes away. What keeps the system stuck in that state is a sustaining effect: a feedback loop that amplifies work (for example, more request retries generating more load) and that usually exhausts some resource. That is why the paper argues that the real root cause is the sustaining loop, not the trigger: several different triggers (a load spike, a brief failure, a deploy) can lead to the same failure state, so addressing only the trigger prevents little. There is also the concept of a vulnerable state: the system works well under high load as long as no temporary overload happens, but a single trigger is enough to start the amplification and take it to the metastable state. Many production systems operate in a vulnerable state permanently, because it is more efficient than operating with slack. Getting out of the metastable state usually requires a strong corrective action, such as drastically reducing the load or restarting the system, and not just waiting for the trigger to pass.
Classic examples: retries and look-aside cache
Two sources of amplification appear as central examples. The first is retries: they mask transient failures and improve the experience in the common case, but add work precisely when the system is overloaded. Queues grow, latency rises, clients time out before receiving a response and resend the request, while the server spends capacity on requests that no one is waiting for anymore; the result is a system that is still up, but delivering little useful work. The second is the look-aside cache, which is an optimization of the common case. In an illustrative example used in summaries of the paper, with a 90% hit rate, only 1 in every 10 requests reaches the database, so the same database sustains about 3,000 QPS in total, but would only handle about 300 QPS if all requests reached it. If the cache is lost, the load on the database is multiplied by 10. Above 300 QPS the system is therefore in a vulnerable state: everything works, but a cache loss turns into an overload that the database cannot drain, sustaining the failure. The paper also mentions other mechanisms, such as slow error handling and emergent effects of load balancing.
Trade-off and mitigations: optimizing the common case creates fragility
The central trade-off is that optimizations for the common case (caching, retries, operating near the capacity limit) make the system more efficient and cheaper, but bring it closer to a vulnerable state in which a temporary overload triggers an amplification cycle. According to summaries of the paper, the mitigations discussed target the sustaining cycle rather than just the trigger: changing the system's policy during overload, prioritizing and enforcing fairness among requests, using fast error paths and outlier handling, and autoscaling. It also mentions aligning organizational incentives, so that teams are not rewarded for changes that improve day-to-day metrics but add fragility, and creating a signature metric that shows the state of the feedback loop, so as to predict failures before they occur, which brings distributed systems closer to control theory. The paper's own caveat applies: it presents ad hoc fixes and leaves a systematic approach open. A critic, Marc Brooker, also argues that the ideas have precedents in control theory and operations research, so the novelty lies more in the framing than in a new phenomenon.
Legacy: from idea to empirical evidence in real incidents
The paper's vocabulary was adopted and tested by later work. The study 'Metastable Failures in the Wild' (OSDI 2022) analyzed real incidents and found 22 metastable failures in 11 organizations, from large providers to small companies. It also reports that at least 4 of 15 major AWS outages in the last decade were caused by this type of failure, and extends the original model with two types of trigger and two types of amplification mechanism, confirmed with example applications that reproduce the failures in a controlled environment. A secondary summary of that study states that retry amplification was the most common sustaining effect, present in more than half of the incidents; this number comes from a secondary source and should be read with caution. In system design practice, the paper serves as a lens for reviewing common decisions in interviews and in production: retry policy with limits, protection against total cache loss, admission control and load shedding under overload, and attention to early signals such as latency outliers, which according to summaries of the paper often appear before the incident.
See the ideas at work
Practice the designs that use this paper in the System Design school.