Concepts
Resilience patterns: circuit breaker, bulkhead, timeout, and retry with backoff
Intuition
Think of a restaurant that depends on an ice supplier that has been running slow. You don't wait forever for the delivery (timeout), you try again but with ever-longer pauses instead of calling every second, and, if the supplier is clearly down, you stop calling for a while and use backup ice. On top of that, you separate the kitchen stations, so that a problem at the dessert counter doesn't lock up the whole dining room, like the bulkheads of a ship keep a hole from sinking everything. The goal is not to keep the supplier from failing, but to keep its failure from taking down the whole restaurant.
How it works
The first defense is the timeout: every remote call needs a deadline, because without one threads and connections get stuck waiting on a slow service and the problem propagates to whoever calls it. Retry recovers from transient failures, but it must be limited in the number of attempts and use exponential backoff with jitter (randomness in the interval), so that many clients do not all come back at the same time; unbounded, synchronized retries multiply the load on an already degraded service and create a retry storm that prevents recovery. Retry is only safe for idempotent operations (repeating produces the same effect as executing once) or with an idempotency key. The circuit breaker is a three-state machine: closed (calls pass through and the breaker counts failures and slowness), open (once a threshold is exceeded, it fails immediately without calling the dependency, giving it time to recover) and half-open (after an interval, it lets a few test calls through; if they succeed it goes back to closed, otherwise it reopens). The bulkhead isolates resource pools (threads, connections, queues) per dependency or per client type, so that the saturation of one pool does not consume the capacity of the others. The set is complemented by load shedding (rejecting part of the load early when the system is saturated, to serve the rest well) and by fallback or graceful degradation (returning cache, a default value or reduced functionality when the dependency fails). The Tail at Scale paper shows the controlled counterpoint: hedging only sends the second copy after a time close to p95, limiting the extra load to a few percent, whereas unbounded retries have no such ceiling. These patterns handle failures and saturation, while hedging and tied requests handle the latency variability of healthy replicas.
In practice
These patterns appear in libraries and service meshes that offer timeouts, retries with backoff, circuit breakers, and per-destination connection limits, and they are covered in references such as 'Release It!' by Michael Nygard, the Microsoft Azure Architecture Center cloud patterns catalog, and the AWS Builders' Library articles on timeouts, retries, and backoff with jitter. You choose them whenever a service depends on others over the network, starting with timeouts on all calls, retries only where there is idempotency and a retry budget, and circuit breaker and bulkhead on the least reliable or non-critical dependencies. Chaos experiments serve to verify that these mechanisms actually contain the failure in practice.
Use it in a design
Practice the cases that rely on this concept in the System Design school.