Netflix: circuit breakers and cascading failure (Hystrix)

01 · CONTEXT

Dozens of dependencies and just one of them slow

This case gathers what Netflix has published about how it protects its API's calls to other services. The API received more than 1 billion calls per day, which fanned out into several billion calls to dozens of subsystems.

According to the post, without fault-tolerance measures, 30 dependencies with 99.99% availability each would result in more than 2 hours of downtime per month.

The post calls the latent dependency the worst kind of failure for a subsystem: a single slow dependency, at high volume, blocks request threads and can quickly saturate all the container's threads, taking down the entire API.

1. The user calls the API, which in turn calls two dependencies over the network.

2. One of the dependencies starts responding with higher latency.

3. Requests waiting on that dependency keep the API's request threads blocked.

4. At high volume, the container's request threads can quickly be exhausted and the entire API goes down.

02 · ISOLATION

One compartment per dependency

Hystrix's answer starts with the pattern bulkhead (bulkhead): isolate dependencies from one another and limit concurrent access to each one. The documentation cites bulkhead, swimlane and circuit breaker as isolation techniques to limit the impact of any dependency.

In practice, each dependency call is wrapped in a command and each dependency gets its own thread pool, so that the latency of just one saturates only the threads of its own pool.

1. The API wraps each dependency call in a command.

2. Each dependency has its own thread pool.

3. Dependency B becomes slow and only the threads in its pool are exhausted; the API's request threads time out or are rejected immediately, without blocking.

4. With the pool full, new calls destined for B are rejected immediately instead of being queued.

5. The fallback returns an alternative response, such as cached data, instead of an error.

6. Calls to dependency A continue through its pool, without depending on B's saturated pool.

Why threads? The documentation acknowledges that it is possible to protect yourself without thread pools, but that requires trusting the client to always fail very fast and behave well. Since client libraries tend to be black boxes, opaque about implementation details, network access patterns and default settings, the team concluded that, in most cases, the benefits of isolating calls in separate threads outweigh the drawbacks.

03 · CIRCUIT BREAKER

Stop trying what is likely to fail

According to Martin Fowler, remote calls can fail or hang without a response until a timeout is hit, and Michael Nygard popularized the circuit breaker pattern to prevent this cascade. The idea of the pattern is simple: after failures reach a threshold, the breaker trips and subsequent calls return an error.

At Netflix, the circuit trips when the command exceeds an error threshold, for example 50% in a 10-second window, and then rejects all requests until health checks succeed.

The main goal is to relieve pressure on the failing systems and reduce request latency, by failing fast or returning a fallback when failure is likely.

1. With the circuit closed, the call follows the normal execution flow.

2. The breaker receives successes, failures, rejections and timeouts and keeps counters in a rolling window.

3. If the volume reaches the minimum and the error percentage exceeds the threshold, the circuit opens.

4. When open, the circuit short-circuits all requests: the command is not executed and the flow goes to the fallback.

5. After some time, the next request is let through to the dependency: this is the half-open state.

6. If that request fails, the circuit goes back to open for the duration of the sleep window; if it succeeds, it becomes closed.

04 · FALLBACK AND TUNING

What to respond when the call fails

Whatever the cause (timeout, rejection or short-circuit), the request always goes through the fallback logic before returning to the user. This gives the command the chance to do something other than fail fast, which throws an exception and makes the application shed load until the dependency becomes healthy again.

The post lists fallback approaches in order of impact on the user experience: read from a cache, even if the data is stale; queue writes to persist when the dependency comes back (eventual consistency); fall back to default values when personalized options cannot be retrieved; and return null or an empty list, which interfaces can ignore.

The documentation asks the fallback to give a generic response, with no network dependency, from in-memory cache or static logic, and calls a fallback that can fail a bad practice. The post adds that the fallback itself is protected by a semaphore, so that a slow implementation making a network call does not take down the entire application.

Fault-tolerance approaches have pros and cons, but combined they form a comprehensive protective barrier between user requests and dependencies. The table summarizes the role of four of them according to the post.

Timeouts at the command layer are the first line of defense, but network timeouts remain important because very slow calls could fill the dependency's pool indefinitely. In the post's example, the pool has size 10 to absorb a burst of requests at the 99th percentile, but with everything healthy it usually has only 1 or 2 active threads, serving calls with a median of about 40 ms.

Because settings can be changed in real time, adjustments do not risk taking down the entire application in case of a problem or a wrong setting.

05 · LEGACY

Hystrix in maintenance, ideas still in force

Hystrix grew out of the resilience engineering work that the Netflix API team began in 2011. Among its goals, according to the documentation, are preventing cascading failures in a complex distributed system and resorting to a fallback, degrading gracefully when possible.