Netflix Zuul 2: asynchronous gateway

01 · CONTEXT

The gateway that is Netflix's front door

According to the post from Netflix's Cloud Gateway team, Zuul 2 does the same thing as its predecessor: it acts as the front door of the server infrastructure, receiving user traffic. The main architectural difference is that it runs on an asynchronous, non-blocking framework, Netty.

The main benefit, expected from the start, is to allow persistent connections from devices and browsers to Netflix at Netflix scale. The post cites more than 83 million members, each with several connected devices, and calls this a massive scaling challenge.

In the InfoQ interview, the team explains the goal: polling accounts for a significant share of device requests to the microservices, and persistent connections make it possible to replace it with push notifications, as well as bidirectional communication between devices and control systems in the cloud.

02 · BLOCKING VERSUS ASYNCHRONOUS

One thread per connection or an event loop

Zuul 1 was built on the Servlet framework. These systems are blocking and multithreaded: they process requests with one thread per connection. The diagram shows what the post describes when backend latency increases.

1. Each device connection occupies a thread of the blocking gateway.

2. The origin starts responding more slowly.

3. Active connections and threads grow, including when devices retry requests because of errors.

4. At this point the node can enter a death spiral, in which accumulated threads raise the load and overload the cluster.

Asynchronous systems work differently: they generally use one thread per CPU core for all requests and responses, and the request lifecycle is handled by events and callbacks. Since there is no thread per request, the cost of a connection is that of a file descriptor and a listener, whereas in the blocking model the cost of a connection is a thread, with heavy memory and system overhead.

1. A few threads serve all connections, and each request's lifecycle is driven by events and callbacks.

2. Opening one more connection costs that of a file descriptor and a listener, not that of a thread.

3. The origin gets slower, as in the blocking scenario.

4. More connections and queued events cost far less than piled-up threads, so the gateway feels the impact less.

5. The team cautions that the resilience advantage did not come without effort and tuning.

03 · FILTERS

The request's path through the gateway

According to the project documentation, Zuul 2.0 is a Netty server that runs inbound filters, proxies the request with a Netty client and returns the response after the outbound filters.

1. The device's request enters the server and first passes through the inbound filters.

2. Inbound filters run before routing to the origin.

3. The endpoint returns a static response or, by default, routes the request to the origin.

4. When the origin's response arrives, the outbound filters come into play.

5. The response goes back to the device after the outbound filters.

Because the server runs on an event loop, the documentation states that it is critical never to block in a filter. If blocking is needed, it says to do it in an asynchronous filter, in a separate thread pool.

04 · MIGRATION

How Netflix got to the asynchronous gateway

The team started with the filters. Since blocking systems can run asynchronous code, it first changed the Zuul Filters and their chaining to run asynchronously. This way, the same filter logic runs on both a blocking and a non-blocking system. In the InfoQ piece, the team describes this as a large upfront investment, about six months or more, a prerequisite for building Zuul 2.

There are now two kinds of filters used together: asynchronous ones, for I/O operations, and synchronous ones, for logic operations that do not require I/O. According to InfoQ, the team used RxJava to chain the filters.

Many services in the Netflix ecosystem were built assuming blocking. Many libraries use thread-local variables to store request context, and that does not work when several requests are processed on the same thread. Much of the complexity of building Zuul 2 was finding out where those variables were used.

In the interview, the team says the gateway's role is to handle mainly roteamento, insights and normalization of incoming requests. One detour along the way was removing a lot of business logic from Zuul and moving it to the origin systems, which also simplified the migration of the blocking code.

05 · PERSISTENT CONNECTIONS

From polling to push

The post says Zuul benefits from greatly reducing the cost of network connections, which enables push and bidirectional communication to and from devices. The team expects that replacing verbose device protocols with push notifications will reduce overall cloud cost; these protocols account for a significant part of API traffic.

1. Devices poll the microservices, which the post says accounts for a significant share of device requests to the microservices.

2. With the asynchronous gateway, devices and browsers can keep persistent connections to the cloud at Netflix scale.

3. With the persistent connection, a control system in the cloud can send a message to the device by push, instead of the device polling.

4. The team planned to add http/2 and websocket support to the gateway.

The post reports that the team planned to add http/2 and websocket support to Zuul 2.

06 · RESULTS

What the change delivered

The results were mixed. According to the post, there was no significant efficiency gain from migrating to async and non-blocking, but the connection scale goals were met.

In the InfoQ piece, the team says it sees a 10-25% efficiency improvement on some clusters, not all, but was disappointed in this area, and attributes the miscalculation to the amount of CPU work done inside the gateway.

The team's conclusion is that the more CPU-bound a system is, the smaller the efficiency gain, and that the less work the system does, the more efficiency you gain with async.