Cloudflare Pingora: replacing NGINX with Rust

01 · CONTEXT

The proxy between Cloudflare's network and the Internet

Pingora is an HTTP proxy that Cloudflare built in-house in Rust. It serves more than a trillion requests per day and, according to the company, needs only a third of the CPU and memory of the previous proxy infrastructure.

The service proxies traffic between Cloudflare's network and servers on the Internet, and powers the CDN, Workers fetch, Tunnel, Stream, R2, and other products. Pingora serves nearly every HTTP request that needs to talk to an origin server, such as on a cache.

02 · NGINX LIMITS

Why NGINX stopped being enough

Cloudflare states that, at its scale, it could no longer get the performance it needed, and that NGINX also lacked the features required by the company's complex environment.

The first limit is architectural. In NGINX each request can be served by only a single worker, which creates unbalanced load across CPU cores and leads to slowness. In addition, requests that do heavy CPU work or blocking IO can delay others.

The most critical problem for Cloudflare is poor connection reuse. Reusing connections from a pool speeds up TTFB because it skips the TCP and TLS handshakes of a new connection. But NGINX's pool is per worker: when adding workers to scale, connections spread across more isolated pools and the reuse rate worsens, resulting in slower TTFB and more connections to maintain.

1. An incoming request is served by a single NGINX worker.

2. Each worker keeps its own connection pool to the origin.

3. When adding workers to scale, connections end up spread across more isolated pools and the reuse rate worsens.

There is also the feature limit. When redoing or redirecting a request (retry or failover), the company sometimes wanted to send it to another origin with different headers, something NGINX does not allow. NGINX's core is C only, which is not memory safe by design, and the Lua that complements it is less risky but less performant.

03 · THE DECISION

Three paths and a rewrite

The company evaluated three options every quarter, for a few years. During that period it followed the path of least resistance, stretching NGINX further, and at some point the return on building its own proxy seemed worth it.

04 · DESIGN DECISIONS

Rust, custom HTTP, threads, and programmable phases

The language chosen was Rust, because it does what C does in a memory-safe way, without losing performance. Cloudflare also built its own HTTP library, to maximize flexibility in handling traffic, since it has to support a lot of bizarre, non-RFC traffic.

For work scheduling, the company opted for multithreading instead of multiprocessing, to share resources, especially the connection pools. It also decided that work stealing was needed to avoid some classes of performance problems, and the Tokio async runtime fit the needs.

Finally, it adopted an event-based programmable interface, similar to that of NGINX/OpenResty. The request filter phase, for example, lets you run code to modify or reject the request when a header arrives.

05 · ON THE REQUEST PATH

Where Pingora fits in

1. Pingora handles the stretch between Cloudflare's network and the servers on the Internet.

2. When the request needs to talk to an origin server, such as on a cache miss, Pingora is what serves it.

3. In Cache Reserve, Pingora uses R2 storage as a cache layer.

4. Inside Pingora, the threads share the connection pools to the origins.

The new foundation also sped up features. Cloudflare was able to add upstream HTTP/2 support without major obstacles, and this made it possible to offer gRPC. Cache Reserve is another example, in which Pingora uses R2 as a cache layer.

06 · RESULTS

What changed in production

According to Cloudflare, the TTFB savings do not come from running faster code, but from the new architecture, which shares connections across all threads, improves the reuse rate, and spends less time on TCP and TLS handshakes.

Part of the CPU savings comes from the Rust code running more efficiently than the old Lua code: in Lua, reading a header requires reading from the C struct, allocating a Lua string, and copying it. Another part comes from opening fewer new connections, because TLS handshakes are expensive compared to using connections that are already established.

On security, the company says it has served a few hundred trillion requests without a single crash caused by the service's own code, and that Rust's memory-safe semantics protect against undefined behavior.