Design a CDN

01 · CONTEXT AND REQUIREMENTS

The problem a CDN solves

A CDN is a network of servers spread around the world whose job is to deliver web content over HTTP and HTTPS efficiently, bringing what people consume close to where they are. Whoever requests an image, a script, or a video segment is served by a nearby server, not by the site's origin server.

To design one, start with the requirements. On the functional side, it must serve objects from nearby points of presence, fetch from the origin whatever it lacks, and allow a changed object to stop being served. On the non-functional side, what matters is low latency, a high share of requests served without touching the origin, protection of the origin from spikes, and service continuity when a point of presence fails. Each of these requirements becomes a decision in the following sections.

02 · REACHING THE EDGE

How the client finds the right POP

Before any caching, the client has to be brought to a nearby point of presence. One way is to use DNS: when resolving the site name, the answer points to the POP that best serves that client, typically the closest in latency.

The second way is anycast, in which several machines share the same IP address and routers send traffic to the nearest one. The benefit goes beyond speed: since the address is the same everywhere, an entire data center can be taken offline and traffic flows to the next nearest one. On the other hand, anycast is not trivial to implement.

1. Three POPs announce the same address. To the client, there is a single CDN address.

2. Routers send the browser's request to the nearest POP on the network.

3. The nearby POP goes down. It stops serving and the browser's next request needs another destination.

4. The network itself starts taking the browser to the next nearest POP, without changing the address the client uses.

There is also a third family of solutions, used by those who control what each server stores. Netflix's steering service chooses the appliances and generates the URLs the client will use, considering file availability, server health, and network proximity to the client.

03 · THE PATH OF A REQUEST

Hit, miss, and the cache layers

1. The browser resolves the name and the CDN's DNS indicates the most suitable POP for it.

2. The browser requests the object from the POP. If it is in the cache, the CDN returns it right there and the request ends at the edge.

3. If the POP does not have the object, it asks the layer above, the regional cache, before going to the origin.

4. If no layer has the object, the request reaches the origin, which fetches the original file from storage.

5. The object returns along the same path and each layer stores it for the next request.

The regional cache exists because a POP's space is limited. It is larger and keeps objects longer, and the POPs in a region share it, which eliminates repeated requests to the origin. This cache works as an intermediate layer and consolidates the requests that reach the origin. The effect is that the origin sees few connections, coming from few places.

There is a second protective effect: when identical requests arrive at the same time at a shared cache, it forwards a single request to the origin and reuses the response for all clients. This is called request collapsing, and it is what keeps a popular object that has just expired from generating a flood of requests to the origin.

04 · DATA FRESHNESS

How long to keep it and how to refresh it

Storing is easy; the hard part is knowing for how long. Each cached response has an age, the time since it was generated, and the maximum validity time works like the TTL of other caches. After that time, the response becomes stale, but it can be revalidated by asking the origin whether it is still valid, instead of being downloaded again in full.

For what never changes, the practice is to give the object a long life and put the version or a hash in the URL: when the content changes, the URL changes. When the response varies by language, for example, the Vary header makes the cache key combine the URL with the request header, and each variant is stored separately.

If you need to replace a file before it expires, there are two options: invalidate the object at the edges or publish a new version under a different name. Whoever updates files frequently should prefer versioned names.

05 · PULL OR PRE-POSITION

On-demand caching and proactive filling

Everything so far is the on-demand model: the object only reaches the edge after someone requests it. Netflix started from another premise for its network: it designed a proactive, directed solution, which it considers much more efficient than the traditional on-demand CDN. The appliances report their health and which files they hold on disk to the control plane, and most content updates are delivered to them in off-peak windows.

1. The appliances report their health and the files they hold on disk to the control plane.

2. In off-peak windows, the library fills the appliances with the content the control plane decided to place.

3. When someone asks to play a title, the request goes through the control plane.

4. The control plane considers file availability, health, and proximity, and generates the URL of the chosen appliance.

5. The device downloads the video straight from the appliance, without going through the origin.

Choosing between the two models is a trade-off. On-demand caching is simple and serves any content, but the first request for each object pays the path to the origin. Proactive filling avoids that first request, but requires predicting what will be watched and when, which only works for predictable content, such as a video catalog.