Papers
Globally Distributed Content Delivery
John Dilley, Bruce Maggs, Jay Parikh, Harald Prokop, Ramesh Sitaraman, Bill Weihl · IEEE Internet Computing, vol. 6, no. 5 (Sept-Oct 2002), pp. 50-58, 2002
The problem: flash crowds and the limits of clusters, mirroring, and multihoming
The paper starts from the flash crowd problem: when a site becomes popular, the request load can overwhelm some point in the chain, whether the front web server, the network equipment, the bandwidth or, on more advanced sites, the back-end transactional infrastructure. The result is a site outage or high response times, which translates into lost revenue and reputation. Akamai was born from MIT research on this problem, starting from the observation that serving content from a single location creates scalability, reliability and performance problems. The authors review the existing approaches: local clustering, mirroring (clusters in a few locations) and multihoming (multiple ISPs). None solves everything. It is hard to scale clusters to thousands of servers; with multihoming, BGP does not converge quickly when a connection fails; and mirroring requires synchronizing the mirrors. Moreover, all three require idle capacity: the cluster must withstand the peak (which can be an order of magnitude above the average), each multihoming connection must carry all the traffic and each mirror must support the entire load, which can more than double the initial infrastructure cost. Independent proxy caches help, but, according to the text, their hit rates tend to be low (25 to 40 percent), partly because of dynamic content.
The core idea: edge servers and mapping via DNS
Akamai's answer is to serve requests from a variable number of surrogate origin servers at the edge of the Internet. At the time of the paper, the system used more than 12,000 servers in more than 1,000 networks. The principle is to direct each client to the server that is at once the closest, available and most likely to have the content. Close depends on topology and link dynamics (lowest RTT and lowest packet loss). Available depends on the load and bandwidth of the server and the data center. Likely depends on which servers hold each customer's content: if all of them served everything, as in round-robin DNS, disk and memory would be consumed by the most popular objects. That is why the system distributes content across the minimum number of servers at each site. This direction is called mapping and uses a dynamic, fault-tolerant DNS system that resolves the hostname according to the requested service, the user's location and the state of the network. Resolution goes through top-level DNS servers, which delegate to low-level servers at the location closest to the client. TTLs are short (from a few seconds to a minute at the final resolution), to allow clients to be redirected when conditions change.
Mapping criteria, consistent hashing, and monitoring
To map a request to a server, Akamai's name servers combine criteria: the requested service (do not send a QuickTime stream to a server that only serves HTTP), the server's health, its load (CPU, disk and network), network conditions (low packet loss and enough bandwidth in the data center), the client's location, and whether the server is likely to have the content, which is decided by a consistent hashing algorithm. To learn the topology, Akamai agents exchange BGP messages as peers of certain edge routers, and the number of hops between autonomous systems serves as a rough measure of distance, combined with live data such as traceroute. Monitoring feeds load balancing: each server reports its load to a monitoring application, which informs the local DNS. If the load exceeds a threshold, DNS assigns part of the content to more servers; past another threshold, the server's IP stops being returned to clients. Agents that simulate users downloading objects measure failures and download times and make it possible to detect and suspend problematic data centers or servers automatically. Monitoring also feeds near-real-time reports on the content served.
Dynamic content at the edge: ESI, split connections, and content control
Ordinary proxy caches cannot cache dynamic content, for example a static page that includes an ad personalized per profile. For this, Akamai uses Edge Side Includes (ESI), a technology that assembles dynamic content on edge servers. Similar to server-side include languages, ESI adds fault-tolerance features (for when the origin is unavailable) and integrates an XSLT engine. The provider splits the page into fragments with independent cacheability; they are stored as separate objects in the edge cache and assembled dynamically on each request, so only uncacheable or expired fragments are fetched from the origin. In the dynamic sites studied (portals and financial sites), the paper reports a reduction of 95 to 99 percent in origin bandwidth. For uncacheable objects, the edge server splits the TCP connection in two, client-edge and edge-origin, which can speed up the response because the edge reacts faster to packet loss. As for the consistency of cacheable objects, TTLs or versioned URLs are used, and the servers accept on-demand purges. The paper also covers authorization, integrity checking before serving each response block, log visibility, and billing.
Reliability, operational challenges and legacy
The authors list the challenges of building and operating the system. On scalability: monitoring tens of thousands of distributed servers with minimal bandwidth, generating new maps every few seconds without adding delay to DNS (the ideal-server computations are done offline, without making the user wait), dealing with incomplete or outdated information, isolating customers from one another, and aggregating logs for billing. On reliability: the principle is to eliminate single points of failure. Some of the equipment in a large network fails every month, so failures must be detected and units taken out of service. DNS detects failures quickly and hands out new IPs; for resolvers with cached responses, resolution can return multiple IPs or a live server at the site can take over the failed one's IP. Replication is massive, both for content delivery and for DNS. The software cannot be updated atomically, so it is deployed in stages and two versions coexist in the network. The conclusion points to the next step: running applications at the edge, with challenges of visibility, isolation (sandboxing) between customers' applications, and data distribution, the conceptual basis of today's edge computing.
See the ideas at work
Practice the designs that use this paper in the System Design school.