Facebook Live: live video for millions

01 · CONTEXT

Why live video is a different problem

Public figures on Facebook can have millions of followers trying to watch the same video at the same time. The engineering post calls thundering herd the case in which too many requests trample the system, causing delay, interruptions, and stream disconnection.

According to the High Scalability write-up of Federico Larumbe's talk, live video traffic is spiky. Live videos appear at the top of the feed, which increases the chance that they are watched, and notifications go out to all of each page's fans. This pattern causes problems in the systems for cache and for load balancing.

As an order of magnitude, the same write-up cites a watermelon video that reached a peak of more than eight hundred thousand concurrent viewers.

02 · THE VIDEO PATH

From the broadcaster's phone to the player

The High Scalability description summarizes the flow in a few steps: the phone sends an RTMP stream to a Live Stream server, which decodes the video and transcodes it to several bitrates; for each bitrate, a set of one-second MPEG-DASH segments is produced continuously.

1. The broadcaster's phone sends the video over RTMP to a Live Stream server.

2. The server decodes the video and transcodes it to several bitrates.

3. For each bitrate, one-second segments are produced continuously and stored in a datacenter cache.

4. From the datacenter cache, the segments go on to caches located at points of presence.

5. The viewer's player fetches the segments from a PoP cache, at a pace of one per second.

The write-up points to two multiplication factors. One sits between the datacenter cache and the many PoP caches: users access the PoP caches, not the datacenter. The other sits inside each PoP, which has a layer of HTTP proxies and another of cache. Different segments sit in different caches, which helps balance the load across the cache hosts.

03 · PROTECTING THE ORIGIN

Layered caches and request coalescing

Instead of clients connecting directly to the stream server, there is a network of edge caches spread around the world. In the implementation described in the post, the video is split into three-second HLS segments. The request for a segment is handled by an HTTP proxy in an edge datacenter, which checks whether the segment is in the edge cache; if it is not in the edge cache, the proxy requests it from the origin cache, another layer with the same architecture, and only then, if it is missing, from the stream server.

According to the post, more than 98 percent of segments are already in an edge cache near the user. Even so, about 1.8 percent of requests got past the edge cache, and with a million viewers that is still a large number.

1. Players request the same segment, at the same time, from HTTP proxies in an edge datacenter.

2. The edge cache returns a miss only to the first request and holds the following requests in a queue.

3. If the segment is missing, the proxy requests it from the origin cache, another cache layer with the same architecture.

4. If the origin cache does not have the segment either, the request goes on to the server that handles that stream.

5. The server returns the segment, which is stored in each layer for the following clients.

6. The requests that were in the queue are answered from the edge cache, as hits.

The post calls this technique request coalescing. Many people request the same segment at the same time, it may not yet be in cache, and so the edge cache returns a miss to the first request and holds the following ones in a queue. The origin cache runs the same mechanism for requests from several edge caches, since the same object can be requested by one edge cache in Chicago and by another in Miami.

The High Scalability write-up describes a second risk inside the PoP: if all the viewers were sent to a single cache host to wait for the segment, that host could become overloaded. The solution described is a cache layer in the proxy itself, so that only the first request to the proxy reaches the cache and the following ones are served directly by it.

04 · GLOBAL BALANCING

Keeping a spike from taking down a PoP

With the datacenter protected from the thundering herd, the PoPs are still at risk. According to the write-up, Live's spikes are so large that a PoP can become overloaded before the load measurement reaches the global load balancer.

A system called Cartographer maps Internet subnets to PoPs and measures the delay between each subnet and each PoP. Each PoP's load comes from counters in the proxies, aggregated per PoP, and each user is sent to the closest PoP that has enough capacity. This is an optimization problem with capacity constraints that minimizes latency.

1. Users from many subnets need to be taken to some PoP; Cartographer decides which.

2. Cartographer measures the delay between each subnet and each PoP.

3. Counters in each PoP's proxies measure the incoming load, and they are aggregated to learn each PoP's load.

4. A capacity estimator extrapolates each PoP's previous and current load to the future load.

5. Each user is sent to the closest PoP that has enough capacity.

In control systems there is a delay to measure and a delay to react. The load measurement window was reduced from a minute and a half to three seconds, but that window still exists. The solution described is to predict the load before it happens: a capacity estimator extrapolates each PoP's previous and current load to the future load.

For the interpolation, the write-up cites cubic splines, which predict more complex traffic patterns than linear interpolation. The first and second derivatives are taken: positive velocity indicates load rising, and negative acceleration indicates velocity decreasing until it reaches zero, when the load starts to fall. The interpolation reduces error and oscillation. At the time of the talk, the prediction used the last three thirty-second intervals.

05 · LATENCY

RTMP: push instead of pull

Building Live for ordinary people was an exercise in reducing latency. To reach two to three seconds of broadcast delay, Facebook decided to use RTMP, a protocol that keeps a persistent TCP connection between player and server throughout the broadcast.

According to the post, the push model plus the small chunks reduce the delay between broadcaster and viewer by five times. Most live products use HLS because it is HTTP-based and easy to integrate with existing CDNs; Facebook implemented RTMP by modifying the nginx-rtmp module and developing an RTMP proxy. The High Scalability write-up sums up the cost: RTMP requires a new architecture because it is not HTTP-based.

Upload is also a challenge. The write-up takes as an example available bandwidth between 100 and 300 Kbps, when audio requires 64 Kbps and standard-definition video requires 500 Kbps. The video bitrate is adjusted according to the available network bandwidth; the decision is made on the phone, by measuring bytes sent over the RTMP connection and taking a weighted average of the last intervals.

06 · TESTING AND NEXT STEPS

How to validate and where to evolve

For testing, a load-testing service distributed across the PoPs was built, which simulates live traffic and can simulate ten times the production load. According to the write-up, it helped find problems in the capacity estimator, tune parameters, and verify that the cache layer solves the thundering herd.

As a future direction, the write-up cites investigating a push mechanism, using HTTP/2, to push segments to the PoPs before they are requested.