Globo.com: live video for the World Cup and BBB

01 · CONTEXT

A peak everyone knows is coming

Globo.com operates a live video platform that, according to the dissertation used in this case, has been used at Rock in Rio and in several editions of Big Brother Brasil. A summary of a talk by the team itself also cites live video distribution at the World Cup, at BBB, and at other shows of the group.

At events like these, the whole audience arrives at the same time, for the same content. This changes the problem compared with an on-demand catalog, and it is what makes the case useful for studying edge content delivery e horizontal scale.

The most concrete publicly available data comes from a UFJF dissertation that analyzed Globo.com server logs during the broadcast of all matches of the 2013 Confederations Cup and the 2014 World Cup. The InfoQ talks have only the public summary.

02 · ARCHITECTURE

Two PoPs, DNS in front and Nginx inside

According to the dissertation, the monitored live video architecture uses Anycast and DASH. The infrastructure has points of presence in two cities, Rio de Janeiro and São Paulo, and it is the DNS server that decides which PoP to route each client to. Within each PoP, requests are balanced across the servers, which run Nginx.

1. In the illustrative design, an encoder and segmenter appears connected to the servers at each point of presence.

2. In the design, the player appears connected to the DNS, which, according to the dissertation, decides which PoP to route each client to, in a system that uses Anycast.

3. The points of presence are in two cities, Rio de Janeiro and São Paulo.

4. In each PoP, requests are balanced across the servers, which run Nginx.

5. Because DASH divides the media into segments of several qualities, the client picks on its own, during playback, among the available alternatives.

As advantages of anycast, the dissertation lists lower latency from the proximity between client and server and network resilience. For load balancing within the PoPs, see load balancing.

03 · THE PEAK

What the logs show about the load

The dissertation concludes that sporting events can attract hundreds of thousands of simultaneous viewers. At the 2013 Confederations Cup, the servers handled up to approximately 180 thousand simultaneous requests, and in the five matches with the highest data volume the transfer rate approached 15 GB/s.

The audience does not vanish at halftime: the dissertation records that the number of connected users stays high even in those periods, when highlights are broadcast.

04 · ADAPTIVE QUALITY

Reducing quality to serve more people

The Confederations Cup match with the highest data volume was not the one with the highest number of clients, which was Brazil vs. Uruguay. According to the dissertation, the quality options were reduced so the provider could serve a client demand that had not been forecast.

1. In the illustrative design, the player requests content from a streaming server.

2. The server reads playlists and segments from a storage, which in the design holds several qualities.

3. According to the dissertation, if there is no bandwidth available the server should reduce the media quality and, with it, the size of the video chunk.

4. In the match with the largest audience, the best resolution had chunks smaller than 500 KB; in the one with the highest data volume, the segments reached 1400 KB.

05 · BALANCING

Two PoPs, uneven load

At the Confederations Cup, the São Paulo servers had extra load compared with those in Rio, with the difference ranging between 70% and 120% and occurring in all matches. According to the planning reported by Globo.com, the bandwidth available at the two distribution points is practically the same.

1. In the design, the two PoPs appear side by side, each with its own load balancer and servers.

2. The dissertation records that the São Paulo PoP had extra load in all matches, between 70% and 120%.

3. According to the reported planning, bandwidth is practically the same at both points.

06 · STACK AND LIMITS

Nginx, Redis, and what is not known

A talk summary describes the platform as built in more than seven languages and on an almost entirely open-source stack, including Nginx, Redis, and Cassandra.

According to a summary of an NGINX case study, the previous RTMP-based stack limited mobile support, caching, and load balancing, and Globo.com migrated to HTTP-based delivery built around NGINX.