Live streaming (Twitch)

01 · CONTEXT AND REQUIREMENTS

Two paths in one product: video and chat

A live streaming platform receives video from broadcasters and delivers it, live, to many viewers. Twitch describes, in an engineering post, a peak of tens of thousands of simultaneous broadcasts delivered to viewers worldwide. Broadcasters send video over RTMP, and to scale delivery to many viewers Twitch uses HLS.

To study the problem, it is worth separating two paths: the video path, which carries many bytes from one broadcaster to many viewers, and the chat path, in which each viewer also sends messages. The two have different requirements and are designed separately.

02 · FROM INGEST TO PLAYER

The video path

RTMP was designed to stream video and audio over the internet and is used mainly in point-to-point communication. HLS, in turn, delivers video over HTTP and is compatible with large-scale HTTP cache infrastructure, which supports delivery to large audiences.

In the middle of the path sits the transcoder. It converts the incoming RTMP stream into HLS with several variants, each with a different bitrate, so that viewers with different bandwidths watch at the best quality possible for their connection. A master playlist gathers the variants, and the player switches between them to adapt to the network.

1. The broadcasting software sends the video to the ingest server over RTMP.

2. The transcoder converts the incoming RTMP stream into HLS, with several bitrate variants.

3. In this design, the segments and playlists of each variant are written to an origin, from which the CDN fetches them.

4. In this design, the CDN edge caches playlists and segments and serves the players; HLS is compatible with HTTP cache infrastructure.

5. The player receives the master playlist and switches between variants to adapt to network conditions.

This first version shows the whole flow, but leaves open what happens inside the transcoder.

03 · TRANSCODING

Aligned variants and the cost of the transcoder

Transcoding costs in two ways. H.264 compression is lossy, so re-encoding degrades video quality, and encoding is a computationally very expensive process, especially at high resolution and high frame rate. That is why Twitch says the ideal would be to transmux, that is, pass through without decoding and re-encoding, the highest variant, saving computation and preserving quality.

The problem with transmux is alignment. The HLS specification requires that the IDR frames of corresponding segments in each variant be aligned, with the same Presentation Timestamp, and that is what lets the player switch variants seamlessly when the network changes. But the IDR interval of the source stream is set by the broadcasting software's configuration, outside the platform's control. If the source only undergoes transmux, there is no guarantee that the variants' segments align, and Twitch observed playback pauses on Chromecast with misaligned segments.

1. The ingest receives the RTMP stream and hands it to the two processing paths.

2. The highest variant is only passed through, without re-encoding: it saves computation and preserves quality.

3. The other variants are re-encoded, and the encoder can force IDR at fixed intervals.

4. At the origin, there is no guarantee that the passed-through variant's segments align with those of the re-encoded ones.

5. The alignment that allows switching variants seamlessly is not guaranteed; Twitch reports playback pauses on Chromecast with misaligned segments.

There is also the cost of generating several variants. The simplest way is to create N independent transcoders, one per variant, but the ideal would be to eliminate redundant decoders and feed the rest of the pipeline with the images decoded by just one. Resizing is also expensive, and avoiding repeating it unnecessarily significantly improves performance. Since the encoder and scaler are very expensive, Twitch highlights the use of multi-core CPUs to process several variants at the same time.

FFmpeg does not meet some specific technical requirements of Twitch's operation, and the series of posts explains why it builds its own transcoder. With its own transcoder and player, the company controls the complete pipeline of ingest, transcoding, CDN, and playback.

04 · LIVE PLAYLIST

How the player follows an endless stream

In a live broadcast the playlist is not fixed: the client must reload it periodically to know which media is available, discovering the added segments. Per the HLS RFC, if the playlist changed the client waits at least the target duration before reloading; if it did not change, it waits half the target duration before trying again.

On the server side, while the broadcast has not ended, each new version of the playlist must bring at least one new segment and come out between half and one and a half times the target duration after the previous one. The server may limit segment availability by removing segments from the playlist, but a removed segment must remain available for a while, so as not to interrupt those still playing it.

1. The player requests the variant's playlist from the CDN edge.

2. In this design, the edge fetches from the origin the playlist and segments it does not yet have cached.

3. The server publishes new versions of the playlist, each with at least one new segment.

4. The player waits at least the target duration before reloading the playlist and, if it has not changed, half the target duration.

The player, for its part, must not choose a segment that starts less than three target durations from the end of the playlist, because that can cause stalls. It also must not reload playlists of variants it is not playing, to reduce load on the server.

05 · CHAT

Long-lived connections, bursts, and limits

For chat, the WebSocket RFC summarizes the earlier problem: bidirectional applications required abusing HTTP with polling to receive updates and separate HTTP calls to send, and every message from client to server carried an HTTP header. The proposal is a single TCP connection for traffic in both directions.

Twitch chat is exposed as an IRC interface based on a modified RFC1459 and the IRCv3 tags specification, and WebSocket clients connect over wss on port 443. The documentation recommends using EventSub subscriptions and API calls instead of IRC, which has limitations and is more complicated to parse.

Long-lived connections need maintenance. The server sends PING to find out whether the client is alive and closes the connection if it does not receive PONG. When it has to close for maintenance, it sends RECONNECT, and the normal path is to reconnect and rejoin the channels. The documentation also warns that message order is not guaranteed and that a message may be sent more than once.

1. Each viewer keeps a persistent connection to a chat server, chosen by the connection balancer.

2. On receiving a message, the server checks the room state, for example whether slow mode is on.

3. In this design, if the message passes the room's rules, the server publishes it to the room bus.

4. In this design, the bus delivers the message to all servers with viewers of the room, and each one relays it to its clients.

5. During maintenance, the server sends RECONNECT; the client's normal path is to reconnect and rejoin the room.

Chat also has limits. Messages sent in rapid succession are silently dropped by the server, which may even close the connection. In rooms with more than a thousand users, the bot stops receiving JOIN and PART messages. And moderators can enable room settings, such as slow mode and followers-only mode, which the server announces with ROOMSTATE.

06 · SUMMARY

Two paths, two protocols