Video conferencing (Zoom)

01 · CONTEXT

A many-to-many system

A video conference is a real-time media system in which each participant sends their audio and video and receives those of the others. The central question of the design is where this media goes through: directly between participants or through an intermediary that distributes it.

Zoom's security whitepaper describes a division that is useful for the design. Zoom operates in hybrid mode: web services such as meeting setup, user management, recordings, chat transcripts, and voicemail recordings are hosted in the cloud, while real-time conference media is processed in colocation and commercial cloud data centers distributed globally.

A distributed network of low-latency software multimedia routers connects Zoom's infrastructure. Through these Multimedia Routers, session data leaving the host's device and reaching the participants' devices is dynamically routed between the endpoints.

02 · TWO PLANES

Control on one side, media on the other

In session setup, the Zoom client connects over HTTPS to Zoom servers to obtain the meeting information and assess the network environment, such as the appropriate Multimedia Router, which ports are open, and whether an SSL proxy is used. With this metadata, the client decides the best real-time communication method, trying to connect on the preferred UDP and TCP ports. For compatibility with corporate SSL proxies, the connection can also be made over HTTPS.

1. The client connects over HTTPS to Zoom servers to obtain the meeting information and assess the network environment.

2. Among the things the client assesses is the appropriate Multimedia Router, in addition to the open ports and the use of an SSL proxy.

3. With this metadata, the client chooses the real-time communication method, trying the preferred UDP and TCP ports.

4. Session data leaving the host's device and reaching the participants is dynamically routed between the endpoints.

03 · MEDIA

Mesh and mixer: two extremes

In a mesh, a participant (A) needs to send its RTP streams and its RTCP packets to each of the others (B and C), each over its respective transport flow. If everyone does the same, everyone has a joint view of the session.

1. Participant A sends its RTP and RTCP streams to B and to C, each over its own transport flow.

2. If everyone does the same, each participant has a joint view of the session.

3. The cost shows up on the endpoint's link: in a mesh with N endpoints, N-1 transmitted packet streams cross the first link from it.

The RFC on RTP topologies points out the cost: in a mesh with N endpoints, N-1 transmitted packet streams cross the first link from the endpoint. Discord also notes that peer-to-peer becomes prohibitively expensive as the number of participants grows, and large group channels require a client-server architecture.

At the opposite extreme is the mixer. In an audio mixer, the intermediary needs to decode the N streams coming from the endpoints; the mixed streams are encoded again, packaged into RTP, and sent. Each endpoint receives a mix of all the other sources, except its own.

A video-switching MCU, in turn, forwards to each participant a single media stream, selected from among the available streams.

04 · SFU

Forward without mixing

In the Selective Forwarding Middlebox (SFM), the intermediary selects which parts of a scalable stream (or which stream, in the case of simulcast) to forward to each receiving endpoint. The decision may depend on factors such as the available bitrate and the desired layout.

The RFC places the SFM in recent video conferencing systems, which take advantage of scalable video coding and simulcast. In contrast to transcoding MCUs, SFMs have extremely low delay and offer high-end system features, such as custom layout, without signal processing in the middlebox.

SFMs are also able to escalar to a large number of simultaneous users and, because they have very low delay, can be chained.

05 · IN PRACTICE

An SFU in production: Discord's voice

At Discord, the voice server has a signaling component and a media relay component, the selective forwarding unit (SFU). Discord's own SFU is responsible for forwarding audio and video traffic within channels.

1. When online, the client maintains a WebSocket connection to a Gateway and updates its voice state through it.

2. Each voice server periodically reports its health and load to the service discovery system.

3. The Guilds server assigns the guild the least-used voice server in the region.

4. The voice server's SFU is responsible for forwarding audio and video traffic within the channel.

5. When offending users are server-muted by moderation, their audio packets are dropped.

In terms of assignment, the online client maintains a WebSocket connection to a Gateway and updates its voice state through it. Each voice server periodically reports its health and load, and the Guilds server assigns the guild the least-used voice server in the region. When offending users are server-muted by moderation, their audio packets are dropped.

Routing media through servers allows moderation: administrators can disable audio and video of participants who are offending. Since every client connects to the media relay server, Discord says it does not need ICE. All voice channels in a guild are assigned to the same voice server.

Even very large voice channels have few simultaneous speakers at any moment, and Discord avoids sending audio during periods of silence.

06 · FAILURE AND SCALE

When the voice server goes down

In case of infrastructure failure, participants can simply reconnect to a new backend server. The step-by-step below shows what Discord does when a voice server dies.

1. When a voice server dies, it fails the periodic ping and is removed from the service discovery system.

2. The client notices the server failure from the cut voice WebSocket connection.

3. The Guilds server confirms the failure, queries service discovery, and assigns a new voice server to the guild.

4. All clients are notified of the new voice server and create a voice WebSocket connection to it to start media setup.

When a voice server dies, it fails the periodic ping and is removed from the service discovery system. The client notices the failure from the cut voice WebSocket connection. The Guilds server confirms the failure, queries service discovery, and assigns a new voice server to the guild; all clients are notified and create a voice WebSocket connection to the new server to start media setup.

On scale, Discord's post is from September 10, 2018. In it, the Gateway, Guilds, and Voice servers scale horizontally, and Discord runs more than 850 voice servers in 13 regions, serving more than 2.6 million simultaneous voice users, with outbound traffic above 220 Gbps.

Video is a harder problem than audio: compared to audio, it requires far more CPU and bandwidth. At the time, Discord saw Scalable Video Coding as a possible solution for a uniform video experience.