Music streaming (Spotify)

01 · CONTEXT

On-demand music with client and peers

Spotify is an on-demand music streaming service. The technical paper that serves as the basis for this case describes a service that delivers music through a combination of client-server access and a peer-to-peer protocol. At the time, the library had more than 8 million tracks and the service had been live since October 2008.

The main reason for creating the peer-to-peer protocol was to scale the service, reducing the load on the servers and bandwidth use. There was an explicit condition: peer-to-peer could not worsen the latency to start a track or increase interruptions (stutter) during playback.

The smartphone clients of the time did not take part in the peer-to-peer protocol and received audio only from the servers.

02 · FROM TAP TO SOUND

The path of a track to the ear

When the user picks a track the client does not have yet, the client sends the server an initial request for about 15 seconds of music over the TCP connection that is already open, and at the same time looks for other clients that can serve the track. If the track is already in the cache local cache, it does not need to be downloaded again.

1. If the track is already in the local cache, the client does not need to download it again.

2. Without the track in cache, the client makes an initial request to the server for about 15 seconds of music, over the connection that was already open.

3. At the same time, the client looks for peers that have the track. The tracker keeps a map of tracks to peers and replies with a limited list of peers that are online.

4. With the peers found, the client downloads data from them. While the buffer is full enough and peers are available, the client downloads only from the peer network.

5. If the buffer runs low, the client requests data from the server, whose connection is considered more reliable than the peers'.

6. What the client downloads usually stays in the local cache. The client only offers to serve peers the tracks it has fully cached.

The decision of where to download from is local: each client looks at the amount of data in its own playback buffer. When downloading from the server, the client limits requests so as not to go beyond about 15 seconds ahead of the playback point, when peers are available for the track.

With these numbers, the paper concludes that large caches and a peer network together significantly reduce the load on the servers. The count includes data downloaded even if the track never ends up playing.

03 · LATENCY AND PREFETCH

Starting fast: prefetch, buffer, and cache

Two important measures in on-demand streaming are the time until playback starts and the frequency of stutters. Looking at the first: in the measured week (March 23 to 29, 2010), the median latency was 265 ms, the 75th percentile was 515 ms, and the 90th percentile was 1047 ms. Since more than half of the data came from the cache, the paper considers it likely that the median measures DRM time plus local processing.

The paper notes that even fully cached material has to wait for a response over the network before playing, unless the track has been synced for offline use. That is why the latency almost always includes at least one RTT to the servers.

For the next track, the client does prefetch. It starts looking for peers and downloading the next track when 30 seconds or less remain in the current track; at 10 seconds or less, it fetches the start of the next one from the server, if needed. The tuning has a cost: late prefetch may not be enough for the next track to play immediately, and prefetching too early wastes bandwidth if the user picks a different track.

For a few weeks, a bug accidentally disabled prefetch of the next track on all clients, which made it possible to measure the effect of prefetch.

04 · FINDING WHO HAS THE TRACK

Tracker, neighborhood search, and two sites

The tracker keeps a map of tracks to the peers that recently reported having the track. It stores only the 20 most recent peers for each track and replies with up to 10 peers that are online.

Clients register with the tracker when they play a track, without periodically reporting the contents of their cache. Besides the tracker, the client sends the search to its neighbors, who forward it to their own neighbors; this way, all peers up to distance two from the client see the request. This limited forwarding is the only routing in the peer network.

Spotify did not use a distributed hash table (DHT) to find peers. The paper points out that downloading new tracks is a frequent operation and lookup time becomes a big problem; other reasons include keeping the protocol simple and the cost low.

The service ran from two data centers, one in London and one in Stockholm. The client picks one uniformly at random to connect to, and the load is evenly distributed.

1. On connecting, the client randomly picks one of the two data centers, uniformly.

2. Each data center has an independent peer network.

3. The separation is not complete: when reconnecting to the other site, the client keeps its old peers, but cannot make new connections to peers attached to the old site.

05 · THE LIMITS OF THE DESIGN

Popularity, NAT, and what peers do not cover

Access to tracks is concentrated: in the measured week, 88% of accesses went to the 12% most popular tracks in the library. Even so, about 60% of the available content was accessed at least once during the week. The requests that reach the servers are less concentrated: 79% of accesses go to the top 21% most popular, in a chart that uses data from February 2010, outside the measurement period.

The peer network has practical limits. Clients do not do NAT traversal, and the fraction of successful connections was 35% in the measured week. To help, the connection request also goes through the server, which asks the other side to try connecting back, and clients use UPnP to ask the home router for a port.

06 · CONTENT INGESTION

From upload to catalog: a video podcast incident

So far the case has covered audio delivery; content ingestion is another path in the system. According to a public Spotify incident report, when a podcast creator publishes an episode, the audio and video go through a series of processing steps before becoming available to users, including transcoding (converting the media into the formats the apps need) and content analysis.

The report covers a publishing delay on June 24. The video transcoding infrastructure hit maximum capacity and a backlog delayed the publication of video episodes for several hours. The text mentions two filas: the medium-priority one, used for new episodes, and the low-priority one, used for updates to old episodes.

1. The creator publishes a new episode, which enters the medium-priority queue, used for new episodes.

2. In typical periods, transcoding was able to scale to handle both low-priority transcoding and the publication of new content.

3. A routine batch job, which reprocesses existing episodes, consumed additional capacity alongside the regular processing of new episodes, and transcoding ran without enough headroom for large spikes.

4. The infrastructure hit maximum capacity and the queues built up: episodes that normally go out in minutes were delayed for hours.

5. Some creators re-uploaded episodes that did not appear, which increased the load. The report acknowledges that the system should have confirmed that the upload was received and queued.

6. The report counts about four hours between the first alerts and the formal incident response.

Among the steps already taken, the report lists increasing transcoding capacity by about 67%, fixing the scheduling bug, and improving monitoring to alert earlier when capacity nears the limit.

The ongoing reliability program includes planning capacity for spikes and incident recovery, not just for steady traffic; prioritizing creators' real-time content ahead of background operations; and extending rate limiting and backpressure along the pipeline.