Prime Video: from serverless microservices to a monolith (90% lower cost)

01 · CONTEXT

A tool that watches every stream

Prime Video offers thousands of live streams and built a tool to monitor every stream watched by customers. It automatically identifies perceptual quality problems, such as block corruption or audio/video sync loss, and triggers a remediation process.

The quality analysis team already had an audio and video inspection tool, but it was never designed for high scale: the goal was to monitor thousands of simultaneous streams. As more streams were added, the team saw that running the infrastructure at high scale was very expensive and that there were bottlenecks preventing it from reaching that number. The account is by Marcin Kolny, published on March 22, 2023.

The service has three pieces: a media converter, the defect detectors and the orchestration that controls the flow.

02 · THE SERVERLESS VERSION

Fast to build, expensive to scale

The first version was a distributed system with serverless components, orchestrated by AWS Step Functions and with detectors on AWS Lambda. According to the team, it was a good choice for building the service quickly. In practice, the way some components were used led to a hard scaling limit at about 5% of the expected load.

1. The media converter receives the stream and turns it into decrypted frames or audio buffers that go on to the detectors.

2. The converter temporarily sends the images to an object store.

3. Each detector is a separate microservice: it downloads the images and processes them in parallel with Lambda. The large number of calls to the storage was expensive.

4. The orchestration made several state transitions per second of stream, so it hit account limits, and Step Functions charges per transition.

5. When a detector finds a defect, it sends a real-time notification.

The two pain points were the orchestration and the data transfer between components.

When reading a box-and-arrow diagram, it is worth asking what each arrow costs.

03 · THE MONOLITH

Everything in a single process

The team first considered fixing the problems separately, ran experiments and decided to restructure the infrastructure. It realized the distributed approach did not bring many benefits in this case and merged all the components into a single process. Data transfer became in-memory, which eliminated the object store as an intermediary and simplified the orchestration.

1. Converter, detectors and orchestration now run inside the same process.

2. Frames go from the converter to the detectors through the process's memory, with no intermediate object storage.

3. The orchestration now controls the components inside a single instance, with simpler logic.

4. The components are the same as in the previous design, so the team reused a lot of code and migrated quickly.

5. The detectors keep sending notifications when they find defects.

04 · THE PRICE OF THE CHOICE

Scaling vertically and cloning the service

The monolith brought a limitation: since all the detectors run on the same instance, they can only scale vertically. The team adds detectors frequently and has already exceeded the capacity of a single instance.

The fix was to clone the service several times, each copy with a subset of detectors, and add a lightweight orchestration layer to distribute client requests.

1. Client requests arrive at a lightweight orchestration layer, which distributes them.

2. Each copy of the service is configured with a different subset of detectors.

3. The detectors within a single instance can only scale vertically; the fix was to clone the service.

4. The detectors in each copy send notifications when they find defects.

Another decision the post describes as non-obvious: the team replicated the expensive media conversion process and placed it close to the detectors. Converting once and caching the result might seem cheaper, but the team did not find it cost-effective.

05 · RESULTS AND LESSONS

What changed and what to take into the interview

According to the post, moving the service to a monolith cut infrastructure cost by more than 90% and increased scaling capacity. Today the service handles thousands of streams and still has room to grow. Running on Amazon EC2 and Amazon ECS also made it possible to use EC2 compute savings plans.

The changes make it possible to monitor all the streams viewers watch, not just the most popular ones.

In an interview, before splitting a system into services, ask where the arrows between components will be and what each one costs in price, in limits, and in data moved. If the scaling and cost bottleneck is precisely the communication between the pieces, merging them may be the answer; if the pieces have very different loads, splitting still makes sense.