Hotstar: 25 million concurrent viewers

01 · CONTEXT

A live audience record

Hotstar is a Disney streaming (OTT) service. According to an account of an AWS re:Invent talk, on July 10, 2019 it reached 25.3 million concurrent viewers, surpassing the service's own previous record of 18.6 million. The peak came during the India vs. New Zealand match of the 2019 Cricket World Cup.

As early as April 2018, the company's engineering blog described the preparation for the IPL, Hotstar's big event of the year. Both accounts, the 2018 post and the 2019 talk summary, argue for scaling ahead of the event instead of reacting to it.

02 · THE PROBLEM

Notification, spike and overload

The post describes a pattern the old platform faced repeatedly: when something interesting happened in the match, a notification was sent, and the resulting traffic spike overloaded the back-end systems and sometimes disrupted video rendering. The team says it learned to deal with these tsunamis the hard way.

1. Something interesting happens in the match and the notification service alerts the apps.

2. The resulting traffic spike hits the back end.

3. The spike overloads the back-end systems.

4. Sometimes the overload even disrupts video rendering.

Automatic retries make things worse. That is why the post calls for clients that infer when something is wrong and add jitter to the requests they make to the servers.

03 · PRE-SCALING

Why not rely on autoscaling

The post's first lesson is to leave capacity headroom instead of counting on autoscaling. Given how long autoscaling takes to act, it is useless for handling spikes; the strategy is to estimate peak concurrency and raise capacity before the event.

Autoscaling does not disappear from the design: the post uses it to keep the right number of servers in the pool, but the fleet is already scaled for the peak. When the team crosses the headroom thresholds, defined after the scale tests, it brings up more capacity in advance.

The talk summary repeats the thesis: traditional autoscaling does not work for Hotstar. The slides list as reasons insufficient-capacity errors, a single instance type per autoscaling group and the group's step size. The post adds that the cloud provider also has physical limits on how much can be scaled automatically, and that you need to work with them and plan capacity ahead of time.

Escalar ahead of time includes, according to the practices listed in the slides: pre-warming the infrastructure before the match, scaling by request count or by platform concurrency, doing proactive, automated scaling based on concurrency and using a spot EC2 fleet with diversified instance types. Asked what the alternative to traditional autoscaling would be, the recorded answer was a plain Python script.

1. Before the match, the team estimates peak concurrency and brings the fleet up to that capacity.

2. During the match, request count and platform concurrency become metrics.

3. If the headroom defined in the tests is exceeded, the scaling script brings up more capacity in advance.

4. Caching in the service layer and TTLs on the client take traffic off the origin and give the fleet headroom.

The slides under the title Race against time list a growth rate of more than 1M per minute, an application boot time of around one minute, 90 seconds of reaction time, push notifications and fully pre-baked AMIs.

04 · PROTECT AND DEGRADE

Protected origin, smart client

The post divides the platform into three pillars: the subscription engine, the metadata engine and the streaming infrastructure. Each has its own scaling needs and was tuned separately, with pessimistic traffic models and ladders that controlled the fleets according to estimated concurrency.

The funnel has an external limit: Hotstar depends on the processing rates of its payment partners, and so, sometimes during a spike, it may add jitter to the funnel so clients advance at an acceptable rate.

On the client, the apps were built to be resilient and degrade gracefully. Panic switches in the infrastructure tell the apps to ease off requests for a moment, with exponential or, at times, custom backoff, which gives the back end time to recover. The primary function is preserved: specific functions can degrade without affecting it.

1. The diagram shows the app and the video, subscription and payment, and recommendation services.

2. When load gets out of control, the panic switches signal the app to ease off requests.

3. The app slows down with backoff, exponential or custom, and the back end gains time to recover.

4. Recommendations and personalization are turned off on purpose to protect what is critical.

5. Video streaming and payment, the critical components, stay up.

05 · REHEARSE FAILURE

Load testing, game days and chaos

To test scale, the post cites the use of Flood.io on top of Gatling. The team also ran game days, simulations of a real IPL event before it happened. They served to tune configurations and train operations, and revealed behaviors at scale tied to SSL handshakes, keep-alives and mismatches between AZs or regions.

Beyond simulated load, the team ran chaos tests (chaos engineering), which simulated various unavailability scenarios to see how the system reacted and recovered. Among the chaos ingredients listed in the talk are tsunami traffic, delayed scaling, bandwidth constraints, increased latency and network failure (CDN).

The talk also presents Hulk, a load generation tool used to prepare the platform for a 50M peak. In chaos tests, what you look for is each system's breaking point, as well as bottlenecks and choke points.

About telemetria, the post says it took work to get a unified view of the metrics that matter for making decisions.

06 · LESSONS AND LIMITS

What to take into the interview

The talk closes with three lessons: prepare for failures, understand the user journey and accept degrading gracefully.

The post adds the principle of ensuring the primary function always works and protecting the touchpoints with the server. Not every failure is fatal, and a cache with an appropriate TTL buys a lot of headroom.