Pokémon GO 2016: 50x the expected traffic at launch

01 · CONTEXT

A launch nobody sized for

Pokémon GO was a mobile app that used more than a dozen Google Cloud services.

The launch was also the debut of a new Google team, Customer Reliability Engineering (CRE). Its first customer was Niantic, and its first mission was precisely this launch. CRE is a model in which Google engineers embed with the customer's teams, creating shared responsibility for the reliability of critical applications.

02 · ESTIMATE VS. REALITY

Target, worst case and what actually arrived

The team aimed at a reference traffic level, called 1X, and estimated a worst case of about 5X that target. Actual traffic reached 50X the initial target, ten times the worst case. A second source summarizes the same point as traffic 50 times beyond what was expected.

Cloud Datastore was the game's main database for recording the Pokémon world and became a direct indicator of the game's popularity.

Within 15 minutes of the launch in Australia and New Zealand, traffic went well past Niantic's expectation. The company asked CRE for reinforcement because of the US launch, planned for the next day.

03 · THE ARCHITECTURE

Players, load balancer, containers and database

The application logic runs on Google Container Engine (GKE), based on Kubernetes. Niantic chose it for orchestrating the container cluster at planetary scale, and the game was the largest Kubernetes deployment on Container Engine to date.

1. The app on the players' phones generates the traffic that reaches the game.

2. Before the switch, traffic went through the Network Load Balancer, which was later replaced.

3. The game logic runs on a container cluster on GKE, orchestrated by Kubernetes.

4. Cloud Datastore stores the game world and serves as a demand thermometer.

In plain terms: load balancer in front, an application layer that scales by adding nodes and a managed database behind it. The case shows what to do when this structure needs to grow far beyond what was predicted.

04 · SCALING MID-FLIGHT

Swapping the engine while the plane is flying

In preparation for the Japan launch, Niantic and CRE upgraded to a new version of GKE, which allowed adding more than a thousand nodes to the cluster. The version switch was done carefully so as not to interrupt existing players while millions of new players were joining.

In the same vein, the teams swapped the Network Load Balancer for the HTTP/S Load Balancer, a global system for HTTPS traffic, with more control, faster connections and higher throughput.

On the capacity side, CRE provisioned extra capacity on Niantic's behalf to stay ahead of growth, and Google provisioned many tens of thousands of cores for the game's cluster.

1. The GKE cluster was upgraded to hold more than a thousand additional nodes, without taking the game offline.

2. The network load balancer gave way to the HTTP/S Load Balancer, global and with higher throughput.

3. CRE provisioned extra capacity for Niantic, keeping the database ahead of growth.

4. With the adjustments, the Japan launch went off without incidents.

05 · RESULT AND LESSONS

What paid off and what was left over

The lessons from the US launch (generous capacity, a new version of Container Engine and the HTTP/S Load Balancer) paid off when the game launched in Japan without incidents. There, the number of new users was triple that of the US launch two weeks earlier.

Not everything went well at launch. Niantic and Google engineers handled each problem in sequence, building and deploying fixes. Given the scale of the cluster, many bugs were identified, fixed and incorporated into the open source Kubernetes project.