Shopify: pods and Black Friday flash sales

01 · CONTEXT

When buying a bigger database stops being an option

In 2015 Shopify reached the point where it was no longer possible to buy a bigger database server. The way out was to split the database into shards, which allowed horizontal scaling and continued growth.

The price was resilience: what was gained in performance and scale was lost in fault tolerance. The reason was in the code, which in several places ran an action on every shard. If any shard went down, the entire action became unavailable across the whole platform, and the team predicted this would become a serious problem as the number of shards grew.

1. Shoppers from different stores reach the same application servers.

2. An action that walks all the shards queries each of them before responding.

3. One shard goes down: the entire action becomes unavailable for the whole platform, even with the other shards healthy.

02 · PODS

Isolate the whole shard, not just the database

Just splitting the database was not enough: each slice needed to be isolated so that a failure would not turn into a general outage. The answer was pods, a name Shopify asks not to be confused with Kubernetes pods. A pod is a set of stores that live in fully isolated datastores.

Outside the datastores, the pod uses shared resources, such as job workers, application servers, and load balancers. The rule is that each of them talks to only one pod at a time, and no action reaches another pod. In the later talk, Shopify explains that the datastores are isolated while the stateless workers are shared, which allows redistributing load when a store runs a flash sale.

Since there is no communication between pods, creating a new pod does not interfere with the existing ones, and this gives horizontal scale. For availability, each unit of work, whether a web request or a job, goes to a single pod, so serving a request requires only one pod to be up.

1. A store's request reaches the load balancer, where the Sorting Hat runs.

2. The Sorting Hat applies a list of rules, finds out which pod the store belongs to, and tags the request with a header.

3. The application server reads the header and connects only to the datastore of the indicated pod.

4. If another pod's datastore goes down, only the stores in that pod are affected; the stores in the first pod keep being served.

In the talk, the Sorting Hat appears as a Lua module that looks up the store's domain in the routing table. It runs in an OpenResty layer, before Rails, which also blocks bots. In the 2018 article's version, it is described as part of the load balancers.

03 · RETRIEVAL

The pod as the atom of disaster recovery

Shopify already had a disaster recovery data center before splitting the system into pods, and carried the idea over to each pod. Each pod got a pair of data centers: one active and one for recovery. The later talk describes the same design across regions, with the pod active in one region at a time and replication from the active one to the other.

With recovery done pod by pod, the promise that one pod's failure does not bring down the platform holds. To move a pod, the team built the Pod Mover, which takes it to the recovery data center in about a minute, without losing requests or jobs. Evacuating an entire data center means evacuating, one by one, the pods active in it.

1. The pod serves from the active data center, and its changes reach the recovery data center through replication.

2. When a switch is needed, the Pod Mover takes the pod to the recovery data center.

3. Once the switch is complete, requests are served by the other data center.

4. The old data center becomes the recovery data center for that pod.

04 · BLACK FRIDAY

Scale, rebalancing, and giant stores

Black Friday and Cyber Monday (BFCM) are Shopify's biggest event of the year. In the year before the talk, the platform totaled 6.3 billion dollars in sales, with peaks of 32 million requests per minute.

To grow without thinking about pods in the code, the rule is that every unit of work operates on a single store. That makes it easy to add pods, and merchants grow at different rates. That is why Shopify needs to rebalance, moving stores to spread load and disk usage. New merchants enter an available pod in round-robin fashion, and one of them may turn out to be a big flash-sale seller.

1. Ghostferry copies to the destination pod all the rows whose store identifier matches the store being moved.

2. New writes keep arriving; MySQL's binlog replicates them until both ends are nearly in sync.

3. The store is locked for writes, and the writes are queued while the jobs in Redis are copied.

4. The routing table points the store to the new pod, the lock is released, and the destination starts serving.

5. The old data in the source pod is deleted later, asynchronously, to reclaim space.

In short, the move copies the store's rows, tracks what changes through the database log, locks writes for a moment, copies the jobs in Redis, updates the route, and only then deletes the old data, asynchronously. The downtime is small: for many stores, under 10 seconds, and for the largest, under 20.

There are also merchants so large that they occupy an entire pod by themselves. For the ordinary large ones, Shopify says it has other mechanisms so that a flash sale does not monopolize the pod's capacity and leave the smaller neighbors unserved.

05 · OVERLOAD

Overload control: bots, circuit breakers, and testing in production

The storefront and the checkout have different profiles. The storefront is mostly reads, while the checkout writes heavily and talks to external systems. That is why pod isolation does not solve everything: each part needs its own protection against spikes.

A first filter is blocking bots, which overload the systems more than real shoppers. A second is the circuit breaker: when an external resource fails repeatedly, the call starts failing immediately, and that makes it possible to offer an alternative. Shopify notes that a degraded service is better than a service that is down. If the sessions Redis goes down, for example, it can show the store as a visitor instead of returning an HTTP 500 error.

The third element is testing before the peak. Shopify runs load tests in production against benchmark stores, at least one per pod, and runs these tests at least weekly. The goal is to find the bottlenecks before a large merchant's flash sale reveals them.

06 · TRADE-OFFS

What you gain and what you pay

The cost of the rule that nothing crosses pods lies in the functions that need to look at multiple stores. They become separate applications or use the data warehouse for analysis. This isolation also fits with a monolith that is modular: the talk describes a single Rails application, replicated across pods, and not microservices.

The design serves as a model of cell-based architecture: a complete, independent, replicable unit, with a router in front. This reading is the course's, not the source's, which uses the name pod.