Stripe: online migrations at scale

01 · CONTEXT

Why migrating production data is hard

Stripe's engineering post describes how the company migrated hundreds of millions of subscription objects (Subscriptions) to a new data model, keeping the service up. Stripe says it makes all infrastructure updates online, without relying on planned maintenance windows.

The post breaks the difficulty into three fronts. On scale, if each object took one second to migrate, migrating sequentially would take more than three years for a hundred million objects. On availability, the subscriptions service could not be paused. On accuracy, changing thousands of lines at once would almost certainly let edge cases slip through, and every service had to keep relying on correct data.

02 · THE OLD MODEL

From one subscription per customer to a dedicated table

At first, each Customer object had at most one subscription, and subscriptions were stored together with the customers. Later, some users wanted customers with multiple subscriptions, and the single field became a list of active subscriptions.

1. With the initial model, the service reads and writes subscriptions inside the customer record.

2. With new features, any change to subscriptions updates the entire customer record, and subscription queries scan customer objects.

This model became a problem as features grew: changing a customer's subscriptions updated their entire record, and subscription queries scanned customer objects. The decision was to store active subscriptions separately, and the new model moves subscriptions into their own table.

03 · DUAL WRITE AND BACKFILL

Duplicate the writes and copy what already exists

The migration starts by creating the new table and writing to both places. In Stripe's case, every newly created subscription was written to both the Customers and Subscriptions tables. The post recommends considering the impact of this extra write on the production database and proposes mitigating it by gradually raising the percentage of duplicated objects while watching operational metrics.

1. In the old model, the service stores subscriptions inside the Customers table.

2. Dual writing begins: each newly created subscription is written to both tables.

3. To control the impact on the production database, the percentage of duplicated objects is raised gradually, with metrics under observation.

4. Old objects enter the new table lazily, when they are updated.

Objects that already existed were copied lazily: whenever one was updated, it automatically moved to the new table. Whatever was left was picked up by a final backfill.

The most expensive part of a backfill on a live database is simply finding out which objects need to migrate, because that would require many queries against the production database and a lot of time. Stripe took that search out of production: it made snapshots of the databases available to the Hadoop cluster and used MapReduce to process the data offline and in a distributed fashion.

1. Finding all the subscriptions to migrate through queries to the production database would be the most expensive part of the backfill.

2. The database snapshots are made available to the Hadoop cluster, which processes the data offline and in a distributed fashion.

3. A Scalding job produces the list of subscription IDs that need to be copied.

4. A fleet of processes copies the subscriptions on the list to the new table in parallel.

5. Once the migration is done, the job runs again to confirm that no existing subscription was left out of the new table.

04 · READ, WRITE, AND REMOVAL

Switch the reads, flip the writes, and retire the old model

With the tables in sync, the next step is to read from the new one. To be confident the data matches, Stripe used GitHub's Scientist: a Ruby library that runs experiments comparing two code paths and alerts you if they diverge in production. If the experimental path raises an error, the rest of the application is unaffected.

The experiment read from both tables and, if the results did not match, raised an error to alert engineers to the inconsistency. After verifying that everything matched, Stripe started reading from the new table.

1. Before the switch, all subscription reads use the Customers table.

2. The Scientist experiment reads from both tables and compares the results.

3. If the results do not match, an error alerts engineers to the inconsistency.

4. After verifying that everything matched, reads start coming from the new table.

5. For writes, the order is reversed: write to the new table first, then archive to the old one.

6. When no code depends on the old model, writing to the old table stops and the new one becomes the source of truth.

The post considers writes possibly the most challenging part: the subscription logic (updates, proration, renewals) is spread across thousands of lines in several services. The key was the incremental process, isolating code paths into the smallest possible unit to apply each change carefully, with both tables consistent at every step.

To make sure no code kept using the old array, the post proposes raising an error when the property is accessed. When nothing else depends on the old field, writing to the old table stops, and the new one becomes the source of truth. The old data is phased out lazily: the array is emptied each time a subscription is loaded, and a final job and migration find whatever is left.

05 · SUMMARY

The four phases side by side and what helped