Papers
Online migrations at scale
Jacqueline Xu · Stripe Engineering Blog, 2017
The problem: changing the data model without taking the system offline
The article starts from a common situation: as the product grows, the data model stops serving it and the team needs to redesign it to get cleaner abstractions and more complex functionality. In production, this can mean migrating millions of live objects and refactoring thousands of lines of code. The author describes how Stripe migrated its hundreds of millions of Subscriptions objects. The model had evolved in stages. At first, each Customer had at most one subscription, stored alongside the customer. Later, the single field became an array of active subscriptions inside the Customer. With new features, this became a problem: any change to a subscription required updating the entire Customer record, and queries about subscriptions scanned customer objects. So active subscriptions were moved to their own table. The article points to three difficulties. Scale: if each object took one second to migrate, copying one hundred million objects sequentially would take more than three years. Availability: Stripe does its infrastructure updates online, with no maintenance windows, so the subscriptions service could not be paused. Accuracy: the table is used in many places in the code, and changing thousands of lines at once would almost certainly let edge cases slip through; every service had to keep trusting correct data.
Step 1: dual writes and backfill of old data
The article describes a common four-step pattern for large online migrations, and the first is dual writing: every new write updates both the old and the new store, keeping the two in sync. At Stripe, newly created subscriptions began to be written to both the Customers table and the new Subscriptions table. Because the extra write can weigh on the production database, the percentage of duplicated objects is increased gradually, with operational metrics watched closely. At this point, new objects exist in both tables and old ones only in the old one. Copying the existing ones started lazily: whenever an object was updated, it was copied to the new table. Finally, a backfill copied the remaining subscriptions. The most expensive part was simply finding the objects that still needed to migrate, since that would require many queries to the production database. Stripe delegated that search to an offline process: database snapshots are available in the Hadoop cluster, where MapReduce, managed with the Scalding library, generated the list of subscription IDs to copy. A multi-threaded fleet did the copying in parallel and, when it finished, the same job was run again to confirm that no existing subscription was missing from the new table.
Step 2: change the read paths with experiments
With the two tables in sync, the second step is to start reading from the new table, but first you need to be sure the data is consistent. Stripe used Scientist, a Ruby library from GitHub that lets you run experiments comparing the result of two code paths and alerting when they diverge in production. The experiment read from both tables, Subscriptions and Customers, and compared the results; if they did not match, an error alerted the engineers to the inconsistency. Scientist generates alerts and metrics in real time, and if the experimental path produces an error, the rest of the application is not affected. Only after verifying that everything matched did reads start coming from the new table. The logic of this step is to make the migration observable: instead of assuming the copy is correct, the team compares real behavior in production and gets a warning as soon as a piece of data diverges. This gives confidence before the next step, which involves changing the writes. The article sums up this gain as transparency and observability: Scientist's experiments warned as soon as a single piece of data became inconsistent in production, and at each stage confidence in the migration grew.
Steps 3 and 4: switch the writes and remove the old data
The third step reverses the order of the writes. Until then, data was written to the old store and copied to the new one; now it is written first to the new store and then archived to the old one, keeping the two consistent so each incremental change can be observed. The article calls the refactoring of the paths that change subscriptions perhaps the hardest part: the logic for updates, prorations and renewals is spread across thousands of lines in several services. The key was the incremental process, isolating each code path in the smallest possible unit, without simply swapping new records for old ones, and using more Scientist experiments to detect inconsistencies. To ensure no code kept using the old array, access to that Customer property was changed to raise a hard error. The fourth step is cleanup: once no code depends on the old field anymore, writing to the old store stops and the new table becomes the source of truth. The subscriptions array of the Customer objects was removed gradually, emptying it each time a subscription was loaded and running, at the end, a final Scalding job and a migration to find what was left. According to the article, the changes never exceeded a few hundred lines of code at a time.
See the ideas at work
Practice the designs that use this paper in the System Design school.