Dropbox Magic Pocket: leaving S3 for its own exabytes

01 · CONTEXT

Why build your own storage

Dropbox stores two kinds of data: the content of files and the metadata about files and users. According to the announcement post, the company has always had a hybrid architecture, with metadata and web servers in its own data centers and file content on Amazon. It was one of the first to adopt Amazon S3, which let it scale fast and reliably.

Scale explains the conversation: about 40 PB of user data when the post's author joined, in 2012, and more than 12 times growth over the following four years, passing 500 PB at the time of the announcement. The numbers are from the time of the announcement.

Dropbox gave two reasons for bringing storage in-house. The first is performance, one of the product's differentiators: controlling the stack end to end lets you optimize it for the use case. The second is the unit economics: with scale and a particular use case, it is possible to customize hardware and software. The team also concluded it would have to build everything from scratch, because nothing in open source was proven at that scale.

02 · THE MIGRATION

From dark launch to ninety percent

In August 2014 came the so-called dark launch: Dropbox mirrored data between two regional locations and considered the system ready to store user data. In addition, it kept extra backups for six more months.

On February 27, 2015, for the first time, it began storing and serving user files exclusively in-house. With the infrastructure validated, it set the aggressive goal of scaling to more than 500 PB in six months.

The stated goal was to serve 90% of the data from its own infrastructure by October 30, 2015. Dropbox says it got there on October 7, nearly a month early, without major service interruptions or data loss.

1. Before: metadata and web servers in its own data centers, and file content on Amazon, in a hybrid architecture.

2. Magic Pocket comes in as the in-house system that stores file content.

3. The goal was to serve 90% of the data from its own infrastructure; Dropbox says it hit the mark ahead of schedule.

Dropbox did not abandon Amazon: it said it would keep investing in its own infrastructure and partnering with Amazon where it made sense, especially globally, and cited the expanded relationship with AWS to store data in Germany for European enterprise customers who requested it.

03 · REQUIREMENTS AND DATA MODEL

Immutable blocks and access that cools quickly

Magic Pocket is an immutable block store: it stores encrypted chunks of files of up to 4 MB and, once written, a block never changes. A file's changes live in a separate system, the FileJournal, which keeps the immutable blocks simple and moves the mutability logic to layers above.

The workload has strong temporal locality: a good part of the data is accessed very frequently in the first hour after upload and less and less afterward. Even so, every block requires low-latency reads. The system is based on hard disks, which have the advantage of being durable, cheap and dense, and reserves SSDs for databases and caches. There is a lot of initial replication and caching for recent uploads and more efficient encoding for the rest.

Durability is a non-negotiable requirement: the text says it has to be effectively infinite in theory. The data uses erasure coding and lives in several geographic regions. The design goals in the announcement were annual durability above 99.9999999999% and availability above 99.99%.

The scale required growing from prototypes of tens of petabytes to several exabytes in about six months. There were surprises: in one case traffic suddenly grew and began to saturate the routers between network clusters, which led to changing data placement and routing and, in the end, the network architecture between clusters.

The simplicity principle shows up in the choices. The system avoids quorum consensus and distributed coordination as much as possible and uses centralized points when that is fault tolerant and scalable. For the block index, instead of a distributed hash table, the team chose a large fragmentado MySQL cluster, a decision it describes as great for simplifying development and reducing unknowns.

Since the 4 MB block is too granular to move when swapping a disk or applying erasure coding, it is aggregated into larger units. The table summarizes the hierarchy.

Each block has a key, in most cases the SHA-256 hash of the block. Recent blocks are replicated directly across several machines; later the buckets are aggregated and erasure coded for efficiency.

04 · ARCHITECTURE AND PROTOCOLS

Zones, cells, and the path of a write

Magic Pocket is a multi-zona architecture, with clusters in the western, central and eastern United States. Each block lives in at least two zones, independently, and is replicated within them. The redundancy protects against natural disasters and large-scale outages and creates clear administrative domains, so a configuration error or a congestion collapse does not propagate across zones.

Within a zone, the Frontends determine where a block should live and issue the read or write commands. The Block Index maps each block to the bucket where it lives and is a large sharded MySQL cluster. The original plan was a dedicated key-value store, but MySQL proved more than capable, and Dropbox already operated thousands of database nodes.

A daemon asynchronously replicates block writes from one zone to the other, and each block reaches the remote zone within one second of the local upload.

Cells are self-contained logical storage clusters of about 50 PB raw each; to gain capacity, you normally add a new cell. Inside a cell, the OSDs are simple nodes as far as the rest of the system is concerned: they store blocks without understanding the cell topology or taking part in distributed protocols. The Replication Table maps each bucket to the volume and the OSDs that hold it. An open volume accepts writes of new data and nothing else; a closed volume is immutable and can be moved safely.

The Master is the cell coordinator and monitors OSD health. There is a single Master per cell, which gives one central point of coordination for placement decisions without the complexity of a distributed protocol. The cost is a ceiling: a cell holds about a hundred petabytes before the Master's CPU and memory become a bottleneck, and having several cells helps with deployment and with isolation against cascading failures. The cell keeps serving reads if the Master goes down. The Master is soft state, because the Replication Table holds the authoritative state of the volumes.

Volume Managers handle the Master's requests to move volumes and run on the same hardware as the OSDs, spreading their network demand onto idle hardware.

1. On a Put, the Frontend checks the Block Index to see whether the block already exists and picks a target volume.

2. Next, the Frontend queries the Replication Table to learn which OSDs hold the chosen volume.

3. The Frontend sends write commands to the OSDs, which fsync the blocks before replying.

4. Once the write is confirmed, the Frontend adds the entry to the Block Index and can reply to the client with success.

5. On failures, if an OSD fails the Frontend tries another volume, possibly in another cell; if the Block Index fails, it forwards the request to the other zone.

6. In the background, the replicator copies the new blocks to the other zone.

Get is the reverse path: the Frontend looks up the cell and bucket in the Block Index, the volume and OSDs in the Replication Table, and reads from one of the OSDs, retrying if any is unavailable. On volumes using erasure coding, the encoding lets each whole block be read from a single OSD, so most reads touch just one disk.

05 · REPAIR AND VERIFICATION

When a disk fails and how to know everything is fine

The Master triggers repair when an OSD stays offline for 15 minutes. Each OSD stores several thousand volumes, so losing one OSD lets its volumes be rebuilt from hundreds of other OSDs at the same time, which spreads the reconstruction traffic across hundreds of network cards and thousands of disks to minimize recovery time.

1. The Master watches the OSDs and triggers repair when one of them stays offline for 15 minutes.

2. The Master's first action is to close all the volumes that were on the failed OSD.

3. The Master builds a reconstruction plan that picks sources and destinations so as to spread the load across as many OSDs as possible.

4. The Volume Managers copy the data from the sources to the destinations, applying erasure coding when needed, and hand control back to the Master.

5. On commit, the Master increments the generation number on the volumes at the new OSDs and then updates the Replication Table.

6. If the Master fails before the commit, the volume stays at the old location and the new Master repairs it again.

The open and closed volume model is the centerpiece: it keeps live traffic from interfering with background operations and allows simpler consistency protocols than would be possible without this split.

On durability, a Markov model with disk failure rates and the worst-case expected repair times leads Dropbox to 27 nines. The company itself cautions that replication is necessary but by no means sufficient, and cites other failure modes: natural disasters, software bugs, operator error, and bad configuration changes.

Dropbox invests heavily in verification: more than 50% of the disk and database load is internal verification traffic. With more than half a million disks in production, they fail in a variety of ways. When it detects a bad disk, the system re-replicates the data, usually in under an hour, but an undetected failure would widen the vulnerability window from hours to potentially months. The disk scrubber, for example, continuously reads every bit on the disk and validates it against checksums, with a full pass roughly every one to two weeks.

There is also a protection called trash. On receiving the order to delete an extent, the node only moves it to a temporary area (trash). The Trash Inspector walks the blocks in those areas and checks them against the Block Index, and after inspection the extent still sits on disk for a day before being removed.

06 · COLD TIER

Paying less for cross-region replication

Access to files cools quickly: more than 40% of reads are of data uploaded in the last day, more than 70% in the last month, and more than 90% in the last year. Dropbox calls frequently accessed data hot and rarely accessed data cold, and the difference opens room to optimize cost by data class.

On file upload, n-way replication across many nodes is used and, afterward, in the background, older data is encoded with erasure coding. Replicating across data centers thousands of miles apart is, in practice, equivalent to storing double the data to better survive disasters at large scale.

The requirements of the cold tier were clear: no compromise on durability or read availability, while write availability does not matter. Since user writes go to the hot tier, the cold tier can pause writes at any time without affecting users, and data is migrated asynchronously when it cools.

The first attempt was a single erasure code spanning all regions, with one software instance. The biggest problem identified was not being able to truly mitigate durability risk: with a single logical region running a single software version, a bug could erase everything, no matter how many copies existed. The project was shut down after more than nine months of active work.

Another idea, from Facebook's warm blob system, was to pair each block with a partner in another geographic region and store the XOR of the two in a third. Dropbox gave up soon after exploring it again: maintaining the global pairing structure brought its own challenges, and the unpredictable deletion patterns required a process to reclaim space when one of the blocks was deleted.

The new solution kept the internal architecture of each region and split the block into fragments distributed across regions. In the three-region example, the first fragment goes to region 1, the second to region 2, and their XOR, as the parity fragment, to region 3.

1. As data cools, asynchronous migration moves it in the background from the hot tier to the cold tier.

2. In the three-region example, the block becomes two data fragments and one parity fragment, the XOR of the two, each in a different region.

3. On Get, the reader asks all three regions, waits for the two fastest responses, and cancels the remaining one.

4. If a region is down, migration simply pauses until all three are back.

5. To delete, each region removes its fragments independently, with different deleter versions and at different times.

On latency, at the 25th, 50th, 75th, and 95th percentiles the difference between the cold and hot tiers stayed constant at about one network round trip. At the 99th percentile, however, the cold tier had lower tail latency. Dropbox's explanation is that the cold tier got the two fastest responses out of three, while the hot tier had more naive retry logic, which fetched from the nearest region and only asked the other if the first attempt failed or timed out. Afterward, Dropbox tuned the hot tier to optimistically fire a retry to the second region if the first attempt did not arrive within a time budget.

There is an elegant property: the cold tier always exercises the worst case, because every read reconstructs from multiple fragments. A region outage does not cause large traffic shifts or an increase in disk I/O in the surviving regions.

On durability, each region remains independent and runs different software versions, which helps catch problems before they affect all copies. Before purging from the hot tier, Dropbox runs several rounds of validation.

The cost math: the hot tier is a 1+1 scheme and the cold tier, in the three-region example, is 2+1. The total replication factor is 2 times the region's internal factor on the hot tier and 1.5 times on the cold tier, which cuts disk usage by 25%. With four regions tolerating the loss of any one, the model would be 3+1, with 33% savings.

07 · LESSONS

What to take into an interview

Dropbox itself sums up the central principle as keep it simple, and most of Magic Pocket was built by a team of fewer than half a dozen people. The company also reminds us that the technical decision should be the right solution to the right problem, not picked because it is new and cool.

Three patterns repeat throughout the case: isolate failure domains (independent zones, cells, regions), separate immutable data from mutation logic, and move data to a cheaper scheme as it cools. This synthesis is study reading, not a claim by Dropbox.