AWS S3 2017: one mistyped command

01 · CONTEXT

A store that the rest of AWS uses

On the morning of February 28, 2017, Amazon S3 in the US-EAST-1 region, in Northern Virginia, stopped serving requests for hours. AWS's official summary is the source for this case. The team was investigating why the S3 billing system was running slower than expected, and the incident began during that routine work.

Two subsystems matter here. The index holds the metadata and location of every object in the region, and is required to serve reads, listings, writes, and deletes. The placement subsystem decides where to store new objects, is used in writes, and only works properly with the index working.

1. Clients send reads, listings, writes, and deletes to the object API.

2. Reads, listings, and deletes query the index, which knows where each object is.

3. A write goes through placement, which chooses where to store the new object.

4. Placement needs the index to be working. If the index goes down, writes go down with it.

02 · THE COMMAND

A mistyped input in a known procedure

At 9:37 AM Pacific time, an authorized team member followed an established procedure and ran a command to remove a few servers from a subsystem used by billing. One of the command's inputs was mistyped, and a larger set of servers than intended was removed.

1. The command was supposed to remove a small number of servers from the subsystem used by billing.

2. With the wrong input, the command also removed servers that supported the index and placement.

3. Without a significant part of their capacity, each of the two subsystems needed a full restart, and the service could not serve requests in the meantime.

4. Because placement depends on the index, recovery had an order: first the index, then placement.

The system was built for exactly this: S3's subsystems were designed to withstand the removal or failure of significant capacity with little or no customer impact. What failed was not the idea of removing capacity, but the limit on how much could be removed at once.

03 · CASCADE

Whoever depended on the storage went down with it

Other AWS services in the region that depend on S3 for storage were also affected while the APIs were down: the S3 console, launching new EC2 instances, EBS volumes when the data came from a snapshot stored in S3, and AWS Lambda.

1. The object storage APIs became unavailable.

2. The console, instance launches, volumes restored from snapshots, and serverless functions depended on it and were affected.

3. The status dashboard's administration console also depended on the storage. That is why the status of each service could not be updated on the dashboard.

Until 11:37 AM, AWS could not update the individual status of services on the status dashboard, because of a dependency of the dashboard's administration console on S3. During that interval, customers were notified through AWS's Twitter account and a banner text. After the event, the dashboard's administration console started running in multiple regions.

04 · RECOVERY

A restart that had not been done in years

Removing and replacing capacity is a core S3 operating practice since launch. A full restart of the index and placement, however, had not happened in the larger regions for many years. With the service's enormous growth, restarting and running the safety checks that validate the integrity of the metadata took longer than AWS expected.

Even with the storage back, some services that depended on it had accumulated a work queue during the outage and needed extra time to recover fully.

05 · FIXES AND LESSONS

Safeguards in the tool and smaller cells

AWS itself pointed out that the tool allowed removing too much capacity too quickly. It was changed to remove capacity more slowly and to block removal when it would take any subsystem below the minimum required capacity level. The other operational tools were also audited for similar safeguards.

One of AWS's most important techniques for recovering services is to split them into small partitions, called cells, which makes it possible to evaluate and test recovery even for the largest service and reduces the blast radius. Even so, the index took longer than expected to come back. Its additional partitioning, which was already planned, was reprioritized to start immediately.

1. Instead of a single block, the service is sliced into independent cells, and a router sends each request to the responsible cell.

2. If one cell needs a full restart, the others keep serving. The damage is limited to a fraction of the whole.

3. Each cell is small enough to have its restart and validation rehearsed and timed frequently.