AWS EBS 2011: re-mirroring storm

01 · CONTEXT

A replicated block cluster and a regional control plane

In April 2011, an incident in the AWS US East region left some Amazon EBS volumes, all in a single availability zone, unable to serve reads and writes. AWS calls these volumes stuck, and instances that tried to use them also got stuck. According to AWS, the event had several root causes that interacted with one another.

EBS replicates each volume's data across several nodes in a cluster. The nodes talk over two networks: a primary one, with high bandwidth and used in normal operation, and a secondary, replication one, with lower capacity and a backup role.

The secondary network was not designed to carry all of the primary's traffic, but to provide reliable connectivity between the cluster's nodes.

When a node loses its connection to the node it replicates to, it assumes the other failed and must find a new node with space for the replica, a process called re-mirroring. In a normally functioning cluster, finding that space takes milliseconds. While the data is being re-mirrored, access to it is blocked until a new primary replica is identified, and to the instance the volume looks stuck.

Above the clusters are control plane services that accept user requests and propagate them to the right cluster. There is one set of them per region, distributed across the zones, and they are also the authority that elects each volume's primary replica, because, for consistency, there can be only one primary replica per volume at any time.

1. Compute instances send their accesses to the node that holds the volume's primary replica, the writable one.

2. The volume's data is replicated on more than one node. Between the nodes there is the primary network, for normal use, and the replication network, with lower capacity, as a backup.

3. The control plane is the authority that elects which replica of each volume is the primary.

4. If a node loses its connection to its peer, it assumes the peer failed and looks for another node with space. Meanwhile access to the data is blocked and the volume looks stuck.

02 · THE TRIGGER

A network change and the re-mirroring storm

The trigger was a network configuration change, performed on April 21 during normal scaling activities in a single zone. It aimed to increase the capacity of the primary network, and one of the steps was to shift traffic from one redundant router to the other. The shift was executed incorrectly and the traffic ended up on the redundant network, which has lower capacity.

For part of the cluster, this meant being left with neither a primary nor a secondary network functioning. Unlike an ordinary network outage, the change disconnected both networks at the same time and left the affected nodes completely isolated from one another.

When the shift was undone and the network came back, the nodes quickly began looking for space to re-mirror. Because many volumes were affected at the same time, the cluster's free capacity ran out and many nodes got stuck in a loop, searching for space nonstop. This led to a re-mirroring storm, and about 13% of the zone's volumes were stuck at this point.

Two factors degraded the cluster further at the start. First, the nodes that couldn't find space did not back off aggressively enough and kept searching repeatedly. Second, there was a race condition in the nodes' code that, with very low probability, brought them down when closing many replication requests at the same time. Nodes failed because of the bug and left more volumes to re-mirror, which fed the storm.

03 · THE CONTROL PLANE

When one zone's failure becomes the region's failure

The degraded cluster immediately affected the control plane. With no free space, the cluster became unable to serve create-volume API requests. Because this API had a long timeout, the slow calls piled up and caused thread exhaustion in the control plane.

The control plane uses a regional thread pool. When queued requests filled it, it lost the ability to serve and began failing requests from other zones in the region as well.

1. The network change disconnected both networks at the same time, and many nodes in the zone's cluster became isolated from one another.

2. With the network back, the nodes looked for space to re-mirror, the free capacity ran out, and they got stuck in a search loop.

3. The cluster no longer served create volume. With a long timeout, the slow calls piled up and exhausted the control plane's threads.

4. Because the thread pool is regional, the control plane began failing requests from other zones as well.

5. The team disabled new create-volume requests in the affected zone, and the other EBS-related APIs recovered.

6. With re-mirroring unsuccessful, retries and new requests increased the negotiations with the control plane, and the load caused a brown out of it. The team then shut off communication between the cluster and the control plane.

04 · THE RECOVERY

Containment, new capacity, and a dedicated control plane

The team developed a way to keep servers in the degraded cluster from contacting others in vain, without affecting essential communication between nodes. After this change, the cluster stopped degrading and more volumes were no longer at risk of getting stuck. Before it, servers brought down by the race condition left an additional 5% of the zone's volumes stuck, and the total of stuck volumes in the zone, when the change was deployed, was 13%.

The priority became putting new capacity into the cluster so that the stuck volumes could find space for new replicas. Two challenges delayed this.

With enough capacity, the team worked to reopen the control plane's access to the zone. There was a large backlog of state changes to propagate in both directions, and the work was done gradually so as not to affect the restored volumes or the control plane.

Recovering the remaining volumes required a more manual process. The team had taken snapshots of them for backups to S3 early in the event, as a precaution, and later developed and tested code to restore them from those snapshots. Volumes without a snapshot, which had suffered machine failure, went through forensics. In the end, 0.07% of the zone's volumes could not be restored consistently.

05 · CUSTOMER IMPACT

RDS and the difference between one zone and several

RDS depends on EBS to store the database and logs, so some databases in the affected zone became inaccessible. Single-AZ instances are exposed to a zone's outages, and at the peak 45% of them had stuck I/O. Each RDS instance uses several volumes, and stuck I/O on any one of them can leave it inoperable until the volume is restored.

In multi-AZ mode, RDS replicates data synchronously between two replicas in different zones and was designed to detect the outage and fail over. Even so, 2.5% of the multi-AZ databases in the US East region did not fail over automatically. The main cause was the rapid succession of a network outage and stuck I/O on the primary, which triggered a bug never encountered before. The primary became isolated, automatic failover risked data loss, and manual intervention was needed.

For applications, AWS states that the impact on running instances was limited to the affected zone, and that many users who used several zones had no significant availability impact. Some applications, or critical components such as the database, live in a single zone, though, and others spread instances but keep critical single points of failure in one zone.

1. Generic example: traffic is distributed across applications in two zones, but both depend on a database that exists in only one zone.

2. The storage problem stayed limited to the affected zone, hitting the instances and the database that depended on its volumes.

3. The application in the other zone does not use volumes from the affected zone, but depends on the database that is there, a single point of failure in one zone.

4. A robust multi-zone deployment, with no critical components tied to one zone, would have let the application continue without impact.

06 · LESSONS

Overload control and isolation between zones

Against a repeat of the storm, AWS points to three safeguards: more spare capacity, with which the cluster would have absorbed the re-mirroring requests and avoided the storm; retry with more aggressive backoff; and the fix for the race condition, whose origin was identified.

So that one zone does not affect the control plane across several zones, AWS promised to improve the timeout logic to avoid thread exhaustion, make the control plane zone-aware so it can shed load intelligently when above capacity (load shedding), and move more functions to per-EBS-cluster services, running in the same zone as the cluster.

AWS also recalls the role of regions: they are completely isolated from one another, and many users use several regions for fault tolerance. But AWS does not replicate data across regions on users' behalf, and taking advantage of that isolation takes effort from those who build applications.