Amazon Aurora: "the log is the database"

01 · CONTEXT

When the network becomes the bottleneck

Amazon Aurora is a relational database service for OLTP workloads offered as part of AWS. The central trait of the architecture is to take several functions out of the database engine (redo log recording, durable storage, crash recovery, and backup with restore) and hand them to its own multi-tenant storage service that scales horizontally and is loosely coupled to the fleet of database instances.

In the cloud, I/O is spread across many nodes and disks of a shared fleet, so individual disks and nodes stop being hot. The bottleneck becomes the network between the database tier, which requests the I/Os, and the storage tier, which executes them. Besides the packets-per-second and bandwidth limits, there is traffic amplification, because a well-performing database issues writes in parallel to the fleet. The performance of the slowest node, disk, or network path can dominate response time.

02 · THE PROBLEM

Why replicating a traditional MySQL does not scale

Instances fail, customers shut them down and resize them according to load, and an instance's lifetime does not track that of storage. That is why, according to the paper, it helps to decouple the storage tier from the compute tier.

The problem is that a traditional MySQL generates many real I/Os for each application write. The data written includes the redo log, the binary log archived to S3, modified data pages, a second temporary write of the page (double-write) against torn pages, and metadata files. Replication amplifies this volume, imposes a heavy packets-per-second load, and creates synchronization points that stall pipelines and stretch latency.

1. The application writes to the active MySQL, which runs in an AZ with network-attached storage.

2. The write goes to the EBS volume, which passes it to a local mirror in the AZ; the acknowledgment only arrives when both finish.

3. The write is then sent to the standby, in another AZ, by synchronous block-level mirroring.

4. Finally, the standby writes to its own EBS volume and the associated mirror.

5. Separately, the binary log is archived to S3.

6. Some of these steps are sequential and synchronous, so latency adds up; the paper sees the model as a 4/4 write quorum, vulnerable to failures and to the performance of slow components (outliers).

To measure network traffic, the authors ran a test with the SysBench write workload on a 100 GB dataset, comparing MySQL mirrored across AZs with Aurora, for 30 minutes on an r3.8xlarge instance. Aurora sustained 35 times more transactions; on the database node, the number of I/Os per transaction was 7.7 times lower, even though Aurora amplifies writes six times. Each storage node sees unamplified writes, since it holds only one of the six copies, which yields 46 times fewer I/Os to process at that tier.

03 · DURABILITY

Six copies in three AZs

The common approach to tolerating the loss of a node is to replicate across three nodes, with a write quorum of 2/3 and a read quorum of 2/3. For consistency, the quorum obeys two rules: every read must be aware of the most recent write, which is formulated as Vr + Vw > V, and every write must be aware of the most recent write to avoid conflicting writes, formulated as Vw > V/2.

The authors consider 2/3 quorums inadequate. In a large fleet, at any moment some set of disks or nodes may be failed and under repair, with failures spread independently across AZs, and the failure of an entire AZ breaks the quorum of any replica that already has a concurrent failure in another AZ. In other words, individual failures per AZ are uncorrelated, but the failure of an AZ is a correlated failure of all its disks and nodes.

Aurora's choice is to tolerate the loss of an entire AZ plus one additional node without losing data, and the loss of an entire AZ without losing the ability to write. To do this, it replicates each item six times across three AZs, with two copies per AZ, using six votes, a write quorum of 4/6 and a read quorum of 3/6. This makes it possible to lose an AZ and one more node (three nodes) without losing read availability, and to lose any two nodes, which includes the failure of an AZ, while keeping write availability. Ensuring the read quorum makes it possible to rebuild the write quorum by adding copies.

With these values, both quorum rules hold: the sum of the read and write quorums exceeds the total votes, and the write quorum exceeds half the votes.

1. The volume is replicated in six copies, two in each of three AZs.

2. An entire AZ fails and takes its two copies with it; writes remain available.

3. With the AZ down and one more node lost, read availability remains.

4. With the read quorum, the system rebuilds the write quorum by adding copies.

Beyond a certain point it is hard to reduce the probability of independent failures (MTTF); that is why Aurora focuses on reducing repair time (MTTR), shrinking the window of vulnerability to a double fault. It partitions the volume into small fixed-size segments, today 10 GB, and each segment is replicated six times in a Protection Group (PG): six 10 GB segments across three AZs, two per AZ. A 10 GB segment can be repaired in 10 seconds on a 10 Gbps link. Losing the quorum would require seeing two of these failures in the same 10-second window plus the failure of an AZ that contains neither of them, and, at the observed rates, that is sufficiently unlikely.

04 · THE LOG IS THE DATABASE

Only redo crosses the network

In Aurora, the only writes that cross the network are redo log records. No page is written by the database tier, neither in background writes, nor at checkpoint, nor on cache eviction; the log applicator is moved to storage, which generates pages in the background or on demand. For the engine, the log is the database: the pages that storage continuously materializes are just a cache of log applications.

Unlike checkpointing, governed by the length of the entire redo chain, page materialization is governed by the length of each page's chain, and only pages with a long chain need to be rematerialized.

1. The application sends the transaction to the writer, whose engine is a fork of MySQL/InnoDB that diverges mainly in how it reads and writes data to disk.

2. The writer sends storage only redo records, grouped in batches per PG, to the six copies.

3. When four of the six copies acknowledge, the batch is durable and the database advances the volume's durable point.

4. Commit is asynchronous: the thread that receives it notes the commit LSN and keeps working, and a dedicated thread notifies the client when the durable point reaches that LSN.

5. The same log stream goes on to the read replicas, asynchronously with respect to the writer.

6. In the background, backup and restore services continuously copy the data changed in storage to S3.

One writer and up to 15 read replicas mount the same shared storage volume, so replicas add neither consumed storage cost nor disk write cost. The replica applies only records with an LSN less than or equal to the VDL and atomically applies the records of a single mini-transaction, in order to see a consistent view; in practice, each replica typically lags the writer by 20 ms or less.

The storage design principle is to minimize foreground write latency and move the bulk of the processing to the background. Each storage node performs eight activities, but only the first two are on the foreground path.

1. The node receives the record, puts it in an in-memory queue, writes it to disk, and acknowledges; only this stretch is on the foreground path.

2. It then sorts the records and identifies gaps in the log, since some batches may have been lost.

3. To fill the gaps, the node gossips with peers in the same group.

4. In the background, it coalesces the log records into new data pages.

5. Periodically, it sends the log and the new pages to S3.

6. Garbage collection of old versions and CRC validation are also periodic; collection can wait while the node is busy with writes, unless the disk is near capacity.

If the backlog builds up, storage throttles foreground activity. Since segments are spread with high entropy across nodes, throttling on one node shows up to the 4/6 write quorum as merely a slow node.

05 · CONSISTENCY AND RECOVERY

No 2PC and no recovery at startup

Instead of a protocol like 2PC, described as chatty and failure-intolerant, Aurora treats consensus asynchronously: it maintains consistency and durability points and advances them continuously as it receives acknowledgments from storage. A node that missed records gossips with the other members of its PG to fill the gaps.

Each log record has an LSN, a monotonically increasing value generated by the database. The database does not allocate an LSN above the sum of the current VDL and the LAL (LSN Allocation Limit), today 10 million. This limit keeps the database from getting too far ahead of storage and introduces backpressure that can throttle writes if storage or the network cannot keep up.

Example from the paper: even with complete data up to LSN 1007, the database may have declared as CPLs only 900, 1000, and 1100; the truncation is then at 1000. The volume is complete up to 1007, but only durable up to 1000.

Under normal conditions the database does not need a quorum read: when reading a page from disk, it fixes a read point (the VDL at the time of the request) and picks a storage node that is complete with respect to that point. The state maintained at runtime allows single-segment reads instead of quorum reads, except during recovery, when the state is lost and must be rebuilt.

In a traditional database, crash recovery reapplies redo from the last checkpoint, which can be expensive; shortening the checkpoint interval helps, but at the cost of interfering with foreground transactions. In Aurora this trade-off does not exist: redo is applied in storage continuously, asynchronously, and distributed, and nothing is required at database startup. The paper reports recovery generally in under 10 seconds, even with more than 100,000 writes per second at the time of the crash.

1. The writer crashes, but storage was already applying redo continuously, in the background and distributed across the fleet; nothing is required at database startup.

2. On startup, the database contacts, for each PG, a read quorum of segments, enough to discover any data that may have reached the write quorum.

3. With the read quorum of each PG, it recomputes the VDL and annuls the records above it; the upper bound of the truncation is inferred from the LSN allocation limit, and the ranges are versioned by epochs and durably recorded.

4. Undo for in-flight transactions is still needed, but can happen with the database already online.

06 · RESULTS AND LIMITS

What the paper shows and what it assumes

In the paper's benchmarks, with the SysBench write workload and a 100 GB database, Aurora was up to 67 times faster than MySQL; with a 1 TB database and a workload that does not fit in cache, still 34 times. In the SysBench OLTP benchmark, with connections rising from 50 to 500 and 5000, Aurora scales from 40,000 to 110,000 writes per second, while MySQL peaks near 500 connections and then drops sharply.

The real workloads section presents results reported by customers who migrated from MySQL. At an education technology company, replica lag peaked at 12 minutes, which affected application correctness and left the replica useful only as a standby; after the migration, the maximum lag across 4 replicas never exceeded 20 ms. At the same company, P95 latencies ranged from 40 ms to 80 ms, far worse than the P50 of about 1 ms, and after the migration they came close to the P50.

The design has an explicit premise: at any moment there is a single writer generating log updates, with LSNs allocated from a single ordered domain, and it is with this simplification that Aurora offers its isolation levels. Concurrency control is implemented entirely in the database engine, without impacting storage, and the writer offers the same isolation levels as community MySQL.

Among the operational lessons, the paper notes that many customers have low tolerance for database software updates, even if that means a planned stop of 30 seconds every 6 weeks or so. The paper describes the ZDP feature, which allows patching a customer while in-flight connections are unaffected.