Skip to main content

Concepts

Sloppy quorum and hinted handoff

Intuition

Imagine you hand a package to three trusted neighbors, but one of them is away on a trip. Instead of refusing the delivery and leaving the customer waiting, you leave the package with another neighbor, along with a note saying 'this belongs to the neighbor who is away'. When the traveler returns, whoever held the package hands it to them, and the delivery never had to stop; the price is that, in the meantime, the package is not with the person you expected.

How it works

In a system like Dynamo, each key has a preference list, defined by the key's position on the consistent hashing ring, whose first N nodes hold the replicas (the list has more than N nodes to cover failures). A strict quorum requires W of the first N nodes in the list to acknowledge the write and R of them to respond to the read, and with R+W>N the read and write sets overlap. In a sloppy quorum, Dynamo uses the first N healthy nodes in the list, which may not be the first N, instead of requiring the N preferred nodes to be reachable: if one of them is down, the write goes to another node further along the ring, which then counts toward the quorum. This stand-in node stores the data with a hint recording which node was the original intended recipient, and this is what is called hinted handoff. When the original node recovers, the stand-in hands the data over to it, and this is how the temporary failure of a replica or of the network stops blocking the write. The cost is that the quorum is no longer formed only by the first N nodes in the list: with R+W>N, the overlap between read and write is guaranteed only if both use the same N nodes, and the stand-in may not be part of the next read. That is why the system relies on reconciliation mechanisms, such as vector clocks to detect conflicting versions, which are returned to the application to reconcile, and read repair to realign replicas, and it chooses to be always writable rather than always consistent.

In practice

You would choose this design when refusing a write costs more than reading slightly stale data, as in the shopping cart, the example the Dynamo paper uses, accepting that the application handles divergent versions. Cassandra, whose paper adopts Dynamo's terminology (preference list), replicates each key on N nodes of the ring and, according to the paper, relaxes quorum requirements in the face of node failures and network partitions; the detail of this relaxation via hinted handoff comes from secondary technical sources, not from the paper. Conversely, if the application cannot tolerate reading stale data, prefer strict quorum with R+W>N and accept refusing writes when fewer than W of the first N nodes in the list are reachable.

Use it in a design

Practice the cases that rely on this concept in the System Design school.