Concepts
Geo-replication, failover, and RPO/RTO targets
Intuition
Imagine a store with an important ledger of accounts. So you don't lose everything if a building burns down, you keep copies of the ledger in buildings in different cities. Then two practical questions come up: if the main building disappears, how much of the latest entries are you willing to lose, and how long are you willing to go without serving customers until another building takes over? The less you are willing to lose and wait, the more expensive it gets to keep the copies up to date and ready to take over.
How it works
Industry terminology (not a claim of the papers): RPO (recovery point objective) is the maximum amount of data, measured in time, that one accepts losing in a disaster; RTO (recovery time objective) is the maximum time accepted until the service is back. The papers show both sides of this trade-off. In Spanner, data lives in zones, which are the unit of physical isolation and the set of replication locations; each tablet is replicated by a Paxos group, with replicas across several zones and synchronous replication, in contrast to Bigtable, which only offered eventually consistent replication between datacenters. The replication configuration is defined per directory (applications choose where each piece of data lives) and directories can be moved between Paxos groups while clients keep operating. Failover is automatic: each group's leader is elected through time-based leases, whose default duration is 10 seconds, and in the test that killed an entire zone without warning, reads dropped to almost zero and recovered in about 10 seconds, tied to the lease duration. The student's reading, as an interpretation and not as a claim of the paper: with writes replicated synchronously by a Paxos majority, the accepted data loss tends to be zero (RPO close to zero) and the RTO is on the order of the leader re-election time. Dynamo chooses another point: the write is accepted even with unreachable replicas, via sloppy quorum and hinted handoff, in which another node holds the write with a hint of the real destination and hands it off when the replica comes back, at the cost of temporarily divergent replicas that read repair and anti-entropy reconcile later. Also as industry vocabulary (Well-Architected and the AWS DR whitepaper), the ladder of strategies goes from backup and restore, through pilot light and warm standby, up to active-active, with lower RPO and RTO and higher cost at each rung; Spanner with active replicas across several zones comes close to the active-active rung.
In practice
Start by defining RPO and RTO per service based on the cost of losing data and of being down, and only then choose the rung of the ladder (industry terms): backup and restore for what tolerates hours, active-active for what barely tolerates any loss. As an example from the papers, F1 uses five Spanner replicas (two on the US west coast and three on the east) and reports that automatic failover was almost invisible, while a Dynamo-style shopping cart prefers to keep accepting writes during failures and resolve divergences later. In multi-region architecture, the central question is how much cross-region write latency you are willing to pay to get an RPO close to zero.
Use it in a design
Practice the cases that rely on this concept in the System Design school.