Serving data from multiple data centers

01 · CONTEXT

Why leave a single datacenter

Serving data from multiple datacenters has a name: multihoming, operating from multiple datacenters at the same time. The challenge appears in the System Design Primer's list of additional interview questions and is a topic of multi-region architecture.

There are two reasons to do this. The first is failure: datacenters fail for many reasons. The second is performance: geolocation lets you bring operations closer to the user, and being closer is only possible if the data is also close to them.

The summary of the talk that serves as the source notes that operating in a single datacenter is easy, and that the hard part is accepting writes in more than one place.

1. The starting point: users, a load balancer, the service, and the database, all in the same datacenter.

2. This is the easiest option to operate.

02 · GETTING THE USER TO THE DATACENTER

DNS, latency, and geolocation

The first decision is how each user reaches a datacenter. The DNS translates a domain name into an IP address, and some DNS services can route traffic using several methods, among them latency and geolocation.

With latency-based routing, the DNS service finds out which region gives the user the lowest latency and responds with that region's record value. The documentation itself warns that actual latency can vary considerably from AWS's data when the resources are not in an AWS region.

With geolocation routing, the choice of resource comes from where the user is. This works by mapping IP addresses to locations, and it is possible to create a default record for IPs that have no mapping.

1. The user asks DNS for the service name and receives an IP address.

2. With latency-based routing, DNS picks the lowest-latency region for that user and responds with its address.

3. With geolocation routing, the choice starts from where the user is.

4. With the address in hand, the user talks to the load balancer of the indicated region, which hands off to a service.

A CDN does something similar for content: it serves from locations closer to the user.

03 · DATA ACROSS DATACENTERS

The replication options and what each costs

Getting the user to a datacenter is the easy part. The hard part is deciding what happens to the data. The talk summary lists the multihoming options: backups, master and replica, multi-master, two-phase commit (2PC), and Paxos. To compare them, it is worth looking at consistency: with strong, any read after a write returns what was written; with eventual, you eventually see what you wrote, but not immediately.

With a single master, one master datacenter receives the writes and the others replicate from it. Replication is usually asynchronous, which brings a risk of data loss if the master fails before the new writes reach the other nodes.

1. Datacenter A has the master database; B has a replica that receives copies from it.

2. Replication is usually asynchronous, and there is a risk of data loss if the master fails before the new writes reach the replica.

3. Geolocation serves reads near the user, but writes remain limited to one datacenter.

4. If the master goes down, failover may be read-only until the master is moved to another datacenter.

In multi-master, all datacenters serve reads and writes. In exchange, the writes must be merged later, and the best you get is eventual consistency. To merge, you need to impose an order on all writes, and there is no global clock.

1. Both databases accept local reads and writes.

2. Writes on one side are sent to the other and merged later; the best possible outcome is eventual consistency.

3. Merging requires imposing an order on all writes, and there is no global clock.

4. If one datacenter goes down, failover is easy because the other already accepts writes.

In multi-master systems, conflict resolution weighs more as write nodes are added and latency grows, and most are weakly consistent or have higher write latency because of synchronization. The summary also says that using more than two datacenters is fundamentally harder.

In two-phase commit, data is written to all datacenters or none, with strong consistency and transactions. The price is that transactions are serialized through a coordinator. Paxos is fully distributed, unlike two-phase commit, and it is enough for a majority of nodes to agree to consider a piece of data persisted; even so, writes have high latency because of the extra coordination rounds.

04 · A REAL DECISION

What Google App Engine chose, according to the talk

The summary we start from describes a talk by Ryan Barrett, lead of the Google App Engine datastore, at the Google I/O conference in 2009. The numbers below are the summary's, from that time, and not current values.

The central goal was a strong consistency model for programmers. In addition, App Engine wanted to survive datacenter failures and have write performance not far from that of a typical relational database.

With these goals, 2PC was never seriously considered because write throughput was very important; writes could end up in the range of two hundred milliseconds. Paxos was also left out: the summary says they did not want to pay the cost of about one hundred fifty milliseconds on writes.

The choice was master and replica across datacenters: true multihoming adds latency from the extra hop between datacenters, and master and replica gives most of the benefits of the better forms while keeping lower-latency writes. At the time, the datastore used this mechanism, with writes taking between thirty and forty milliseconds.

05 · IN THE INTERVIEW

How to organize the answer

A strong answer separates three questions. First, how the user reaches a datacenter, which is a routing decision. Second, where writes are accepted, which defines the possible consistency. Third, what happens when a datacenter goes down, which shows the real cost of the previous choice.

Whoever chooses master and replica should talk about the data-loss window and the read-only mode during failover. Whoever chooses multi-master should talk about how conflicting writes are ordered and merged. Whoever chooses consensus or two-phase commit should talk about the latency price.