GitHub 2012: failover promotes a lagging replica

01 · CONTEXT

A new cluster that switches primaries on its own

The availability post GitHub published in September 2012 is the source of this case. It describes two outages early in the same week and points to the automatic failover of the main production database as the root cause of both.

In mid-August, operations replaced a pair of DRBD-based MySQL servers with a three-node cluster. The cluster presents the application with two virtual IPs, one read-write and one read-only, managed by Pacemaker and Heartbeat. Percona Replication Manager coordinates replication and moves the active role (the master that accepts reads and writes) and the standby role (read-only replica) between the nodes.

According to the post, the new design gave more efficient failovers. In the previous architecture, switching databases required a cold start of MySQL; in the new cluster MySQL runs on all nodes all the time, and failover just moves the virtual IP after flushing transactions and setting the read_only variable.

1. The application talks to the database through the read-write virtual IP, which points to the node in the active role.

2. The standby node is a read-only replica of the active node.

3. The cluster manager runs the health checks and moves the virtual IP and the roles between the nodes.

4. Since MySQL already runs on all nodes, failover boils down to moving the virtual IP, with no cold start.

02 · MONDAY

Two failovers nobody wanted

It started with a database migration that generated more load than operations had ever seen from this kind of migration. The load was so heavy that the Percona Replication Manager health checks failed on the master.

In response, the manager moved the active role to another server in the cluster and stopped MySQL on the node it judged failed. The chosen database had a cold InnoDB buffer pool and performed poorly; the site's query load on that cold cache soon made the health checks fail again, and the active role went back to the original server.

1. With the migration load, the manager's health checks fail on the active node.

2. The manager moves the active role to another server and stops MySQL on the node it judged failed.

3. The chosen node has a cold InnoDB buffer pool and responds poorly to the site's query load.

4. The health checks fail again and the active role returns to the server where it was.

5. The author of the post turns on maintenance-mode, in which the manager performs no health checks or automatic failover.

Site performance recovered gradually, as the buffer pool reached normal levels.

03 · TUESDAY

The partition and the election of the lagging node

The next morning, a developer alerted operations to incorrect query results coming from the node in the standby role. The investigation concluded that the previous day's maintenance-mode had prevented actions that would make the node elected as standby switch replication masters and start replicating.

While trying to turn maintenance-mode off, a Pacemaker segfault occurred that resulted in a partition of the cluster state. Two nodes, called a and b, rejected most of the messages from the third, c, while c rejected most of the messages from the other two.

In the first cluster, the election was interrupted by messages from the second and MySQL was stopped. In the second, a single-node one, node c was elected and later messages from the two-node cluster were discarded. Node c was precisely the one operations had already determined to be out of date.

The team detected the problem and shut down the out-of-date node, to end the partition and prevent further data divergence. As a result, all access to the production database, and therefore to github.com, went down.

1. The three nodes exchange cluster messages, and the cluster requires majority agreement before acting.

2. A Pacemaker segfault partitions the cluster: a and b reject most of c's messages, and c rejects theirs.

3. In the cluster of a and b, the master election is interrupted by messages from the other side and MySQL is stopped.

4. In the single-node cluster, c is elected, and c was the node operations had already determined to be out of date.

5. The team shuts down the out-of-date node to end the partition, and all access to the production database goes down.

04 · THE DAMAGE

Data out of sync between stores

The data divergence made inconsistencies between MySQL and other data stores possible. GitHub uses Redis to look up dashboard feed entries and repository routes from ids automatically generated by MySQL; for records created in the window, the relationships between the two stores went out of sync.

05 · THE DECISION

Failover only when operations says so

The post states that if someone in operations had been asked whether the failover should have occurred, the answer would be an unequivocal no. It notes that there are many situations in which automatic failover is a great availability strategy, but concludes that availability of the primary production database is not one of them.

For that reason, the Pacemaker configuration was changed so that failover of the active role occurs only when initiated by a member of operations. The company is also investigating ways to keep unavoidable failovers from hurting performance, including mechanisms to warm the buffer pool of the slave databases, and is auditing the Pacemaker and Heartbeat stack.

1. Failover of the active role now depends on a member of operations, and the manager does not initiate it on its own.

2. For unavoidable failovers, the company is investigating ways to warm the buffer pool of the slave databases.

3. The Pacemaker and Heartbeat stack undergoes an audit focused on the code path that triggered the segfault.