GitHub: 43 seconds of partition, 24 h degraded

01 · CONTEXT AND ARCHITECTURE

MySQL on two coasts, with Orchestrator in command

Most of GitHub's user-facing services run in its own data centers, and the data lives in MySQL clusters. Each cluster follows the classic primary-replicas arrangement: a single node accepts writes, applications direct writes to the primary, and each cluster has up to dozens of read replicas.

Who decides when the primary has died and who replaces it is Orchestrator, which handles failure detection and failover. Its nodes talk to each other through Raft consensus, with one or two nodes per data center, so that no single data center holds a majority.

1. Users talk to the applications, which write to the East Coast primary and read from replicas.

2. The primary replicates to the replicas on both coasts; the West Coast ones hold a remote copy.

3. Three Orchestrator nodes, in different places, form a Raft group and watch the primary.

The 2018 post describes what this arrangement handles: when a data center is completely isolated, the failover decision is delegated to the quorum of the non-isolated data centers. In the October 2018 incident, the Orchestrator nodes on the West Coast and in the East Coast public cloud formed a quorum and started failing clusters over to the West Coast. The incident post notes that the Orchestrator configuration must be aligned with the applications' expectations.

Read this way, the lesson is not in how the Orchestrator leader is elected, but in promoting primaries from one coast to the other: that promotion is what the applications and cross-coast replication must withstand.

02 · THE PARTITION

Forty-three seconds without a network

At 22:52 UTC on October 21, routine maintenance to replace failing 100G optical equipment cut connectivity between the East Coast network hub and the primary East Coast data center. Connectivity returned after 43 seconds.

1. The link between the network hub and the primary data center goes down; the primary stays alive, but is no longer visible from the outside.

2. The Orchestrator node inside the data center loses contact with the other nodes, and the Raft group starts deselecting the leader.

3. The West Coast nodes and the East Coast public cloud nodes can still see each other and form a quorum.

4. With a quorum, Orchestrator starts directing the clusters' writes to the West Coast.

The network came back before almost anyone noticed, but Orchestrator's decision had already been made. At 22:54 UTC, alerts began to indicate failures across several systems.

03 · FAILOVER AND DIVERGENCE

The correct failover that went wrong

Orchestrator reorganized the cluster topologies on the West Coast. When the network returned, the clusters in the two data centers contained writes that the other did not have.

1. Orchestrator promotes a new primary on the West Coast, which starts accepting writes.

2. The old East Coast primary was left with writes that never reached the West Coast.

3. The East Coast applications now write to the West Coast, and cross-coast latency weighs on every operation.

4. The team pauses webhook delivery and Pages builds to protect the data already received; the queue starts to grow.

The divergence was small: one of the busiest clusters had 954 writes in the affected window, and the total not replicated to the West Coast was relatively small. Even so, the West Coast data center spent more than 30 minutes receiving writes.

The design point is here: during the partition there were two primary candidates, and each side had reason to believe it was right. It is the classic consistency problem when replication is asynchronous: however small the difference, nobody knows which side should prevail without an external criterion.

04 · THE RECOVERY

What the twenty-four hours cost

At 23:07 UTC the team manually locked the deployment tool, and at 23:13 UTC it became clear that the problem affected several clusters. At 23:19 UTC came the deliberate choice to pause webhooks and Pages builds, and at 00:05 UTC the plan: restore from backups, synchronize the replicas at both sites, and return to a stable topology.

1. The backup of the affected clusters is started and becomes the source of the restore.

2. The East Coast replicas are restored from backups; restoring several terabytes takes hours.

3. Gradually the restored replicas start replicating from the West Coast; the larger clusters are still restoring.

4. The primaries return to the East Coast, and the West Coast goes back to being a replica.

Even with the primaries back at 11:12 UTC, replication lag was growing instead of falling at 13:15 UTC, and dozens of replicas were still several hours behind. Only once the replicas had synchronized was the failover to the original topology possible, at 16:24 UTC.

There were still more than five million hook events and 80 thousand Pages builds in the queue, and the team had to balance that load against normal traffic. At 23:03 UTC on October 22 everything had been processed, after 24 hours and 11 minutes of degradation. Manual reconciliation of a few seconds of writes was still in progress.

05 · WHAT THE DESIGN SHOULD HAVE

Write availability versus consistency

In CAP terms, during the partition the system chose to keep write disponibilidade availability by promoting another primary, and paid with consistency: two sides with different histories. In multi-region architecture, failing over automatically across regions with asynchronous replication turns a short network failure into a data problem.

1. Orchestrator now has a regional boundary: within a region it promotes replicas automatically.

2. A partition between regions does not promote anyone on its own: the primary stays the same, and the decision to switch regions goes through a person.

3. The long-term goal is to serve traffic from more than one data center at the same time, in an active/active/active design.

GitHub listed as actions configuring Orchestrator to prevent promotion of primaries across regional boundaries, the goal of N+1 redundancy at the facility level, and testing its assumptions more proactively.

06 · LESSONS

What to take into the interview

The lessons below are a didactic interpretation of the case, not claims made in the post.

First, failover automation needs to know the topology: promoting a node in the same region and promoting a node in another region are very different risk decisions. Second, with asynchronous replication, failover has a data cost, and the design must say in advance who wins when there is divergence. Third, recovery time depends on the size of the data and the replicas' lag, not on the duration of the failure. Fourth, pausing secondary work, such as webhooks, is a legitimate tool for protecting integrity.