GitLab 2017: rm -rf on the primary database and 5 backup methods that failed

01 · CONTEXT

One primary, one standby, and a false sense of security

On January 31, 2017, GitLab.com went down for hours after production data was deleted from the primary database server. The company's public post-mortem is the source of this case. The central lesson fits in one sentence: having a backup is not the same as having disaster recovery, and only a tested restore proves that it exists.

The architecture was simple. One primary database received all the traffic, and a hot-standby secondary existed only for failover. A single database had to bear the entire load, and the replica, by design, did not serve as a backup: it copies a wipe as well.

1. The application sends all queries to the single primary.

2. The primary replicates its data to the secondary, which stays on standby.

3. The secondary exists to take over if the primary fails. That does not make it a backup: whatever is deleted there disappears too.

02 · THE CHAIN OF EVENTS

From a load spike to a wiped directory

Two events coincided: a spike in spam and a process trying to remove an employee mistakenly flagged for abuse. The load on the primary rose, and it discarded WAL (write-ahead log) segments before the secondary copied them, so replication stopped.

Without WAL archiving, the secondary had to be rebuilt by hand: wipe the replica's data directory and copy everything from the primary. This work was manual, not automated, and poorly documented. During the attempts, the command to wipe the directory was run on the primary, and about 300 GB were already gone when the engineer noticed the mistake and stopped it.

03 · THE BACKUPS

Four procedures, none usable

When looking for the backups, the first finding was an empty storage bucket. There were four recovery procedures in place, and none met the need at that moment.

1. The daily logical dump was supposed to feed the bucket. When they looked, the bucket was empty.

2. The job ran on an application server and used an old version of the dump tool against a newer database version, so it ended with an error.

3. The failure notice went out by email, but the lack of DMARC made the destination reject it. The failure was silent.

4. The disk snapshot went to staging for testing. It worked, but it was not made for disaster recovery.

5. The replica served failover and had already been wiped during the rebuild. Azure snapshots were not enabled on the database servers.

04 · THE RECOVERY

Restoring from staging

The only option that reduced the loss was a manual snapshot taken by an engineer about six hours before the failure; the alternative was to lose almost a day of data. The copy came from staging, which used slow, cheap disks, and took about 18 hours, with the disks as the bottleneck. On the afternoon of February 1, the database came back without webhooks, restored later.

1. Without the primary, users cannot use the service.

2. Staging held the only usable copy, from hours before the failure.

3. The copy to production was stuck on slow, limited disks, and the restore took many hours.

There was permanent loss: an estimated at least thousands of projects and comments and hundreds of new accounts. Git repositories and wikis were unavailable, but did not lose data. To avoid reusing IDs that may have been issued before the outage, the database's ID counters were advanced by a large margin.

05 · LESSONS

What the design needs to prove

The post lists improvements that address each failure: automated restore tests, investigation of point-in-time recovery with continuous archiving, and backup monitoring. Observability here means alerting on what should exist, such as a recent backup, and not only on what failed noisily.

The underlying reasoning comes from the text itself: the ideal environment is one in which you can make mistakes and recover quickly and with little impact, which requires practicing recovery regularly.