Atlassian 2022: deletion script takes down 775 customers (883 sites) for up to 14 days

01 · CONTEXT

A site is the container that was deleted

On April 5, 2022, starting at 7:38 UTC, 775 Atlassian customers lost access to their products, and the outage lasted up to 14 days for some of them. The company's public post-incident review is the source of this case.

In a multi-tenant architecture, a single service serves multiple customers, and each data store contains data from multiple customers. The container that groups a customer's licensed products is the site, and it was the site container that was deleted in this incident.

1. The site is the container that groups a customer's licensed products.

2. A site's data does not live in a single database or repository: it is distributed across the architecture.

3. Each data store holds data from multiple customers, and Atlassian had to manually extract and restore individual pieces of the backups.

02 · THE DELETION

A request with the wrong ID type

Atlassian needed to remove a legacy standalone app, Insight – Asset Management, from the sites of customers who had it installed, and the engineering team used an existing script and process for that.

The company describes two problems. The first was a communication failure between the team that requested the deletion and the one that carried it out: instead of the app IDs, the IDs of the entire site were provided. The second was the deletion API, which accepts both site and app IDs and assumes the input is correct; there was no warning to confirm the type of deletion requested.

Peer review followed the standard process, focused on which endpoint was called and how, and did not cross-check the site IDs provided to see whether they referred to the app or to the entire site. Testing in staging would not have caught the error either, because those IDs did not exist in that environment.

1. The script deletes the sites in sequence, following the input list it received.

2. On the first production run, the script ran against 30 sites and deleted the app from those sites, with no other effects.

3. The API accepts a site ID or an app ID and assumes the input is correct, with no warning about the type of deletion.

4. On the next run, the list contained site IDs in place of the app IDs, for a set of 883 sites.

5. The sites were deleted immediately, with no warning signal to the engineering teams.

03 · IMPACT AND DETECTION

Internal monitoring did not see the deletion

The script started running at 7:38 UTC and finished at 8:01 UTC on April 5, and the result was the immediate deletion of 883 sites, representing 775 customers.

The Jira family, Confluence, Atlassian Access, Opsgenie, and Statuspage became unavailable to the affected customers. The company states that it was not a cyberattack and that there was no unauthorized access to customer data.

The first support ticket from an affected customer was created at 7:46 UTC. Internal monitoring did not detect a problem, because the sites were deleted through a standard flow. The major incident management process was triggered at 8:17 UTC.

The same script also deleted customers' identifiers and contacts, such as the site URL and the administrators. As a result, customers were prevented from opening technical tickets through the normal support channel, and it took days to obtain a reliable list of contacts.

04 · RECOVERY

Could restore one site, not a large batch

The first customers were restored on April 8, and all sites were progressively restored by April 18.

Two concepts guide the analysis of disaster recovery. RTO answers how fast data can be recovered and returned to the customer; RPO, how recent the data recovered from the backup is. The company itself sums up that, in this incident, it missed the RTO and met the RPO.

Atlassian could restore an individual data store to an earlier point in time and knew how to recover a single site. What did not exist was the ability to select a large set of sites and restore, from the backups, all of their interlinked products.

1. In Restoration 1, each deleted site required creating a new site, and each product, service, and data store had to restore its data.

2. The new site came with new identifiers, which are considered immutable, and many systems embed them in data.

3. Restoration 1, once optimized, took about 48 hours per batch of sites and recovered 53% of the affected users across 112 sites.

4. Restoration 2 recreated the site records in all systems, reusing the old identifiers, which removed more than half of the mapping steps.

5. Restoration 2 took about 12 hours per site and recovered 47% of the affected users across 771 sites.

With the first approach, restoring all customers would take three weeks, which is why the company proposed the second. Reusing the identifiers removed more than half of the steps of mapping between old and new identifiers, including the need to coordinate with each third-party app vendor for each site.

Manual validation also required mobilization: the company had to mobilize more than 450 support engineers to run checks, with 24/7 coverage shifts.

05 · DATA LOSS

The restore point and the caveat

The databases have full and incremental backups that allow choosing a point in time within the 30-day retention period. For the main data stores, Atlassian used a restore point five minutes before the sites were deleted, and this fixed point gave consistency across all data stores. The company reports that it met its one-hour RPO.

The company's summary says no customer lost more than five minutes of data. The data section carries a caveat: for 57 customers restored early on, the lack of consistent policies and the manual recovery of backup snapshots caused some Confluence and Insight databases to be restored to a point more than five minutes before the deletion. The inconsistency was discovered in an audit after the restoration, and the company says it recovered the rest of the data and contacted those customers.

06 · LESSONS

Reversible deletion and rehearsed restoration

The main announced change is to globally prevent the deletion of customer data and metadata that has not gone through a process of soft delete. In this model, data is removed from production only after a retention period expires. The company also immediately blocked bulk site deletions until the changes were made.

For soft delete, Atlassian plans a staged rollout: test first on its own sites and then move customers progressively. It also requires testing the restoration of deleted data before running in production, and a tested rollback plan.

In the program for DR, the company says it will use the automation and the learning from the incident to speed up the program to the RTO defined in its policy, and that it will start running regular DR exercises that restore all products for a large set of sites.