Slack: scaling datastores with Vitess

01 · CONTEXT

A datastore in the path of every message

Since the start of Slack, MySQL has been the storage engine for all data, in an active-active setup. In 2017 the migration to Vitess began, a horizontal scaling system for MySQL, and, according to the engineering post of December 1, 2020, it already served 99% of the query load.

Storage is critical because every message sent is persisted before moving through the real-time websocket stack and appearing to the other channel members. That is why storage access needs to be very fast and very reliable.

At the time of the post, the peak was 2.3 million queries per second, of which 2 million were reads and 300 thousand were writes, with median latency of 2 ms and p99 of 11 ms.

02 · THE ORIGINAL MODEL

Shards per workspace, active-active

Slack was born as a simple LAMP stack and stored everything in three primary MySQL-based clusters: the shards, the metadata cluster and the so-called kitchen sink.

The shards held practically all customer data, such as messages, channels and DMs, partitioned by workspace id. The metadata cluster was a lookup table linking the workspace to the shard, and the kitchen sink held what was not tied to a workspace.

1. The webapp monolith manages all data access: it has the logic to query the workspace metadata and then create the connection to the shard.

2. First it queries the metadata cluster to find out which shard the workspace is on.

3. Then it connects to the shard, which has at least two MySQLs in different data centers replicating to each other asynchronously.

4. The application prefers one of the sides by a simple hash; if a host fails, it retries the request on the other, and both accept reads and writes.

5. Data that does not belong to any workspace, such as the app directory, goes to the kitchen sink.

6. If an entire shard suffers an outage, every customer whose data is on it is left without Slack.

03 · WHY CHANGE

The limits of the per-workspace model

In the fall of 2016 Slack was handling hundreds of thousands of MySQL queries per second and thousands of hosts with shards, and the performance teams were constantly working around limitations of the per-workspace architecture. The table summarizes the limits that the source itself describes.

04 · ALTERNATIVES

Evolve in place or replace

The goal was, instead of putting all of a workspace's messages on the same shard, to shard them by channel id, which would spread the load much more evenly.

Slack wanted to stay with MySQL on its own servers in the cloud. There were thousands of distinct queries in the application, some with MySQL-specific constructs. Leaving the relational paradigm would be a much more disruptive change, and so Slack practically ruled out NoSQL, such as DynamoDB and Cassandra, and NewSQL, such as Spanner and CockroachDB.

Putting the new sharding model in the application itself was considered and even got prototypes, but there was already a lot of coupling between the application logic and how the data is stored. Besides, that path did not solve the operational problems or allow read replicas to be used more efficiently.

Vitess is built on top of MySQL and uses it as its storage and replication engine. Its built-in sharding lets you shard and grow the database without extra application logic, and it handles primary failovers and backups, keeping the application unaware of the topology through a lock server.

05 · ONLINE MIGRATION

From prototype to production data

The first production use was a small feature: integrating an RSS feed into a channel. For that, Slack had to develop integration points to route queries to Vitess, a generic backfill system that clones existing tables while the application does double writes, and a parallel double-read system with comparison.

1. The backfill system clones the existing table from the legacy database into Vitess.

2. Meanwhile, the application does double writes from code.

3. A parallel double-read system compares the responses to ensure the tables in Vitess have the same semantics as the legacy ones.

4. The first feature migrated was small, the integration of an RSS feed; it showed resilience and reliability and gave confidence to keep investing.

5. Three years after the start, 99% of Slack's MySQL traffic was already on Vitess.

The migration moved tables: a single table, making up 20% of the query load, got its own case study, and the post announced a future article on the change of strategy to move entire shards instead of tables.

06 · VITESS IN PRODUCTION

Keyspaces, regions and new services

At the time of the post, Slack operated several Vitess clusters, with dozens of keyspaces in geographic regions around the world. Each keyspace is a logical collection of data that scales by roughly the same factor: number of users, teams and channels. Vitess is used both by the webapp monolith and by other services.

1. The webapp and the other services talk to the Vitess access layer, and the application stays unaware of the database topology.

2. The lock server holds the cluster configuration metadata, so the view is updated and consistent across clients.

3. Data lives in keyspaces, logical collections that scale by users, teams and channels.

4. In March 2020 queries rose 50% in a week, and one of the busiest keyspaces was scaled horizontally with Vitess's split workflows.

5. In the international data residency product, Vitess is the storage layer, with clusters in six regions.

Slack did not adopt Vitess off the shelf: it identified gaps for its own needs and invested in filling them. Contributions include refactoring the topology metadata service to scale across isolation regions, improvements to MySQL query compatibility and vtexplain, a simulation engine that helps understand how queries will behave on Vitess.

Choosing Vitess over building a sharding layer in the monolith made it possible to use the same technology in all new services. In the CNCF case study, the connection latency to Vitess is one millisecond more on average, imperceptible to users.

07 · ASSESSMENT

Was it the right choice?

In hindsight, Slack considers Vitess the right solution. The team says this does not mean they would not have found a way to scale without it, but that, with their requirements, they would have arrived at a very similar solution.