Discord: trillions of messages

01 · CONTEXT

A database for messages that are never deleted

Discord decided early on to keep all chat history forever, so users can come back at any time and see their data on any device. That called for a database that grew just by adding nodes, with good data model for reading messages by channel.

The choice was Cassandra. The partition key was the channel, and the messages inside it were ordered by a Snowflake ID, which carries the timestamp. So that a partition would not grow without bound, the channel was cut into time buckets.

1. The client sends the message to the messages API.

2. The API writes to the partition formed by the channel and a bucket, a static time window. The bucket keeps the partition of an old channel from growing without bound.

3. Each partition is replicated on three nodes, and operations use quorum consistency.

In 2017 the cluster had 12 nodes, with replication factor three. By early 2022 it already had 177 nodes and stored trillions of messages. The volume was not the main problem: it was how it was distributed across channels.

02 · WHAT BROKE

Hot partitions

A server of friends sends orders of magnitude fewer messages than a server with hundreds of thousands of people. In Cassandra, a read costs more than a write, since it consults memory and possibly several files on disk. Many concurrent reads on the same channel create a hot partition.

1. An announcement arrives at a huge server and all members open the same channel.

2. The reads all land on the same partition, on the same three replicas.

3. The replicas saturate. Because they also store other channels and operations use quorum, latency rises even for people unrelated to the announcement.

Adding nodes does not solve this problem. Spreading different partitions helps with total volume, but a partition cannot be split across nodes, and the popular channel stays stuck on the same replicas. This is a classic performance antipattern: the bottleneck is in the access pattern, not in total capacity.

03 · THE OPERATIONAL COST

Compaction backlog and GC pauses

Cassandra writes to immutable files on disk, the SSTables, and depends on background compaction to keep reads fast. With overloaded nodes, the team fell behind on compactions, and reads got more expensive.

1. Each write goes to memory and then to immutable files on disk (SSTables); compaction merges those files in the background.

2. With the node overloaded, compaction falls behind and each read consults more files, getting slower.

3. A long garbage-collection pause takes node A out of service, and nodes B and C absorb its load.

Because it is written in Java, the database required constant garbage-collection tuning, and the pauses caused latency spikes. In extreme cases an operator had to restart and manually babysit the node. This kind of cost only shows up for those with good observabilidade.

04 · THE DECISION

ScyllaDB and a data service in front of the database

The team migrated to ScyllaDB, which is Cassandra-compatible and written in C++, and therefore has no garbage collector. It was a database selection decision driven by the real problem, and even so ScyllaDB can also have hot partitions.

That is why the data services came in: intermediary services, written in Rust, between the API and the database, to control concurrency over hot partitions. Their main function is request coalescing, a form of very short-lived cache: the first request creates a task and the following ones subscribe to it, so the database receives just one query. In practice, this significantly reduces traffic spikes against the database.

1. A data service goes between the API and the database. The application no longer talks directly to storage.

2. Requests are routed by channel with consistent hashing, and all requests for a channel land on the same data service instance.

3. Identical simultaneous requests become a single database query (request coalescing).

4. Storage changes to ScyllaDB: Cassandra-compatible, written in C++, with no garbage collector.

Even so, Cassandra still had hot partitions, only less often; the data service bought time to prepare the ScyllaDB cluster and carry out the migration.

The migration had to move trillions of messages with no downtime. New data began to be written to both databases at the same time, and a migrator rewritten in Rust moved messages at up to 3.2 million per second. Validation compared a small percentage of reads across the two databases, and the cutover happened in May 2022.

05 · INTERVIEW LESSONS

What to take from this case

The partition key is a product decision: it reflects how users access the data, not just how you write it. Protecting the database from bursts of identical reads can be cheaper than adding nodes, and isolating the application from storage behind a service makes switching databases feasible.

Switching databases did not eliminate the underlying problem: hot partitions remain possible. The layer in front of the database is still part of the solution. In the interview, start with the access pattern and only then decide between scaling horizontally and changing the key or the access layer.