Figma and Notion: horizontal sharding of Postgres

01 · CONTEXT

When a single Postgres is no longer enough

Notion and Figma both reached the limit of a single Postgres. This case compares how each one decided on, designed and carried out horizontal sharding, focusing on how to move the data without stopping the product.

At Notion, product usage was going to outgrow the Postgres monolith, which had gone through five years and four orders of magnitude of growth. Whoever was on call woke up to CPU spikes on the database.

The inflection point was Postgres's VACUUM, which began stalling constantly and stopped reclaiming the space taken up by dead tuples. The bigger risk was transaction ID wraparound, a safety mechanism in which Postgres stops processing all writes; the team treated it as an existential threat to the product.

Scaling the instance vertically was not enough: query and maintenance process performance usually degrades well before the table reaches the maximum size the hardware supports.

At Figma, the database grew almost a hundredfold since 2020. In 2020 it was a single Postgres on the largest physical AWS instance; by the end of 2022 there was already a cache, read replicas and a dozen vertically partitioned databases.

Vertical partitioning also had a ceiling, because the smallest unit it separates is a table. Some tables, with several terabytes and billions of rows, were already too large for a single database, and the team saw reliability impact during vacuums. The highest-write tables would soon exceed the maximum IOPS supported by Amazon RDS.

02 · CHOICES

Build the routing or adopt a ready-made cluster

Notion opted for application-level sharding: defining its own partitioning scheme and routing queries through the application logic. It considered off-the-shelf solutions, such as Citus for Postgres and Vitess for MySQL, but their cluster logic is opaque and the team wanted control over how data is distributed.

Figma evaluated CockroachDB, TiDB, Spanner and Vitess. Switching to any of them would require a complex data migration between two stores, and the team already had extensive experience operating RDS Postgres in-house. A NoSQL alternative was also left out: Figma's relational model is very complex and NoSQL APIs do not offer that versatility.

Instead of reimplementing a generic sharded relational database, which made no sense for a small team, Figma built a smaller, tailor-made set of features. One example: it chose not to support atomic cross-shard transactions, because the failures of those transactions could be worked around.

Splitting a table across several physical databases has a price. According to Figma, foreign keys and global unique indexes are no longer guaranteed by Postgres, and writes to some databases may succeed while others fail, which requires product logic that is resilient to these partial failures.

03 · KEYS AND ROUTING

How each query finds its shard

Notion sharded all tables reachable from block by foreign key. Not all of them needed to be split, but if a record stayed in the main database and the related block lived on another shard, the lack of cross-database transactions could cause inconsistencies: deleting the block could succeed while updating the comments failed.

Since each block belongs to exactly one workspace, the workspace ID became the partition key. Users generally query one workspace at a time, which avoids most cross-shard joins.

The team set limits of 500 GB per table and 10 TB per physical database and arrived at 480 logical shards evenly distributed across 32 physical databases. The number 480 was chosen because it has many divisors, which allows adding or removing hosts while keeping the distribution uniform; with any power of two, you would have to double the number of hosts to scale.

1. A request arrives at the application; it generally queries data from a single workspace.

2. The application uses the workspace ID, the partition key, to find the logical shard. All routing happens in the application itself, with a single source of truth.

3. The query goes straight to the physical database that holds that logical shard, with no second routing layer inside Postgres, as there would be with natively partitioned tables.

4. Since most queries stay within a single workspace, most cross-shard joins are avoided.

At Figma there was no single good shard key. A unified key would require a composite key, a new column on every table, expensive backfills and a major refactor of product logic; so the team chose a handful of keys, such as UserID, FileID and OrgID, and grouped related tables into colos. Within a colo there are joins between tables and full transactions, as long as they are restricted to a single shard key.

Many of these keys used auto-incrementing or timestamp-prefixed IDs, which would create hotspots. Migrating to more random IDs required an expensive, lengthy data migration, so Figma routed by the hash of the shard key instead. The cost is that range scans become less efficient, since sequential keys land on different shards; this pattern is not common in Figma's code.

To route, Figma built DBProxy, a Go service between the application and PgBouncer. Queries without the shard key require scatter-gather, which spreads the query across all shards and merges the responses, and each one generates the same load it would on an unsharded database. For that reason the team restricted the query language: joins only between tables in the same colo and via the shard key. To choose this subset, it ran logical planning in shadow mode on real production traffic.

04 · MIGRATION AT NOTION

Double-write, backfill, verification and cutover

Notion's playbook for any migration has four steps. In double-write, writes go to both the old and the new database. In backfill, with double-write on, the old data is migrated. In verification, the integrity of the new database is checked. In cutover, you actually switch to the new database. This is the basis of an online data migration.

1. An audit log records all writes to the tables being migrated; this is what the catch-up will read from.

2. The catch-up process walks the log and applies each update to the new databases, making the necessary modifications.

3. With writes already propagating, the backfill migrates the old data. It compares versions before writing and ignores records with a more recent update.

4. Catch-up and backfill can run in any order: the new databases end up converging to a replica of the monolith.

5. In verification, a script samples random UUIDs and checks the adjacent ranges. In shadow reads, the application reads from both databases, compares, discards the sharded copy and logs the discrepancies.

6. At cutover, the server was stopped so the catch-up could finish propagating the writes to the shards, and the product was offline for five minutes of scheduled maintenance.

Notion chose the audit log because logical replication did not keep up with the write volume of the block table during the initial snapshot. The final script took about three days to backfill production. The team also prepared and tested a reverse audit log, to go back from the shards to the monolith if needed; in the end, it did not need to roll back.

Verification had costs and caveats. Shadow reads increased API latency, but gave confidence that the cutover would be smooth. The migration and the verification were implemented by different people: with the same person on both steps, there would be a greater chance of repeating the same mistake and weakening the verification.

05 · MIGRATION AT FIGMA

Logical sharding first, physical split later

Figma started from different goals. It wanted to avoid backfills of large tables, keep the ability to roll back even after a physical sharding operation was completed, and avoid double-write, which is hard to implement without downtime or without giving up consistency.

The core idea was to separate logical sharding, at the application layer, from physical sharding, in Postgres. Reverting logical sharding was a simple configuration change; reverting a physical operation is possible, but requires more complex coordination to keep the data consistent.

1. Application queries go through DBProxy, a Go service between the application and PgBouncer.

2. In DBProxy, the parser turns the SQL into a syntax tree and the logical planner extracts the query type and the logical shard IDs.

3. The physical planner maps logical shards to physical databases and rewrites the query for the right shard.

4. Before the physical split, each logical shard is a Postgres view and all reads and writes go through them. The system looks sharded, even with the data on a single host.

5. If a bug appeared, it was enough to redirect traffic back to the main table, in seconds.

The views carried a risk: in some cases they could fundamentally change how the Postgres planner optimizes queries. To validate, Figma collected a corpus of sanitized production queries, ran load tests with and without views and confirmed minimal overhead in most cases and under ten percent in the worst. It also built a shadow reads framework, which sent read traffic through the views to compare performance and correctness.

1. With logical sharding validated, the physical operation comes next: copy the data from one database, distribute it across several and redirect read and write traffic.

2. Instead of filtered logical replication, each destination receives the entire data set and can only read and write the subset that belongs to its shards.

3. The operation goes from one database to one to N, and must tolerate the case where only some of the databases succeed.

4. DBProxy starts routing to the new databases and updates its topology state so it does not send queries to the wrong database during the splits.

5. On the first table, failover had only ten seconds of partial availability on the primaries and no availability impact on the replicas.

The first horizontally sharded table went to production in September 2023. End to end, sharding that first table took about nine months. Figma saw no latency or availability regressions after sharding.

06 · LESSONS AND TRADE-OFFS

What each path cost and what was left to do

In retrospect, Notion would say it should have sharded earlier. It waited until the database was under heavy pressure and had to be frugal with migrations so as not to increase the load, which kept it from using logical replication for double-write. The partition key was not yet populated in the old database, and populating it would have worsened the monolith's load; so each row was populated at the time of writing to the shards, with a custom catch-up script.

Notion also noted that double-write throughput was the bottleneck of the cutover. With one more week optimizing the script so that catch-up took under 30 seconds at cutover, it might have been possible to switch at the load balancer, with no downtime. And it would have combined the primary key and the partition key into a single column.

Figma, for its part, lists what is still missing: atomic cross-shard transactions for critical cases and distributed unique indexes, since today only indexes that include the shard key are supported. It also intends to reassess its in-house approach against an open source or managed solution.