Canva: 50 million uploads per day

01 · CONTEXT

Canva's media service

According to the engineering team, users upload 50 million new media items per day, and Canva is built with microservices: each service exposes an API, has isolated persistence, and belongs to a small team.

On the access pattern, the media service handles far more reads than writes, most media is rarely modified after it is created, and reads concentrate on recently created media, with the exception of the stock photo and graphics library.

02 · THE MYSQL LIMIT

When the relational database stopped keeping up

For much of Canva's history, most microservices were thin layers over MySQL hosted on AWS RDS. The database was scaled first vertically, with larger instances, and then horizontally, with eventually consistent reads on read replicas in some services.

1. The typical design of Canva's microservices is a thin layer over MySQL hosted on RDS; the diagram shows this design for the media service.

2. For horizontal scaling, some services started reading from MySQL replicas, with eventually consistent reads.

3. Hard limits on MySQL's replication rate imposed a ceiling on the write rate on read replicas.

The following table gathers the MySQL limits the team reports.

03 · BUYING TIME

Extending MySQL and choosing the destination

In mid-2017, with the number of media items approaching 1 billion and growing exponentially, the team began investigating migration paths, with a strong preference for incremental approaches.

To extend MySQL's life, the team moved the content metadata, the most modified part of the schema, to a JSON column, denormalized some tables to reduce lock contention and joins, and removed foreign key constraints.

Near the end of the MySQL solution's life, the team implemented a simple sharding scheme, to avoid the ext3 table file size limit and the replication throughput ceiling.

1. The solution was optimized for the most common request, lookup by ID used when loading designs.

2. Less common requests, such as listing all of a user's media, required inefficient scatter-gather queries.

In parallel, the team investigated and prototyped long-term solutions. With a short deadline, a preference for a managed solution, prior experience with simpler DynamoDB services, and a prototype, it chose DynamoDB as the provisional target. It also needed a migration strategy with no impact on users and a cutover with no downtime.

04 · LIVE MIGRATION

Copying the history without stopping the product

The team considered writing to both databases on every create or update, building and replaying an ordered log of all changes, among other options. The chosen approach would give full control over the data mapping, allow migrating progressively, live, and take load off the MySQL cluster as early as possible.

The design uses message queues that only identify the media item, instead of carrying the update content. The diagram shows the flow in six steps.

1. The current state of each media item is read from the MySQL primary by a worker at the time of copying it.

2. Queued messages only identify which media item was created, updated, or read, without the update content.

3. A worker processes each message, reads the current state from the MySQL primary, and updates DynamoDB if necessary, which allowed messages to be reordered or repeated arbitrarily.

4. Creates and updates go to the high-priority queue and reads to the low-priority one; workers only read from the low-priority queue after draining the high-priority one.

5. A scan process walks through all media, starting with the most recently created, and with backpressure only advances while the low-priority queue is approximately empty.

6. Eventually consistent reads of individual media items are now served from DynamoDB, with fallback to MySQL for the few media items not yet replicated.

The team implemented a dual-read process with comparison to test replication. Requests that do not identify the media by ID, such as finding all of a user's media, could only be served after all media had been replicated.

05 · CUTOVER

The riskiest part: switching the writes

Switching all writes to DynamoDB was the riskiest part of the process. The new code used transactional and conditional writes to guarantee the same contracts as the previous implementation.

To reduce risk, the team wrote a cutover runbook, using the flag system to be able to switch reads back to MySQL in seconds, and rehearsed it as the change moved through the development and staging environments. The production cutover went smoothly, with no downtime or errors.

06 · RESULT AND TRADE-OFFS

What was gained and what was lost

Canva's monthly active users have more than tripled since the migration. According to the team, DynamoDB has been very stable, with autoscaling as Canva grows, and costs less than the AWS RDS clusters it replaced.

Looking back, the team says that, facing the same problem today, it would strongly consider mature hosted NewSQL products again. The media service today stores more than 25 billion user-uploaded media items, with another 50 million per day.