YouTube: scaling MySQL with Vitess

01 · CONTEXT

A MySQL that needed to grow

YouTube stored metadata such as users, tags, and descriptions in MySQL. Vitess was created in 2010 to solve the MySQL scaling problems faced by the YouTube team, and this case tells the path that led to it.

The evolution followed a common path: a single server, then a primary with several read replicas, then partitioning, and finally, sharding of the data.

02 · REPLICAS

First, scale reads

To relieve the database, YouTube created a primary for writes and a replica for reads. Since reads kept overloading the replica, it added more replicas, which served as a temporary solution.

1. The database now has a primary that receives the writes and replicas that serve the reads.

2. When reads still overload the replica, the remedy is to add more replicas.

3. Since replication is asynchronous and the replicas run on weaker machines, they can fall well behind the primary.

4. The writes all still go to the primary; when they exceed what it can handle, the remaining option is to split the data into shards.

Replicas have a cost. Updates cause cache misses that go to disk, and slow I/O makes replication slow; in addition, the source says that, in a replication architecture, incremental write gains cost a lot of money.

One of the solutions was to prioritize traffic by splitting the data into two clusters: one for watching videos and another for everything else.

03 · SHARDING IN THE APPLICATION

When writes exceed the limit

Finally, writes exceeded what the primary could handle and YouTube had to do sharding of the data. The application layer was changed so that, before any database operation, the code identified the right shard for the query.

1. Before each operation, the application code identifies the right shard for the query.

2. The query goes straight to the chosen shard, which holds only the portion of users assigned to it.

3. Another user may be on the other shard, so writes and reads spread across the shards.

High Scalability reports the effects of partitioning: better cache locality and less I/O, a 30% reduction in hardware, and replica lag reduced to 0.

The cost of this design appears in the Vitess documentation: MySQL has no native sharding, which forces you to write the sharding code and embed it in the application.

04 · VITESS

A proxy in the middle

Vitess took this logic out of the application code: a proxy between the application and the database started routing and managing the interactions. According to the documentation, since then YouTube has scaled its user base by a factor of more than 50, and Vitess served all of YouTube's database traffic for more than five years.

In general terms, Vitess is an open-source database clustering system for scaling MySQL horizontally; VTGate is the proxy that accepts MySQL protocol connections and routes queries to the appropriate shards.

1. The application connects to VTGate as if it were an ordinary MySQL server.

2. VTGate routes each query to the right tablet, taking into account the sharding scheme, latency, and availability.

3. Each VTTablet sits next to a MySQL instance and handles its queries, connections, and replication.

4. The Topology Service keeps the cluster state and makes it possible to discover where the tablets are.

5. In steady state, no call to the topology service is on the critical path of a query.

To the application, a keyspace appears as a single database, even when it maps to several MySQL databases. A shard is a subset of the keyspace and usually has one MySQL primary and possibly several replicas.

Vitess also protects and operates the database: it rewrites queries that can hurt performance and helps manage the server lifecycle, including detecting and recovering from primary failure.

05 · TRADE-OFFS

What changes with shards

Sharding brings costs in consistency, according to the Vitess documentation: reads across shards may not be consistent with each other, and transactions within a single shard remain ACID. In best-effort mode, transactions across shards can fail midway and leave partial commits; 2PC mode increases the write cost by about 50%.

Reads that tolerate slightly stale data should go to replicas. VTGate can be configured not to serve data from instances whose lag is above a threshold.