Instagram: 14 million users with 3 engineers (Postgres and per-shard IDs)

01 · CONTEXT

A startup, few engineers, a lot of traffic

In December 2011, the Instagram team described its own stack as a snapshot of how a startup with a small team can scale to more than 14 million users in a little over a year. At that moment there were only 3 engineers.

The principles for choosing each piece were to keep everything very simple, not reinvent the wheel, and prefer proven, solid technologies when possible. Theme: scaling horizontally.

02 · THE FRONT DOOR

From DNS round-robin to a managed load balancer

Every request went through load balancing machines of load balancing. The first version used 2 nginx machines with DNS round-robin between them, and the weak point was the time DNS takes to update when one of the machines had to be taken out of service.

1. The app queries DNS, which alternates the answers between the two nginx machines.

2. To take a machine out of circulation you have to wait for DNS to update; this is the weak point the team pointed out.

Later the team moved to Amazon's Elastic Load Balancer, with 3 NGINX instances behind it, which can be swapped and are automatically taken out of rotation if they fail a health check. SSL also began to be terminated at the ELB, which relieves the nginx CPU.

03 · APPLICATION, DATA, AND CACHE

The full stack in 2011

The application layer grew from a few machines to more than 25, and it is the easy part to scale horizontally because the machines are stateless. The team observed that the load was CPU-bound, not memory-bound.

1. The request enters through the ELB, which also terminates SSL, and is distributed to one of the NGINX instances.

2. NGINX forwards the request to an application server, a layer the team expanded as it grew.

3. The application reads from and writes to PostgreSQL, which runs in master-replica with Streaming Replication.

4. The application uses Memcached as a cache and Redis for feeds and sessions.

5. Photos go straight to S3, and CloudFront serves as a CDN to improve image load times around the world.

6. Tasks such as sharing to other networks go to a queue and are handled by background workers.

Most of the data (users, photo metadata, tags) lived in PostgreSQL. The main cluster had 12 large-memory instances and twelve replicas in another zone. The team found that EBS network disk did not offer enough seeks, so keeping the working set in memory was extremely important.

04 · WHEN ONE DATABASE ISN'T ENOUGH

Sharding without switching databases

By the end of 2012 Instagram was receiving more than 25 photos and 90 likes per second. To keep the important data in memory and quickly available, the team started to shardar the data, that is, to split it into many smaller buckets, each holding a part.

The team evaluated some NoSQL solutions, but concluded that the best fit was to distribute the data across a set of PostgreSQL servers.

They had to solve how to assign unique identifiers to each piece of data in the database. The typical single-database solution, the auto-incrementing primary key, stops working when data goes into several databases at the same time.

The team's requirements were: IDs sortable by time, to sort a list of photo IDs without fetching more information; IDs ideally 64 bits, for smaller indexes and better storage in systems like Redis; and as few new moving parts as possible.

Of the paths evaluated, Twitter's Snowflake came closest, but the complexity of operating an ID service weighed against it. The choice was a conceptually similar approach, brought inside PostgreSQL.

05 · LOGICAL SHARDS AND THE ID

Thousands of logical shards over a few physical servers

The system has several thousand logical shards, mapped in code to a much smaller number of physical shards. Each logical shard is a Postgres schema, and each sharded table exists inside each schema.

1. The application keeps in code the map of which logical shard lives on which physical server.

2. Each physical server hosts a set of logical shards, and each logical shard is a Postgres schema.

3. To grow, you move a set of logical shards from one database to another, without regrouping the data.

Each ID combines three fields: time, logical shard, and an auto-incrementing sequence.

With this sequence, the system can generate 1024 IDs per shard per millisecond.

ID generation lives inside each table of each shard, using PL/pgSQL and the auto-increment function Postgres already offers. The ID returns to the application server through the INSERT's RETURNING.

06 · QUEUES AND OBSERVABILITY

Heavy work out of the request, and eyes on the system

Sharing a photo on Twitter or Facebook, or notifying subscribers in real time about a new photo, becomes a task sent to Gearman, a task queue. Doing this asynchronously lets the media upload finish quickly, while the heavy work runs in the background.

1. The user posts the photo and the application completes the upload.

2. The application enqueues the heavy tasks, such as sharing to other networks and the feed fan-out.

3. About 200 Python workers consume the queue, split among the services they share to.

The feed fan-out also goes through Gearman, which is why posting is as responsive for a new user as for one with many followers.

With more than 100 instances, monitoring the whole system was important. The team used Munin for graphs and alerts, with custom plugins for business metrics such as signups per minute and photos posted per second, plus Pingdom for external monitoring and PagerDuty for notifications and incidents. Theme: monitoring and observability.