From zero to millions of users

01 · CONTEXT

Start simple: everything on one server

This case has no specific product: the question is how a system grows from a few users to many, and what changes at each step. In an interview, the good answer does not jump to the final architecture; it starts from the simplest and only adds a piece when it can say which bottleneck it solves.

The starting point is a single server running everything: web application, database, and cache. When traffic grows, the first step is to separate the web tier from the data tier, which lets each be scaled independently.

1. At the start, the client talks to a single server that concentrates the application and the data; it is simple to operate, but any limit of that server is the limit of the system.

2. The database moves to its own server: now the web tier and the data tier grow at different paces, each with the resources it needs.

02 · SCALE

Vertical, horizontal, and the stateless web tier

There are two ways to add capacity. Vertical scaling puts more CPU and memory on the same machine; it is simple, but has a hard limit, and a single machine is also a single point of failure. In contrast, horizontal scale adds more servers and is what sustains large systems.

With more than one web server, someone has to split the traffic. This is the role of the load balancer, which distributes incoming requests among the servers in the pool. Clients now talk only to the load balancer, and the web servers are no longer accessed directly.

For any server to be able to handle any request, session state moves out of the web servers into a shared store. A stateless system is simpler, more robust, and more scalable, and this is what allows web servers to be added or removed according to load.

03 · DATA AND READS

Replication, cache, and CDN

In the data tier, replication separates the roles: the primary database accepts only writes, and the replicas receive copies of the data and serve the reads. This helps performance and availability. If the primary goes down, a replica is promoted to the new primary and a new replica takes its place; in production, this promotion is more complicated than it seems.

Two pieces take load off the database and the application. The cache stores in memory what is read frequently and modified rarely. A CDN is a network of geographically dispersed servers that delivers static content, such as images, videos, CSS, and JavaScript, from a point close to the user.

1. The load balancer splits requests among the web servers; none of them is the single entry point.

2. The web servers do not hold session state; therefore any of them serves any request and instances can be added or removed.

3. Static content is delivered by the CDN, from close to the user, and never even reaches the web servers.

4. What is read a lot and changed little stays in the cache; the web servers query the cache before going to the database.

5. Writes go to the primary database and reads go to the replica, which receives copies of the data from the primary.

6. If the primary database goes down, the replica is promoted to the new primary and the system keeps operating.

04 · ASYNC AND OPERATIONS

Queues, multiple data centers, and observability

Slow work does not need to block the response. With a message queue, the producer publishes the message and the consumers process it later, at their own pace. This decoupling allows producers and consumers to scale independently, and that is why the queue is a preferred architecture for scalable and reliable applications.

To serve users in distant regions and survive a site outage, the system can run in more than one data center. In a significant outage, all traffic is directed to a healthy data center. As the system grows, logs, metrics, and automation also become essential: monitoring error logs helps identify problems in the system.

1. The web server publishes the task to the queue and responds without waiting for the result; the queue decouples the producer from the consumer.

2. Consumers take messages off the queue at their own pace; if the queue fills up, just add consumers.

3. Web servers and consumers send logs and metrics to a central point, where errors can be searched.

05 · SHARDING

When a single database is not enough

When data grows beyond what a single database can handle, horizontal scaling of the data tier is sharding: add more servers and split the data among them. The cost is new problems, such as the celebrity problem, also called a hot key: excessive access to a specific shard can overload it.