Stack Overflow: a monolith on a few servers

01 · CONTEXT

The case of the site that stayed on a few servers

Stack Overflow is the question-and-answer site for programmers. It became a classic example that a heavily used system does not need to be split into dozens of services. In 2009, High Scalability summed up the point: the case is a strong argument that scaling up is still viable for a potentially large class of problems. Here we see two snapshots: 2009, from High Scalability, and February 2016, from Nick Craver, an engineer at Stack Overflow. The numbers always belong to their time.

In 2009 the site had 16 million page views per month and ran on very few machines: two web servers and one database server with 48 GB of RAM. The architecture had two tiers, with the web code accessing the database directly.

1. The browser requests the page from one of the web servers, which runs the application code.

2. The web code queries the database directly, with no service layer in between.

3. The source records that, most of the time, the bottleneck was the database, which concentrates the work.

02 · THE 2016 ARCHITECTURE

A lot of traffic, few machines

On one day in February 2016, the Stack Overflow load balancer received about 209 million HTTP requests, of which about 66 million were page loads. For this, the post lists 4 SQL Server servers, 11 web servers, 2 Redis servers, 3 tag engine servers, 3 Elasticsearch servers, and 4 HAProxy load balancers.

The author states that, after the optimizations and the new hardware, a single web server would be enough, and that this has been tested by accident, successfully, a few times. He himself cautions that working does not mean it is a good idea: the spare machines exist on purpose.

1. DNS sits with an external provider with servers close to almost everyone. Traffic reaches the load balancer, which terminates TLS and picks the destination by the host header.

2. The load balancer spreads requests across the web servers. The question-and-answer application is multi-tenant: a single program serves all the sites in the network.

3. Before going to the database, the web tier checks the cache: first the application's own, then Redis.

4. If no cache has the data, the query goes to SQL Server, the source of truth that Redis and Elasticsearch data also come from.

5. The primary takes almost all of the load and copies data asynchronously to replicas, including one in another, disaster recovery data center.

6. Searches leave SQL and go to Elasticsearch; the service tier runs the tag engine and keeps the search index up to date.

The tag engine is queried, for example, when someone opens the questions page of a tag. It runs in the service tier, separate from the web tier, and not inside it.

03 · TWO-LEVEL CACHE

Redis as the second level, and invalidation

Stack Overflow uses a cache with two levels. The first, called L1 in the post, is the HTTP cache inside the application itself on the web servers; the second, L2, is Redis. When a server misses on both, it fetches the value from the origin, for example a database query, and writes the result to the local cache and to Redis. The next server may miss L1, but finds the value in L2 and saves the query.

A local cache across several servers creates a consistency problem: if one server removes a value, the others keep copies. The post solves this with Redis's own publish/subscribe mechanism, which tells the other servers to clear L1. Despite about 160 billion operations per month, each Redis instance stays below 2% CPU.

1. Server A receives a request and finds the value neither in the local cache nor in Redis.

2. It fetches the value from the database and writes the result to the local cache and to Redis.

3. Server B misses the local cache, but finds the value in Redis and does not query the database.

4. When A removes the value, it publishes a notice on Redis and B clears its local copy, keeping the layers coherent.

04 · MONOLITH AND SCALE

Database up, application out, monolith where it helps

The main question-and-answer application is a monolith that is multi-tenant: a single program serves all the sites in the network, and the post says the whole network could run in a single application pool on a single server. Even so, there are other separate applications, such as Careers and the APIs.

There is also a service tier, separate from the web tier, for heavy work such as the tag engine. The reason given is that periodically loading the changed posts is expensive, and they do not want to repeat that load on all nine web servers: three times is enough and gives safety.

In 2009 the same source organized the lessons around scaling up and out: databases scale up and applications scale out, and that is why it is useful to separate the application and database functions so each can grow independently. The table summarizes the pros and cons the source points out.

The lack of redundancy of scale up, pointed out in 2009, appears answered in 2016: the post states that all servers and services have dual redundancy in another data center, and explains that the surplus of web servers serves gradual builds, headroom, and redundancy. See disponibilidade e load balancing for these patterns.

The split, therefore, is not dogma: the monolith is kept where it helps and the expensive load is separated where cost demands it.

05 · WHERE THE DATABASE HURT

Bottleneck, joins, and what to take out of SQL

In 2009 the source records that the database was the bottleneck most of the time. It also says Stack Overflow copied part of the database design from Wikipedia, which was a mistake: it would require a large refactoring to avoid excessive joins, which still cost a lot even with almost the whole database in RAM. The same source reminds us that memory is cheap and that filling the server with RAM gives performance almost for free. It is an example of performance antipattern.

In 2016 the use of SQL is simple on purpose: the post says simple is fast, and the database has only one stored procedure. What is expensive leaves SQL: search goes to Elasticsearch, because SQL CPU is expensive and Elastic CPU is cheap, and the design of busca stays independent of the transactional database.

For real-time updates, such as notifications and votes, the site uses websockets instead of polling, which is much more efficient at their scale, with about 500 thousand simultaneous connections at the peak.

The response-time gain came from new hardware and a lot of tuning in the applications themselves: a question page took on average about 22.7 ms, of which about 19.1 ms in the application. The author stresses that performance is still a feature.

06 · LIMITS AND LESSONS

What to take away and what not to copy

In an interview, before proposing microservices, ask what real problem the split would solve. The case shows a design with a monolith, layered cache, a strong database with replicas, and separate search, and the point is to justify each piece by a bottleneck, not by fashion.