Q&A platform (Quora)

01 · CONTEXT

Depth instead of speed

Quora is a social question-and-answer platform. The source course contrasts it with search engines: while those prioritize speed, Quora prioritizes depth and conversation.

Where a search engine can hit a dead end for lack of content, the community can generate new answers on demand. According to the course, the platform serves hundreds of millions of monthly active users, who generate thousands of questions per day across hundreds of thousands of topics.

The course states that the chapter is inspired by Quora's engineering blog; this article describes the design proposed by the course.

02 · REQUIREMENTS

What the system needs to do

The course's functional requirements are five: post questions and answers with text, images, and videos; upvote, downvote, and comment on answers; search existing questions; receive a feed personalized by the user's interests; and see answers ordered by usefulness, with the highest-quality content first.

The resource estimate starts from one billion users, of which 300 million are active per day. It assumes that 15% of questions contain an image and 5% contain a video, with images of 250 KB and videos of 5 MB.

03 · INITIAL DESIGN

Web and application layers, two stores

In the initial design, the web servers accept HTTP requests and pass them to the application servers. They run manager processes that delegate work to worker processes through a routing layer; this layer manages task queues, in which managers enqueue and workers dequeue tasks to execute.

1. The user sends an HTTP request; the web servers accept it and pass it to the application servers.

2. A manager process, on the web servers, enqueues the task in the routing layer.

3. The worker processes, on the application servers, dequeue the tasks to execute them.

4. Questions, answers, comments, and votes are critical data that requires high consistency and live in MySQL.

5. View counts, ranking scores, and recommendation attributes are high-throughput data and live in HBase.

The ranking requirement has two ends in the material: the requirement (answers ordered by usefulness) and the data (ranking scores in HBase). How the score is computed is not detailed in this article.

04 · LIMITS

Where the initial design fails

The course itself points out limits of the initial design. First, to process a request, data payloads travel between web and application servers, which adds network I/O latency.

Second, in-memory priority queues are a single point of failure: if a queue goes down or its process terminates unexpectedly, the enqueued tasks may be dropped and require manual reprocessing. Replicating these queues increases memory usage.

1. To process a request, data payloads travel between web and application servers, which adds network I/O latency.

2. In-memory priority queues are a single point of failure.

3. If a queue goes down or its process terminates unexpectedly, the enqueued tasks may be dropped and require manual reprocessing.

4. Replicating the in-memory queues increases memory usage.

05 · FINAL DESIGN

The fixes proposed by the course

The course's answer combines three changes: vertical sharding of MySQL, replacing HBase, treated as high latency, with MyRocks, and using Kafka for asynchronous processing of tasks.

1. Requests arrive at powerful, homogeneous service hosts, which handle in-memory cache, queues, and routing.

2. MySQL now uses vertical sharding, which spreads the load across MySQL servers.

3. MyRocks replaces HBase, described as high latency, for the high-throughput data.

4. Kafka becomes the means of asynchronous task processing.

06 · EVALUATION

Scale, consistency, and recovery

In the evaluation, the course describes using Amazon EC2 instances with 32 cores, a 20 MB shared cache, and high network throughput. These powerful hosts handle in-memory cache, queues, and routing, and their uniformity simplifies horizontal scaling. For the database, vertical sharding spreads the load across MySQL servers.

The evaluation also covers varied consistency models, which meet specific data needs, and disaster recovery, with remote backups and restore strategies to ensure the system's durability.