Search ranking

01 · CONTEXT AND SCOPE

Before designing, clarify the search

The source treats the problem as an interview project: before designing, clarify whether the search is general, like a web search engine, or specialized, like product search. The case here is e-commerce item search: the user types a text query and the system shows the most relevant items.

The cited business goal is to increase CTR and conversion. As requirements, the source lists response time, accuracy, and scalability, with a scale of fifty million daily active users.

In the ML formulation, relevance can be defined as a weighted sum of click, successful session, and conversion. The input is the text query and the output is the ranked list of the most relevant items.

1. The diagram has three boxes: the client, the search service, and the item index.

2. The edges connect the client to the search service and the service to the index.

Educative describes the search ranking module of its course as a walk through ranking system design, choice of metrics, and result filtering. Only this public description was used here.

02 · METRICS

Offline and online do not measure the same thing

The source separates offline and online metrics. Among the offline ones, it lists precision and recall at k, MRR, mAP, and NDCG, and picks NDCG because of non-binary relevance.

03 · LAYERED ARCHITECTURE

From query to result

The source's architecture is layered: query understanding, candidate generation, first-stage ranker, second-stage ranker, blender, and filter, in that order.

1. The user types a text query in the search box; the system must show the most relevant items.

2. The query first goes through query understanding, which gathers spell checker, normalization, expansion or relaxation, and intent classification.

3. Candidate generation searches the inverted index and focuses on recall.

4. In the source's example, the first-stage ranker uses a linear model.

5. In the source's example, the second stage uses a deep neural network and ends with five hundred items.

6. The blender assembles the results page and the filter comes last, in the source's order.

Query understanding gathers spell checker, normalization, expansion, which adds alternatives, or relaxation, which removes terms (the source gives the example of removing good), and intent or domain classification.

Candidate generation focuses on recall and reduces millions or billions of items to tens of thousands. For text retrieval, the source uses an inverted index, which maps words to their positions in a set of documents; after query expansion, it searches this index and gets items with a relevance score.

The relevance score is a weighted linear combination. The source's examples are term matching, for example TF-IDF, and item popularity, for example the number of reviews or purchases.

The source recommends multi-stage ranking when there are more than ten thousand items to choose from or when the query rate per second exceeds ten thousand. With a hundred thousand items, the example uses a linear model in the first stage and a deep neural network in the second, ending with five hundred items.

The blender produces the results page (SERP) and combines results from several sources, such as text search and visual search.

04 · RANKING

Pointwise in the first stage, pairwise in the second

For ranking, the source describes two families of models. Pointwise takes a user and item pair and returns a relevance score. Pairwise takes a user and two items and predicts whether the first's score exceeds the second's; the source describes it as more natural for ranking and more complicated.

In the second stage, LambdaMART is a variation of MART with an objective function changed to improve pairwise ranking; LambdaRank is neural-network-based, with a pairwise loss that minimizes ranking inversions. The source says to choose between the two according to the available training data and capacity. For the first stage, it indicates latency on the order of microseconds.

05 · DATA AND TRAINING

Labels that come from engagement

The labels come from online engagement data. Positive samples are from users who engaged, with a click, dwell time above T, add to cart, or purchase. Negatives are from items without engagement, plus random negatives, for example from pages ten onward.

To generate training data, one of the source's options is human raters, which it points out as expensive and not scalable. The other is online engagement data, in which each type of engagement gets a score.

As an example of volume, the source starts from five million queries per day, which become ten million samples per day. It recommends using at least one week of data to capture daily patterns and, for the temporal split, three weeks, with the first two thirds for training and the last week for validation and testing.

Among the features, the source organizes the query-item pair ones, such as text matching on title, description, and category, and historical engagement, for example the item's click-through rate for that query.

06 · FILTER AND WRAP-UP

What happens after ranking

After ranking, the source foresees a re-ranking with business logic, which includes filtering inappropriate items and ensuring diversity, between exploration and exploitation.

The source points to two ways of filtering: rule-based filters and aggregators, or an ML model. In the model, it is a binary classification of the probability that the item is inappropriate, with data from human raters and user feedback, such as reports and ratings.

Among the other talking points, the source's list mentions position bias.