Visual search

01 · CONTEXT AND REQUIREMENTS

What a visual search needs to do

A visual search helps users discover images visually similar to a selected image. In the book's problem statement, the images that appear first in the list should be the most similar to the query image. The scope covers only images, without videos, and personalization is out of scope: a query image produces the same results regardless of who searches.

The problem statement assumes 100 to 200 billion images on the platform, and the system must retrieve similar images quickly. The Pinterest paper describes a real system that indexes billions of images.

According to the paper's abstract, a small engineering team can build, launch, and maintain a large-scale, cost-controlled visual search system with widely available tools. One of the challenges the text cites is controlling development cost, both human and computational, at a startup.

02 · FRAMING

From the problem to the ML task

The book frames visual search as a ranking problem and adopts representation learning: the model maps images to points in an N-dimensional space called the embedding space. The embeddings are learned so that similar images end up close to each other in that space.

To rank, the input images are transformed into embedding vectors, and the similarity between the query image and the others is computed from the distance in the embedding space. The images come out ranked by these similarity scores.

For simplicity, the model uses only the image pixels, without metadata. Since models usually require fixed-size images, the image goes through preprocessing before entering the network.

The book chooses neural networks because, unlike many traditional models, they produce the embeddings needed for representation learning. Architectures based on CNNs, such as ResNet, or on Transformers, such as ViT, work well with images.

03 · CONTRASTIVE TRAINING

How the model learns to put similar images close together

In contrastive training, the model learns to distinguish similar images from different ones. Each training example contains a query image, a positive image similar to it, and negative images different from the query. The true label of the example is the index of the positive image.

What remains is deciding where the positive image comes from. The book compares three options.

The book chooses self-supervision for two reasons: there is no upfront cost, since the process can be automated, and frameworks such as SimCLR have shown promising results with a large dataset. It notes that you can switch labeling methods if the experiment results are unsatisfactory, and that it is also possible to combine the options.

The simplified contrastive loss is computed in three steps. First, the similarities between the query image and the embeddings of the other images; dot product and cosine similarity are widely used for this, while Euclidean distance tends to perform poorly in high dimensions because of the curse of dimensionality. Then, softmax makes the values sum to one, which allows interpreting them as probabilities. Finally, cross-entropy measures how close the predicted probabilities are to the true labels.

The book also mentions taking a pretrained contrastive model and fine-tuning it with the training data, which greatly reduces training time compared to training from scratch.

04 · SERVING AND INDEXING

From embedding to results list

In production, the book separates a prediction pipeline, which serves the query, from an pipeline indexing pipeline, which maintains the index.

1. The query image arrives at the embedding generation service, which preprocesses it and uses the trained model to obtain the embedding.

2. With the embedding in hand, the nearest neighbor service retrieves similar images from the embedding space, querying the index table with the image embeddings.

3. Re-ranking incorporates business logic: it filters out inappropriate results, avoids private images, and removes duplicates or near-duplicates.

4. The final list, ranked by similarity, is shown to the user.

The indexing pipeline handles the other side: all the platform's images are indexed to improve search performance, and the service keeps the index table up to date. When a creator adds a new image, the service indexes its embedding so that it can be discovered by the neighbor search.

1. Creators upload images and the system stores them along with their metadata.

2. The indexing service indexes the new image's embedding so that it can be discovered by the neighbor search.

3. Storing the embeddings of all images in an index table increases memory usage; there are optimizations to reduce it, such as quantization.

05 · NEIGHBOR SEARCH

Exact or approximate

Exact search, also called linear search, scans the whole index table and computes the distance from each point to the query point. In large systems, with N reaching billions, linear complexity is too slow. By contrast, in approximate nearest neighbor (ANN) algorithms, a specific data structure reduces the search complexity to sublinear, usually at the cost of preprocessing or additional space.

ANN algorithms fall into tree-based, LSH, and clustering-based families. Interviewers typically do not expect knowledge of every detail, and a high-level overview is enough.

Exact search guarantees exactness. ANN methods may not return the exact points, but they are more efficient at finding the closest ones; the book uses ANN to find embeddings of similar images. Two widely used libraries are Faiss, from Meta, and ScaNN, from Google.

06 · EVALUATION

How to measure whether the search is good

Search systems usually use the offline metrics MRR, recall@k, precision@k, mAP, and nDCG. The book assumes an evaluation set with candidate images and a similarity score for each query-candidate pair.

nDCG divides the DCG by the DCG of an ideal ranking. Online, the book cites CTR, which shows how often users click on the displayed items, and the average daily, weekly, and monthly time spent on the suggested images.

07 · REAL SYSTEM

What Pinterest describes in the paper

The paper describes the features extracted from images, the distributed and incremental feature extraction infrastructure, and the real-time search service. The deep features are binarized and compared by Hamming distance. For the object detectors, classifying the category first makes it possible to run the detectors only on images with a high prior probability, reducing computational cost and false positives.

With a collection of more than one billion images, it is critical to update the features incrementally, avoiding unnecessary recomputation. Old images do not change, so their features are not recomputed. The initial computation takes a little over a day on a cluster of several hundred 32-core machines, and steady-state incremental processing requires only about 5 machines.

1. One job gathers the signatures of newly uploaded images and groups them by date into epochs; another identifies missing epochs and enqueues jobs in the queue.

2. The jobs run on an automatically launched cluster of instances, sized according to the size of the update.

3. The features are stored in bulk in the repository, organized by feature type, version, and date; old images are not recomputed.

4. The index is sharded by doc-ID, and each machine stores the indexes and features of a subset of the collection.

5. Each leaf ranker re-sorts the top candidates with additional metadata, and the root ranker gathers the best results from each leaf ranker.

In the static evaluation, the image annotations served as a relevance proxy. The 16-layer VGG's FC6 brought a substantial precision gain, with acceptable latency for the applications. In the live experiment, Pinterest preferred to reduce recall if that improved relevance; the test started with 10% of eligible traffic and, after three months, visual related pins increased total repins in Related Pins by 2%.