Classic design · Mid
YouTube video search
Classic interview design. Key points: multimodal embeddings, inverted index fusion with ANN, and reranking.
Short answer
Classic interview design. Key points: multimodal embeddings, inverted index fusion with ANN, and reranking. In a video service, the user types text into the search box and the system shows the videos most relevant to that text. The ByteByteGo chapter fixes the scope to keep things simple: text-only queries, English only, and no personalization.
01 · CONTEXT AND SCOPE
Searching videos by text
In a video service, the user types text into the search box and the system shows the videos most relevant to that text. The ByteByteGo chapter fixes the scope to keep things simple: text-only queries, English only, and no personalization. Relevance is determined by the video's visual content and textual data.
The exercise numbers are one billion videos on the platform and ten million video-query pairs to train the model. The notes from a public ML interview repository add a scale requirement of 50M DAU.
02 · FORMULATION
Two searches that complement each other
The ML objective is to rank videos by relevance to the query. The chapter splits the work into two components: a visual search, which compares the query with the videos' visual content, and a text search, which compares the query with titles, descriptions, and tags.
| Aspecto | Visual search | Text search |
|---|---|---|
| What it compares | The query with the videos' visual content | The query with the videos' titles, descriptions, or tags |
| Technique | Two encoders, one for text and one for video, and a dot product between the embeddings | Inverted index, with Elasticsearch as an example engine |
| Training cost | Requires training the encoders | None, because the inverted index is not ML-based |
The inverted index is not ML-based, so the text search has no training cost, and training concentrates on the visual search encoders. Merging the two lists is the role of the fusion layer, covered later.
03 · MODELS AND TRAINING
Text and video encoders
Before entering the model, the text becomes a numeric vector in three steps: normalization, tokenization, and converting tokens to IDs. Since the dataset already comes annotated, the chapter skips the data engineering step.
For text, statistical methods are fast but do not capture contextual meaning and produce sparse representations. Transformers generate different embeddings for the same word depending on context, and the chapter chooses an architecture like BERT.
| Aspecto | Video-level model | Frame-level model |
|---|---|---|
| How it works | Processes the whole video to create the embedding, usually with 3D convolutions or Transformers | Samples frames, generates one embedding per frame, and aggregates the embeddings, for example by averaging |
| Custo | Computationally expensive, because it processes the entire video | Generally faster and computationally cheaper |
| Temporal aspects | Captures temporal understanding, according to the notes from a public repository | Generally does not understand temporal aspects, such as actions and movements |
The chapter adopts a frame-level model, such as ViT, and trains both encoders with contrastive learning.
The data already arrives annotated as video-query pairs, which removes the need for data engineering.
The text goes through normalization, tokenization, and conversion of tokens to IDs before reaching the text encoder.
On the video side, the frame-based model starts by preprocessing the video and sampling frames.
The text becomes an embedding in a Transformer such as BERT, and the video becomes an embedding in a frame-based model such as ViT, with the frame embeddings aggregated.
The text and video encoders are trained with contrastive learning.
04 · EVALUATION
How to measure a search
In the evaluation set, each query has a single associated video. Therefore the numerator of precision@k is at most 1, and precision@k and mAP help little. Recall@k also fails: if one model puts the relevant video at position 15 and another at position 50, both have recall@10=0. MRR fixes the shortcomings of recall@k and is the chosen offline metric.
| Online metric | What the source points out |
|---|---|
| CTR | Does not track whether the clicked videos are relevant to the user |
| Video completion rate | The user may watch only part and still find the video relevant |
| Total watch time | It is a good indicator of result relevance |
05 · SERVING
Query, fusion, and indexing
In the prediction service, the visual search encodes the query and uses the nearest neighbor service, with approximate algorithms (ANN) to speed up the search. The text search uses Elasticsearch. The fusion layer combines the two lists, and the re-ranking service applies business logic and policies.
The diagram starts with the search service, which receives the text query.
On the visual side, the query is encoded into an embedding.
The nearest neighbor service uses approximate algorithms (ANN) to find the video embeddings most similar to the query's.
On the text side, Elasticsearch returns videos whose titles and tags overlap with the query.
The fusion layer receives the two lists and combines them into one.
Re-ranking adjusts the list with business logic and policies, and the final list goes back to the user.
| Aspecto | Weighted sum | Additional model |
|---|---|---|
| How it combines | Re-sorts the videos by the weighted sum of the predicted relevances | Uses an extra model to re-sort the videos |
| Custo | It is the simplest option | More expensive, because it requires training, and slower at serving |
| Chapter's choice | Adotada | Not adopted |
On the other side of the search is indexing. A trained video encoder computes the embeddings, which are indexed, and Elasticsearch indexes titles, manual tags, and automatically generated tags. When the video arrives without tags, an independent model, the auto-tagger, generates tags that may be noisier but are still worthwhile.
The creator uploads the video and, in general, provides tags to identify it.
A trained video encoder computes the embeddings, which are then indexed.
When manual tags are missing, the auto-tagger generates tags, which may be noisier but are still worthwhile.
Titles, manual tags, and automatic tags go into Elasticsearch's text index.
06 · NEXT STEPS
What the chapter leaves for later
The chapter itself acknowledges that the design was simplified and that in practice the system is much more complex. Among the improvements it lists are a multi-stage design, with candidate generation and ranking, more video features, such as duration and popularity, and the use of interactions, such as clicks and likes, to build and label data.
Another point raised is the presence of near-duplicate videos in the final output, which can hurt the searcher's experience.
Frequently asked questions
What is YouTube video search?
Classic interview design. Key points: multimodal embeddings, inverted index fusion with ANN, and reranking.
What does the YouTube video search design cover?
It is organized in these sections: Searching videos by text, Two searches that complement each other, Text and video encoders, How to measure a search, Query, fusion, and indexing, and What the chapter leaves for later.
How long is YouTube video search and what level is it?
It takes about 14 minutes to read and is rated Mid level.
Now design it yourself
Run this case as an interview in the System Design school: you make the decisions, the tutor challenges them and you see how your design scores.