Twitter/X recommendation algorithm (open source)

01 · CONTEXT

From the ocean of Tweets to the feed

Twitter describes the challenge as distilling about 500 million Tweets posted per day down to a handful of Tweets in the For You feed. The engineering post is dated March 31, 2023 and focuses on that home timeline feed.

The post organizes the pipeline into three stages: fetching the best Tweets from different sources (candidate sourcing), ranquear each Tweet with a machine learning model, and applying heuristics and filters, such as removing Tweets from blocked users.

The feed is assembled by Home Mixer, a service built on Product Mixer, Twitter's own Scala framework for assembling content feeds. It connects candidate sources, scoring functions, heuristics, and filters.

02 · THE PIPELINE

One request, end to end

Per request, the system tries to extract the best 1500 Tweets from a pool of hundreds of millions. On average, For You is 50% Tweets from accounts the user follows and 50% from outside the network, but this varies by user.

1. The app requests the For You feed from Home Mixer, the main service that builds and serves home timelines.

2. Home Mixer fetches candidates from different sources: the search index, the in-memory interaction graph, and SimClusters ANN.

3. Candidates come from two groups: accounts the user follows and accounts they do not follow.

4. The gathered candidates are scored by the heavy ranker, regardless of which source they came from.

5. After scoring, Home Mixer applies heuristics and filters, such as author diversity and visibility filtering.

6. Finally, the feed is mixed with ads, who-to-follow recommendations, and prompts, and returned to the app.

03 · CANDIDATE SOURCES

In and out of network

The In-Network source is the largest and fetches the most relevant and recent Tweets from accounts the user follows. It ranks those Tweets with a logistic regression model, and the most important component of that ranking is Real Graph, which predicts the probability of engagement between two users.

In the repository, the search index appears as the source of in-network posts, and the documentation says about half of the posts come from it; the index (Earlybird) uses a light ranker to rank posts. The post also reports that the team stopped using the 12-year-old Fanout Service, which provided in-network Tweets from a per-user cache.

Out of network, the problem is harder: how do you know whether a Tweet will be relevant if the user does not follow the author? The post describes two approaches. The first walks the graph of engagements and follows, asking which Tweets the people the user follows engaged with recently and who likes Tweets similar to theirs.

To run these traversals, the team developed GraphJet, which maintains a real-time graph of interactions between users and Tweets. According to the post, these heuristics serve about 15% of home Tweets, and embedding-based approaches became the largest out-of-network source.

04 · SIMCLUSTERS

Communities as embeddings

SimClusters discovers communities anchored on influential users, with its own matrix factorization algorithm. The post cites 145k communities, updated every three weeks. Users and Tweets are represented in community space and can belong to several; the more users from a community like a Tweet, the more it becomes associated with that community.

The SimClusters documentation shows that the follow graph can be viewed as a bipartite graph between producers, who are the followed users, and consumers. The representations feed Tweet recommendation through the online SimClusters ANN service.

1. From the follow graph, similarity between producers is measured by the cosine between their followers, and community detection by Metropolis-Hastings sampling assigns a community to each producer.

2. In production, the Known For set covers the top 20M producers and about 145000 communities.

3. User and Tweet embeddings are built on top of the inferred communities, and a job maintains the mapping between communities and Tweets, keeping the highest-scoring ones.

4. At request time, Home Mixer calls SimClusters ANN, which returns Tweet candidates given a SimClusters embedding.

5. The traditional algorithm would fetch 15,000 embeddings through API calls; approximate cosine reduces I/O operations to save response time.

05 · RANKING AND FILTERS

Heavy ranker and weighted sum

With about 1500 candidates, scoring directly predicts the relevance of each Tweet and is the main ranking signal. At this point all candidates are treated equally, regardless of source.

Ranking uses a neural network of about 48M parameters, trained continuously on Tweet interactions to optimize positive engagement. It considers thousands of features and produces ten labels, each the probability of a type of engagement.

The heavy ranker README explains that the outputs are combined into a final score by a weighted sum of the probabilities. The weights come from a configuration file and can be adjusted at any time; the table shows some values from April 5, 2023.

1. Before scoring, Home Mixer fetches about 6000 features needed for ranking.

2. The heavy ranker receives features of the Tweet and of the user it is being recommended to.

3. The model outputs are combined into a final score by a weighted sum, with weights coming from a configuration file read by the serving layer.

4. The heavy ranker is one of the final stages of the funnel, followed mainly by filtering heuristics.

5. If the scored-Tweets pipeline fails, a fallback pipeline in reverse chronological order exists.

After scoring, filters and heuristics balance the feed: they avoid too many consecutive Tweets from the same author, balance in-network and out-of-network Tweets, and exclude out-of-network Tweets with no second-degree connection, as a quality safeguard.

06 · SCALE AND LESSONS

Production cost and the limits of open source

The post reports that the pipeline runs approximately 5 billion times per day and takes under 1.5 seconds on average. Each run consumes 220 seconds of CPU, almost 150 times the latency the user perceives in the app.

Navi, a model server written in Rust, is described as high-performance, and its ONNX variant is used for home recommendation. The Navi README says the test and benchmark code and several configuration files were not included, for data security.