TikTok/ByteDance Monolith (+ For You)

01 · CONTEXT

Recommendation that needs to learn in real time

Monolith is a recommendation system described in a paper by authors affiliated with ByteDance. The paper opens by saying that a scalable, real-time recommendation system is vital for many businesses driven by time-sensitive customer feedback, such as short-video ranking and online ads.

The paper's starting point is that general-purpose deep learning frameworks fall short in this scenario. One reason is that they completely separate the batch training stage from the serving stage, which prevents the model from interacting with customer feedback in real time.

Recommendation data differs from problems like language or computer vision in two respects: the features are mostly sparse, categorical and change dynamically, and the distribution of the training data is non-stationary, a phenomenon the paper calls concept drift.

Since the same person can change interests every minute, the paper argues for updating the serving models with new feedback as close to real time as possible.

02 · ARCHITECTURE

Worker, parameter server and two training stages

The architecture follows TensorFlow's distributed Worker-ParameterServer model: worker machines execute the graph computations, and PS machines store the parameters and update them according to the gradients. The parameters split into two sets: the dense ones, which are the neural network's weights, and the sparse ones, which are the embedding tables of the sparse features.

Training is split into two stages. The batch stage retrains on historical data, useful when the model architecture changes. After deployment to serving, training does not stop: in the online stage, the worker consumes real-time data and updates the training PS, which synchronizes its parameters with the serving PS periodically.

1. In the batch stage, the worker trains on historical data; this serves to retrain when the model architecture changes.

2. After the model enters serving, training does not stop: in the online stage, the worker starts consuming real-time data.

3. The worker updates the training PS with what it learned from the new data.

4. The training PS periodically synchronizes its parameters with the serving PS.

5. The model server answers the user with the serving PS's parameters, and the synchronization takes effect immediately on the user's side.

03 · EMBEDDINGS

A collision-free embedding table

A recommendation system's embedding table is enormous: users and items are orders of magnitude more numerous than the words of a language model, and the table must grow as new IDs are admitted. Such a table would hardly fit in the memory of a single host, and some frameworks represent it with fixed-size dense variables.

Many systems work around this with hashing, which assumes that IDs have uniform frequency and that collisions are harmless. According to the paper, as the table grows the chance of key collisions increases and model quality deteriorates.

Monolith developed a key-value HashTable for the sparse parameters, built on a Cuckoo Hashmap, which inserts new keys without colliding with existing ones. Table lookups and updates are also native TensorFlow operations.

Inserting every new ID would exhaust memory quickly, so the table filters. The paper notes that rare IDs contribute little to quality and that old IDs rarely contribute to the current model. There are two filtering methods before admission: an occurrence threshold, adjustable per model, and a probabilistic filter. In addition, IDs expire after a period of inactivity, adjustable per table.

04 · STREAMING ENGINE

Kafka, Flink and the online joiner

The streaming engine uses a Kafka queue to record user actions and another for features. At the center is a Flink job, the online joiner, which concatenates the features with the action labels and produces training examples, written to yet another Kafka queue. That queue is consumed by both online training and batch training, and the design allows switching between the two.

1. User actions go to a Kafka actions queue.

2. The features go to another Kafka queue, the features queue.

3. The joiner, a Flink job, concatenates the features with the action labels and produces training examples.

4. Actions and features arrive with no guarantee of temporal order, and a unique key per request lets the two be paired.

5. When the action arrives, the joiner first queries the in-memory cache and then the on-disk key-value store, where the features wait.

6. The examples go to the examples queue: online training reads straight from it, and batch training goes through HDFS.

The joiner faces practical problems. Actions and features arrive with no guarantee of temporal order, so each request carries a unique key that lets the two be paired. And an action may arrive a few days later, while keeping all features in cache simply would not fit in memory. That is why there is an on-disk key-value store: when an action log arrives, the joiner first queries the in-memory cache and, if it does not find it, the on-disk store.

There is also class imbalance: negative examples can be orders of magnitude more numerous than positive ones. A common strategy is negative sampling, which changes the distribution of the trained model; to compensate, Monolith applies a log odds correction at serving, so that the online model is an unbiased estimator of the original distribution.

05 · SYNCHRONIZATION AND FAILURES

Synchronizing without stopping serving and accepting losses

Production models are usually several terabytes, and the serving PS cannot stop serving while it is updated. Transferring the entire model from the training PS would strain the network and the PS's memory, because receiving the new model would require twice the model's size in memory.

Monolith's solution is incremental synchronization. A touched-keys set holds the IDs whose embeddings have been trained since the last synchronization, and only that subset of the sparse parameters goes from the training PS to the serving PS, at intervals of minutes. The incremental package is light on the network and does not cause a memory spike.

1. Only part of the embeddings is trained at a time, and the training PS keeps the keys touched since the last synchronization.

2. At intervals of minutes, only the touched embeddings go to the serving PS; the package is light on the network and does not cause a memory spike.

3. The serving PS keeps serving the model server while it receives the updates.

4. The dense parameters change more slowly and are synchronized less often, which can leave them at an older version than the sparse ones.

5. If a training PS fails, it loses a day of updates, because the snapshot of all training PS is taken every day.

The dense parameters change more slowly than the sparse embeddings, so they are synchronized less frequently, on a daily level. This can leave the dense part at a more stale version than the sparse one; the paper states that this inconsistency can be tolerated.

For failures, Monolith snapshots all training PS every day. In case of failure, a PS loses a day of updates, but the performance degradation was tolerable in the paper's experiments. The paper presents this as a trade of system reliability for real-time learning.

In the synchronization experiments, model quality improves consistently as parameter synchronization increases, and the paper's conclusion is that the interval should be as short as possible. The paper itself limits this to what compute cost and reliability can bear.

06 · O FEED FOR YOU

What TikTok's public page describes

The TikTok newsroom page, published in June 2020, explains that the For You feed is powered by a recommendation system that delivers each person the content they are likely to be interested in. It covers product signals and rules and does not describe the training infrastructure.

Among the factors are user interactions, such as videos liked or shared, accounts followed, comments and content created; video information, such as captions, sounds and hashtags; and device and account settings, which get lower weight than the other measured points.

A strong indicator of interest, such as watching a long video from start to finish, gets more weight than a weak indicator. Neither follower count nor an account's past performance is a direct factor in the recommendation.

People who join the app can choose interest categories; those who do not choose start with a generalized feed of popular videos. Each new interaction helps the system learn the user's interests.

There are also variety and eligibility rules: the feed generally does not show two videos in a row with the same sound or from the same creator and does not recommend duplicate, already-seen or spam content. Newly uploaded or under-review videos, and spam, may also be ineligible for recommendation.

Read together, the two sources cover different layers: the paper shows how to build an online training setup with a collisionless embedding table, and the newsroom shows that product signals and rules enter the ranking. This case does not connect one to the other beyond that.