Meta DLRM

01 · CONTEXT

A recommendation model that does not fit on one device

Neural-network-based recommendation models differ from other deep networks because they have to work with categorical data. The company's AI blog says that the lack of public details on models and of representative datasets was holding back the research community, and announces the open-sourcing of DLRM. The paper provides the implementation in PyTorch and in Caffe2.

These systems are used in tasks such as ad click-through rate prediction and ranking.

02 · ARCHITECTURE

From categorical data to click probability

DLRM handles two kinds of input. Categorical features go through embeddings; continuous ones, through a bottom MLP. Then the second-order interactions are computed explicitly, and the result goes through a top MLP and a sigmoid, which gives the click probability.

The interaction is the dot product between all pairs of embedding vectors and the already processed dense features. According to the paper, this mimics factorization machines and significantly reduces the model's dimensionality, by considering only cross terms.

1. Categorical features are processed by embeddings.

2. Continuous features go through the bottom MLP.

3. The interaction layer computes the dot product between pairs of embedding vectors and the processed dense features.

4. The top MLP processes the interactions and a sigmoid returns the click probability.

03 · TWO BOTTLENECKS

Embeddings weigh on memory, MLPs weigh on computation

According to the paper, DLRM has up to several orders of magnitude more parameters than other common models, such as convolutional networks, and training can take weeks or more. That is why it is important to parallelize it efficiently.

04 · COMBINED PARALLELISM

Split the embeddings, replicate the MLPs

The paper's scheme combines both: model parallelism on the embedding tables, to ease the memory constraint, and data parallelism on the fully connected layers, to scale the computation. According to the paper, this combination is a unique requirement of DLRM, and the paper says that neither Caffe2 nor PyTorch supports this combination.

1. The embedding tables are split across the devices, and each one stores only a part of them.

2. At the end of the lookup, each device has the vectors from the tables it stores, but the top MLP and the interaction need all the embeddings.

3. The all-to-all communication, called a butterfly shuffle, redistributes the lookup result for the whole minibatch into pieces of the minibatch for all devices.

4. Each device processes different samples with a replica of the MLPs, which is data parallelism.

5. On the way back, the MLPs' parameter updates are accumulated with allreduce and applied to the replicas synchronously, leaving the parameters consistent before each iteration.

6. In the version described in the paper, the shuffle transfers are explicit copies, and the authors intend to optimize them.

05 · MEASUREMENT AND LIMITS

How the benchmark is used and where it stops

The paper presents DLRM as a benchmark for algorithmic experimentation and systems co-design. The benchmark generates random and synthetic inputs; synthetic ones make it possible, for example, to avoid sharing a dataset for privacy reasons, expressing the categorical features through distributions.

With public data, the Kaggle ads dataset has 13 continuous and 26 categorical features, which defines the size of the MLP input and the number of embeddings in the model.

In the profile of an example model, most of the time goes to embedding lookup and to the fully connected layers. On the CPU these layers weigh quite a lot, and on the GPU they are almost negligible.