Google Wide & Deep (Google Play)

01 · CONTEXT

Memorize or generalize

This case comes from a Google article about the Google Play app store. A recommendation system is seen there as a ranked search system: the query gathers information about the user and the context, and the output is an ordered list of items. The task is to find the relevant items in a database and sort them by objectives such as clicks or purchases.

The central challenge is balancing two capabilities. Memorization is learning the frequent co-occurrence of items or variables and exploiting history; it tends to produce recommendations more pertinent to the items the user has already acted on. Generalization, in turn, tends to increase the diversity of what is recommended.

Each side has a weak point. Cross-products of sparse variables memorize well, but do not generalize to pairs that did not appear in training. Embeddings generalize to never-seen pairs but, when interactions are sparse, can over-generalize and recommend less relevant items; for these exception rules, linear models with cross-products are more parameter-efficient.

02 · THE MODEL

Wide, deep, and joint training

The wide component is a generalized linear model, and one of the most important transformations is the cross-product, which captures interactions between binary variables and adds nonlinearity to the model.

The deep component is a feed-forward neural network. Each sparse categorical variable becomes a low-dimensional dense vector, the embedding, which starts random and is trained to minimize the final loss function.

The two components are combined and trained jointly, which is different from an ensemble. In an ensemble, the models are trained separately and only combine their predictions at inference. In joint training, all parameters are optimized at the same time, including the weights of the sum of the two parts, and the gradients flow from the output to both parts simultaneously.

One consequence is model size: in joint training the wide part only needs to complement the weaknesses of the deep part with a few cross-product transformations, instead of being a full wide model.

03 · THE SYSTEM

Retrieve candidates and rank

The store's catalog is too large to score everything on every query: with more than a million apps, that is infeasible within the latency requirement, which is usually on the order of 10 milliseconds. That is why the first step is retrieval, which returns a short list combining learned models and human-defined rules, and only then does ranking sort that list. The article focuses on the ranking model.

1. The query, with user and context variables, originates when the person opens the store and reaches the recommendation service.

2. Retrieval narrows the catalog to a short list of candidates, combining learned models and human-defined rules.

3. The model servers receive the candidates and the user features to score each app.

4. The scores come from one inference pass through the model; apps are sorted from highest to lowest score and shown in that order.

5. User queries, impressions, and actions are written to logs, which become training data.

04 · PRODUCTION

Continuous training and fast serving

The pipeline has data generation, training, and serving stages. Each example corresponds to an impression and the label is the app acquisition. Vocabularies are also generated: tables that map categorical feature strings to integer IDs.

In the experiment, the wide part uses the cross-product between the apps installed by the user and the apps shown. The embeddings are concatenated with the dense features into a vector of about 1200 dimensions, which feeds 3 ReLU layers and the logistic output. Training used more than 500 billion examples.

Retraining from scratch on every new dataset is expensive and delays the updated model, so a warm-start system initializes the new model with the embeddings and linear weights of the previous one. Before going to the servers, the model goes through a dry run and a quality comparison against the previous one.

1. The logs become labeled examples and the vocabularies are generated in the data generation stage.

2. The trainer receives the examples and is initialized with the embeddings and linear weights of the previous model, instead of training from scratch.

3. Before moving on, the new model goes through a dry run and has its quality compared to the previous one as a sanity check.

4. Once trained and verified, the model is loaded onto the model servers.

In serving, each request must be handled in about 10 ms. Scoring all candidates in a single batch with one thread took 31 ms; splitting into smaller batches and using multithreading cut client-side latency to 14 ms, including serving overhead. At peak, the servers score more than 10 million apps per second.

05 · RESULTS

What the experiments showed

The online experiment was a 3-week A/B test. The control used the previous model, a highly optimized wide-only logistic regression with many cross-products. Wide & Deep raised the app acquisition rate on the main page by +3.9% over the control, and had a +1% gain over the deep-only model; both results were statistically significant.

The offline AUC of Wide & Deep is only slightly higher, and the impact is more significant in online traffic. The authors suggest as a possible reason that, offline, impressions and labels are fixed, while the online system can generate exploratory recommendations that combine generalization and memorization and learn from new user responses.

The system was put into production on Google Play, an app store with more than one billion active users and more than one million apps.