Gmail Smart Compose

01 · CONTEXT AND CHALLENGES

Completing the sentence while the person types

Smart Compose is a Gmail feature that uses machine learning to offer sentence-completion suggestions as the person types, so that email is written faster. It is a good example of ML system design.

Because suggestions appear on every keystroke, the model should ideally respond within 100 ms so the person does not notice a delay. At the same time, Gmail has more than 1.4 billion users, so the model needs enough capacity to suggest tailored completions in subtly different contexts.

There was also a data constraint: the researchers had no access to the emails and therefore had to develop and train a system on a dataset that they themselves cannot read.

02 · THE MODEL

What context the model sees and why a hybrid model

Typical language models learn to predict the next word conditioned on the prefix already typed. In an email, however, the typed words are only one signal: the model is also conditioned on the subject and, if the person is replying, on the previous email.

One approach would be to treat the problem as sequence-to-sequence (seq2seq) machine translation, where the source is the subject plus the previous email and the target is the email being written. Prediction quality was good, but the approach missed the latency constraint by orders of magnitude.

The solution combined a bag-of-words (BoW) model with a recurrent language model (RNN-LM), faster than seq2seq models with only a slight quality sacrifice. Subject and previous email are encoded by the average of each field's embeddings, and the joined averages feed the RNN-LM at each decoding step.

1. The diagram starts at the editor, where the signals the model receives come from.

2. The subject and the previous email are encoded by the average of each field's embeddings.

3. The joined averages are handed to the recurrent model at each decoding step, along with the words already typed.

03 · TRAINING AND SERVING

Train fast and serve at typing speed

Training runs over billions of examples and requires hyperparameter tuning. To speed it up, the team used a full TPUv2 Pod in the experiments and managed to train a model to convergence in less than a day.

Even with the hybrid model, the initial version on standard CPU had an average latency of hundreds of milliseconds. By moving most of the computation to TPUs, the average latency dropped to tens of milliseconds and a single machine could serve more requests.

1. The editor sends a suggestion request on every keystroke, and the response should ideally arrive within 100 ms.

2. In the initial version, inference ran on standard CPU, with an average latency of hundreds of milliseconds.

3. With most of the computation sent to TPUs, the average latency dropped to tens of milliseconds and one machine could serve more requests.

04 · PRIVACY AND NEXT STEPS

Bias, memorization, and model evolution

Language models can reflect human biases, with unwanted word associations and sentence completions. As for privacy, the models cannot expose private information, and the team ran extensive tests to ensure that only phrases common to multiple users were memorized.

As evolution, the team cites more advanced architectures, which will only go to production when the latency constraint can be met, and personal language models, to emulate each person's writing style.