Google Translate

01 · CONTEXT

From phrase-based to neural translation

Phrase-based machine translation (PBMT) breaks the input sentence into words and phrases that are translated largely independently. Neural machine translation (NMT) treats the entire input sentence as a unit of translation.

NMT faced obstacles to production: according to Google's research blog, it was neither fast nor accurate enough for a service like Google Translate. The technical report lists three inherent weaknesses. One is slower training and inference; another is poor robustness with rare words; the third is sometimes failing to translate all the words of the source sentence.

In production, Google Translate's web and mobile apps began using GNMT for all Chinese-to-English machine translations, about 18 million per day. That pair was one of more than 10,000 language pairs in the service. The blog credits TensorFlow and TPUs with making it feasible, providing enough computing power within the product's strict latency requirements. The technical report also states that the implementation requires no language-specific tuning.

Treating translation as a machine learning system means looking together at the model, the evaluation, and the service that responds quickly.

02 · THE MODEL

Encoder, attention, and decoder

GNMT has three components: an encoder network, a decoder network, and an attention network. The model is a deep LSTM network with 8 encoder layers and 8 decoder layers, using residual connections and attention connections from the decoder to the encoder. The attention connects the bottom layer of the decoder to the top layer of the encoder, to improve parallelism and reduce training time.

Translation is an open-vocabulary problem, with names, numbers, and dates. To handle this, GNMT breaks each word into wordpieces using a trained model and, on output, converts the wordpiece sequence back into words. Wordpieces balance the flexibility of characters with the efficiency of words, and a vocabulary of 8,000 to 32,000 wordpieces gave good BLEU accuracy and fast decoding.

A single wordpiece model is used for the source and target languages. This way, the same string is segmented the same way on both sides, making it easier to learn to copy.

1. Before entering the network, the source text is broken into wordpieces by a trained model.

2. The encoder turns the source sentence into a list of vectors, one per input symbol.

3. Attention connects the encoder and decoder and lets the decoder focus on different regions of the source for each word generated.

4. The decoder produces the output one symbol at a time, until it generates the end-of-sentence symbol.

5. Beam search looks for the highest-scoring sequence, and the coverage penalty favors translations that cover the entire source.

6. The generated wordpiece sequence is then converted back into words.

03 · DECODING

Choosing the best output

Beam search looks for the sequence that maximizes a scoring function. Without length normalization, it tends to favor shorter results on average. The coverage penalty favors translations that cover the entire source sentence, according to the attention module.

To improve throughput, many sentences of similar length can be grouped into the same batch and decoded in parallel, taking advantage of hardware optimized for parallelism.

04 · PRODUCTION

Training and serving within the latency budget

In training, the system follows the classic data-parallelism paradigm, with 12 replicas running in parallel on separate machines. Each model replica is partitioned into 8 parts, placed on 8 GPUs that usually sit in the same machine.

At inference, using reduced-precision arithmetic can lower the cost considerably, and in GNMT the matrix multiplications run in 8-bit integers. In the paper's measurements, decoding on the TPU had minimal loss in log-perplexity and none in BLEU, and was 3.4 times faster than on the CPU.

The CPU, in turn, was 2.3 times faster than the GPU. Among the reasons, the authors cite that beam search forces a nontrivial data transfer between the host and the GPU at every decoding step.

1. The diagram starts with the app or website and the load balancer at the service's entry.

2. Behind the load balancer appear the translation service, the decoding host, and the TPU accelerator.

3. In TPU decoding, the embedding lookup and the attention module stay on the CPU, and the remaining quantized operations are sent to the TPU.

4. Matrix multiplications, which are the expensive operations, run in 8-bit integers.

5. In the paper's measurements, the TPU decoded 3.4 times faster than the CPU, with minimal quality loss.

6. The authors use TPUs to serve production traffic and CPUs to evaluate the model during training.

05 · QUALITY AND LIMITS

Measuring translation and recognizing what fails

BLEU does not capture all of a translation's quality, so the authors also use side-by-side evaluation with people. Raters fluent in both languages tend to capture quality better than BLEU. In the blog, the side-by-side scores range from 0 to 6.

Refinement with reinforcement learning improved BLEU but barely improved human perception of quality.

The blog reports a reduction in translation errors of more than 55% to 85% across several major language pairs, measured on sentences sampled from Wikipedia and news sites with bilingual human raters. Even so, GNMT still makes mistakes a human translator would not, such as dropping words and getting proper names or rare terms wrong, and it translates isolated sentences without considering the context of the paragraph or page.