Speech recognition (ASR)

01 · CONTEXT AND REQUIREMENTS

What a speech recognizer needs to deliver

In the Google paper on speech recognition (ASR) on phones, the dominant paradigm is to stream the audio from the device to the server, while the decoded results stream back to the user.

For this kind of model to be useful, it needs to decode speech in streaming, in real time. It also needs to be robust to the long tail of usages and be able to use user-specific context.

Latency, here, is the delay between the person speaking and the text appearing on screen. The model needs to be at least as accurate as a conventional system, without increasing that latency.

02 · HOW TO MEASURE

WER, latency, and what WER does not see

Speech recognition research usually compares systems by word error rate (WER). The Whisper paper recalls that WER is based on string edit distance and penalizes all differences between the model output and the reference transcript, including harmless stylistic differences.

Because of this, systems whose transcripts humans would judge correct can still have a large WER due to minimal formatting differences. Whisper responds by extensively standardizing the text before computing WER.

For speed, the Google paper uses the real-time factor: processing time divided by the audio duration, at the ninetieth percentile (RT90). Lower values indicate faster processing and lower perceived latency. The metrics of error and speed appear together in the paper's quantization table.

03 · SERVER-SIDE STREAMING

From audio to text, as a stream

1. The device streams the audio to the server; this is the dominant paradigm described for speech recognition on phones.

2. The recognition service runs a streaming model, which does not need to wait for the whole utterance to start processing.

3. The decoded results stream back to the user; the delay between speaking and seeing the text is the latency.

4. The metrics node represents where word errors and speed are tracked; the Google paper reports WER and RT90.

Replacing this design with one that runs entirely on the device has, according to the paper, important implications for reliability, latency, and privacy.

04 · ON-DEVICE WITH RNN-T

An end-to-end model that fits on the phone

Earlier attempts at large-vocabulary on-device ASR usually shrank the traditional components: acoustic, pronunciation, and language models. The paper's alternative is an end-to-end model, which predicts character sequences directly and simplifies training and inference.

CTC makes a strong independence assumption: the labels are conditionally independent of each other given the audio. RNN-T removes this assumption by conditioning on the full history of previous non-blank labels. In both, an alignment label cannot depend on future acoustic frames, and this makes it possible to build streaming systems that do not wait for the whole utterance.

1. The paper's encoder is made up of layers of unidirectional LSTM cells.

2. A time-reduction layer in the encoder speeds up training and inference.

3. The prediction network depends only on label histories, not on the audio.

4. The joint network, feed-forward, receives the output of the encoder and that of the prediction network.

5. Because the prediction network's computation is independent of the acoustics, state caching is used to avoid redundant computation on identical histories.

6. Contextual biasing injects prior knowledge, such as the user's contacts, during inference; the paper uses shallow fusion.

To run faster than real time on the phone, the paper uses separate threads for the encoder and the prediction network, in an asynchronous pipeline, and quantizes the parameters from thirty-two-bit floating point to eight-bit fixed point. Symmetric quantization reduces RT90 by sixty-four percent compared with the floating-point model, which is twice as fast as real time.

Overall, the system decodes speech twice as fast as real time on a Google Pixel and improves WER by more than twenty percent relative to a conventional embedded CTC model on voice search and dictation.

05 · LONG AUDIO AND BATCH

When there is no streaming: audio windows

The Whisper paper reports a training set of 680 thousand hours of labeled audio. The model is trained on thirty-second audio segments and does not consume longer audio at once.

For audio of minutes or hours, Whisper transcribes thirty-second segments in sequence and shifts the window according to the timestamps predicted by the model. The authors observed that beam search and temperature scheduling, based on the repetitiveness and the log-probability of the predictions, are crucial for transcribing long audio reliably.

In a batch design, a message queue separates the file upload from the transcription job. The choice of queue, storage, and worker is from the design proposed here, not from the paper.

1. In this design, the client connects to the audio storage, where the uploaded files are kept.

2. In this design, the storage connects to the transcription queue, which enqueues one request per file.

3. The transcriber node applies Whisper's windowing strategy: thirty-second segments in sequence, with the window shifted by the timestamps predicted by the model.

4. In this design, the transcriber writes the final text to the repository of finished transcripts.