Streaming E2E ASR for mobile
Streaming End-to-end Speech Recognition For Mobile Devices
In one sentence
Google showed that a single RNN-T (recurrent neural network transducer) model can run on a phone in real time, streaming words as you speak, and beat the conventional server-style CTC-based recognizer on both latency and accuracy.
Why this paper exists
On-device recognition used a classic pipeline: a CTC-based acoustic model, a pronunciation lexicon and a large n-gram language model compiled into a decoding graph. The pieces are big, trained separately and hard to shrink for a phone. End-to-end models (one network from audio to text) were promising, but attention encoder-decoders like LAS (Listen, Attend and Spell) need the whole utterance first, so they cannot stream.
The core idea
Use the transducer (RNN-T): its output at each step depends only on the audio and text so far, so it streams naturally. Then do the product engineering: fast on a phone CPU, robust across use cases, and able to use personal context like contact names.
How it works
- Model: an RNN-T with a stack of unidirectional LSTM layers as the encoder (unidirectional means it only looks at the past), a small LSTM prediction network over previous output labels, and a joint network that combines both to predict the next word-piece or a blank.
- Speed tricks: a time-reduction layer that stacks neighbouring frames so upper layers run at a lower frame rate, plus layer normalization to stabilise LSTM training.
- Small and fast on device: parameters are quantized from 32-bit floats to 8-bit integers, which shrinks the model about 4× and speeds up inference.
- Context biasing: contact names and other user-specific phrases are boosted during beam search with a biasing model (shallow fusion with a small contextual FST: a weighted finite-state graph of the phrases).
- Text normalization: spoken numbers ("five dollars") must come out written ("$5"); the paper addresses this by adding training data for numeric cases. Check the paper for the exact method.
Build the intuition
Analogy: a conventional recognizer is a kitchen with three separate chefs (acoustic model, dictionary, language model) passing plates. RNN-T is one chef who does everything, so it fits in a small kitchen (the phone) but needs extra tricks to learn things the dictionary chef used to know, like rare names.
With a bidirectional LSTM encoder, each frame would depend on future audio, so you would wait for the end of the utterance. That one choice is the line between streaming and offline.
Numbers worth remembering
- RNN-T on device beat the conventional CTC-based model on latency and accuracy across several test categories (per the abstract; check the results tables for exact WERs).
- 8-bit quantization: ~4× smaller model, and decoding runs faster than real time on a Pixel-class phone (check the paper for the exact speedup).
- Output units are word-pieces, not characters, which reduces the number of decoding steps.
Where it shows up today
This is the ancestor of Gboard's offline dictation and of the industry default "transducer for streaming". NVIDIA's streaming models (Cache-aware streaming Conformer), Zipformer transducers in sherpa-onnx (Squeezeformer / E-Branchformer / Zipformer) and TDT (TDT) inherit the same logic. Emformer (Emformer) swaps the LSTM encoder for a streaming Transformer.
What it got wrong, and what came next
- LSTM encoders are less accurate than Transformers and Conformers (Conformer); later Google work moved to Conformer transducers, and Emformer / Cache-aware streaming Conformer made attention encoders stream.
- RNN-T still trailed a server model on rare words and proper nouns; Google's follow-up two-pass work added a LAS rescoring pass on top of the streaming output.
- Running one blank or label decision per frame is slow; TDT (TDT) skips frames by predicting durations.
Check yourself
Q1Why RNN-T and not an attention encoder-decoder for on-device streaming?
An attention encoder-decoder computes attention over the whole encoded utterance, so it normally needs all the audio first. RNN-T emits outputs frame by frame using only past audio and past labels. That gives streaming output with no extra machinery.
Q2What does the time-reduction layer do and why does it help?
It concatenates adjacent frames, halving (or more) the frame rate for higher layers. Fewer frames means fewer LSTM steps and fewer joint-network evaluations, which cuts compute and latency on a phone CPU.
Q3How does the model handle contact names it rarely saw in training?
During beam search, hypotheses that match phrases in a user-specific biasing list get a score bonus. This injects personal context without retraining the model.
When you're back at a laptop
- Load torchaudio's Emformer RNN-T pipeline or a sherpa-onnx streaming transducer and run it on your own recording; print partial hypotheses as audio chunks arrive.
- Apply PyTorch dynamic int8 quantization (
torch.ao.quantization.quantize_dynamic(model, {torch.nn.LSTM, torch.nn.Linear}, dtype=torch.qint8)) to a small LSTM model and compare size, CPU speed and WER.