ASR · Phase 5: Scale and multilinguality
Google USM
Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages
In one sentence
USM (Universal Speech Model) is a ~2B-parameter Conformer pretrained with BEST-RQ on millions of hours of unlabelled YouTube audio in 300+ languages, then trained with a much smaller labelled set, showing self-supervision at scale gives strong multilingual ASR with far less labelled data than Whisper.
Why this paper exists
Whisper Whisper relied on 680k hours of weakly labelled audio, which is concentrated in a few languages. Many languages have lots of YouTube audio but almost no transcripts. Google wanted captions for 100+ languages, so it needed to learn mostly from unlabelled audio.
The core idea
Pretrain the encoder on huge unlabelled multilingual audio with a simple self-supervised objective, then use a small labelled set to teach it to output text. Most knowledge comes from audio alone.
How it works
- BEST-RQ pretraining data2vec / w2v-BERT / BEST-RQ: mask parts of the log-mel input; a frozen random projection plus a random codebook maps each original frame to a discrete ID; the Conformer Conformer predicts the IDs of masked frames. Nothing in the target generator is learned.
- Multi-objective supervised pretraining: continue with the self-supervised loss, a supervised ASR loss, and text-injection objectives using unpaired text.
- Fine-tuning: attach a CTC CTC or RNN-T RNN-T decoder. Residual adapters (small per-task layers) let one frozen backbone serve many tasks.
- Long-form: chunk-wise attention keeps it stable on long YouTube audio.
Build the intuition
Analogy: a child hears thousands of hours of speech before learning to read; reading then takes a few lessons. Pretraining is the listening; labelled data is the reading lesson.
Why a random quantizer works: targets only need to be consistent (the same sound gets similar IDs), not meaningful. Predicting a masked frame's ID from context forces the model to learn phonetics and language structure.
Numbers worth remembering
- ~2B parameters, Conformer encoder.
- ~12M hours of unlabelled YouTube audio, 300+ languages.
- Labelled data is a small fraction of Whisper's (check the paper for exact hours).
- Evaluated on YouTube captions (~73 languages) and FLEURS (102 languages), reporting lower WER than Whisper.
Where it shows up today
- Google says its Chirp speech models in Google Cloud are built on USM; it also serves YouTube captions.
- BEST-RQ became a common pretraining choice for large speech encoders, including in NVIDIA NeMo.
- Same philosophy as wav2vec 2.0 wav2vec 2.0, HuBERT HuBERT, MMS MMS and IndicWav2Vec IndicWav2Vec.
What it got wrong, and what came next
Check yourself
Q1How does USM differ from Whisper?
Whisper trains end to end on weakly labelled pairs. USM first learns from unlabelled audio with BEST-RQ, then uses a much smaller labelled set, so it covers languages that have audio but few transcripts.
Q2In BEST-RQ, what is frozen and why does it help?
The random projection and codebook that create targets. Not learning the targets avoids the collapse and complexity of jointly trained quantizers like wav2vec 2.0's, making pretraining simpler and more stable at scale.
Q3Why use residual adapters in a 2B model?
They add a few trainable parameters per task or language while the backbone stays frozen, so you train and store only the adapters instead of full copies.
When you're back at a laptop
- USM is not downloadable; write a toy BEST-RQ target generator in PyTorch (random projection, random codebook, nearest-neighbour IDs on log-mel frames).
- Check that IDs are stable for the same audio and differ between silence and speech.
- Compare FLEURS Hindi or Tamil WER for USM (from the paper) with Whisper and MMS.