wav2vec 2.0

Why this paper exists

Supervised ASR needs thousands of hours of transcribed audio. English has that (LibriSpeech has 960 hours); Tamil, Odia and Bhojpuri mostly do not, and transcription costs money per hour. Untranscribed audio (radio, podcasts, call recordings) is nearly free.

Text NLP had solved the analogous problem with BERT: predict masked words on unlabelled text, then fine-tune on a small labelled set. Speech has no words to mask. It is a continuous signal with no given vocabulary, so "predict the masked token" has no obvious target.

The predecessor vq-wav2vec (2019) first quantized audio into discrete units and then trained a separate BERT over them. That meant two stages trained separately, and the quantization threw away information before the Transformer saw it. wav2vec 2.0 does everything in one network trained end to end.

The core idea

Learn the vocabulary and the BERT at the same time. A quantizer turns each 20 ms frame into one of a finite set of learned codewords: these are the "tokens" the model must predict. The Transformer sees the audio with some spans masked out (its input stays continuous, never quantized) and, at each masked position, has to say which codeword was really there. Instead of a softmax over all codewords, it is scored contrastively: its output must be more similar to the true quantized frame than to around 100 quantized frames taken from elsewhere in the same utterance.

Because the distractors come from the same speaker and recording, the model cannot win by encoding voice or microphone. It has to encode what was said, which is exactly what ASR needs.

How it works

raw waveform, 16 kHz
        |
  CNN feature encoder (7 conv layers)
        |   z: 50 frames/s x 512 dims
        +------------------+
        |                  |
  mask ~half the     quantizer (Gumbel-
  frames (spans)     softmax, 2 x 320)
        |                  |
  Transformer              |  q: target per frame
  context network          |
        |  c: one vector    |
        |  per frame        |
        +---> contrastive <-+
              loss on masked frames

1. Feature encoder

Seven 1-D convolution layers, 512 channels each, applied to the raw waveform. The strides multiply to 320 samples, so at 16 kHz you get one output vector every 20 ms (50 per second), each seeing about 25 ms of audio. Ten seconds of audio becomes a 500 × 512 matrix z. No mel spectrogram is computed anywhere.

2. Masking

Choose about 6.5% of frames at random as span starts and mask the 10 frames following each one (spans may overlap). Roughly half of all frames end up masked, in spans averaging around 300 ms. A masked frame's z is replaced by one learned "mask" vector shared across positions. Spans matter: one masked 20 ms frame could be guessed by interpolating its neighbours, while a 300 ms gap forces the model to use context (phonotactics, words).

3. Context network

A Transformer (BASE: 12 blocks, 768 dims; LARGE: 24 blocks, 1024 dims) over the masked sequence. Instead of absolute position embeddings it adds the output of a wide convolution over the input as a relative positional signal. Output: a context vector c_t per frame.

4. Quantizer: where the targets come from

The quantizer reads the unmasked encoder output z_t, so a target exists even for frames the Transformer could not see.

• There are G = 2 codebooks, each with V = 320 learned entries.
• A linear layer maps z_t to G × V = 640 logits, 320 per codebook.
• In each codebook one entry is chosen. The two chosen entries are concatenated and linearly projected to give q_t. In principle that is 320 × 320 = 102,400 possible units.

Choosing an entry is a hard argmax, which has no gradient. The fix is Gumbel-softmax with a straight-through estimator:

p_{g,v} = exp((l_{g,v} + n_v) / τ)
          / Σ_k exp((l_{g,k} + n_k) / τ)
n = -log(-log(u)),  u ~ Uniform(0, 1)

• l_{g,v}: logit for entry v of codebook g.
• n_v: Gumbel noise, so sampling explores different entries.
• τ: temperature, annealed from 2 down to about 0.5 during training, which moves the soft choice toward a hard one.

Forward pass: take the argmax (a true one-hot pick). Backward pass: use the gradient of the soft probabilities p as if they had been used. The codebook therefore learns, even though each frame is mapped to exactly one entry.

5. Contrastive loss

For each masked time step t:

L_m = -log  exp(sim(c_t, q_t) / κ)
            / Σ_{q' in Q_t} exp(sim(c_t, q') / κ)

• c_t: the Transformer's output at the masked position (after a linear projection to the size of q).
• q_t: the true quantized target at t.
• Q_t: the candidate set, q_t plus K = 100 distractors, the quantized vectors of other masked frames of the same utterance, sampled uniformly.
• sim(a, b): cosine similarity, aᵀb / (|a| |b|).
• κ: temperature, 0.1.

This is a 101-way classification where the "classes" are built on the fly from the utterance itself. It is the InfoNCE loss from CPC, with quantized targets.

6. Diversity loss

The contrastive loss has a cheap way out: send every frame to the same few codewords. Then targets look alike and the task gets easy. To block this, a diversity term pushes up the entropy of the average codeword usage:

L_d = (1 / (G V)) Σ_g Σ_v  p̄_{g,v} log p̄_{g,v}
L   = L_m + α L_d,   α = 0.1

p̄_{g,v} is the softmax probability of entry v in codebook g, averaged over all frames in a batch (without Gumbel noise). Σ p̄ log p̄ is negative entropy, so minimising it spreads usage over all 320 entries of each codebook.

7. Fine-tuning

Discard the quantizer. Add a linear layer mapping each c_t to ~30 characters plus the CTC blank, and train with CTC (CTC) on the labelled data, with the CNN frozen and SpecAugment-style masking (SpecAugment). Decode greedily, or with beam search and an n-gram or Transformer LM.

Build the intuition

Analogy: a listening test with a twist. You hear a sentence with a few hundred milliseconds bleeped out and you are shown 101 short clips, all cut from the same speaker's recording. Exactly one of them is what was under the bleep. Voice, room and microphone are the same in every clip, so the only way to choose is to know which sound fits the words around the gap.

A tiny contrastive example: suppose K = 2, so three candidates. Cosine similarities between c_t and the candidates are 0.8 (true), 0.5 and 0.3.

• With κ = 0.1, the logits are 8, 5 and 3. The softmax probability of the true one is e^8 / (e^8 + e^5 + e^3) ≈ 2981 / 3149 ≈ 0.95, so the loss is -log 0.95 ≈ 0.05.
• With κ = 1, the logits are 0.8, 0.5 and 0.3, the probability is about 0.43 and the loss is about 0.86.

A small temperature makes small differences in cosine similarity decisive. It sharpens the competition, so the model has to separate the true unit clearly from near-misses.

What if you quantized the Transformer's input too? That was vq-wav2vec's design, and this paper's ablation shows it is worse, because the Transformer loses acoustic detail it could use. The best setting is continuous inputs and quantized targets: quantizing only the target removes detail (speaker, noise) the model should not spend capacity predicting.

Numbers worth remembering

• Frame rate: one vector every 20 ms (50 Hz) from raw 16 kHz audio.
• Model sizes: BASE ~95M parameters, LARGE ~317M.
• Pretraining data: LibriSpeech 960 h, or LibriVox ~53k h (the Libri-Light audio), with no transcripts.
• Quantizer: 2 codebooks × 320 entries; 100 distractors; κ = 0.1; about 49% of frames masked.
• Headline: LARGE pretrained on ~53k h and fine-tuned on 10 minutes of labels got WER 4.8 / 8.2 on LibriSpeech test-clean / test-other (with an LM). Using all 960 h of labels it got 1.8 / 3.3.

Where it shows up today

• Multilingual pretraining: XLSR / XLS-R pretrain the same model on many languages, and MMS (MMS) scales it to 1,400+ languages for ASR in 1,100+.
• Indian languages: IndicWav2Vec (IndicWav2Vec) is wav2vec 2.0 pretrained on 40 Indian languages and fine-tuned with CTC. It is the usual starting point when you have only tens of hours of a language.
• Descendants of the objective: HuBERT (HuBERT) and WavLM (WavLM) replace the contrastive target with cluster IDs; w2v-BERT and BEST-RQ (data2vec / w2v-BERT / BEST-RQ) combine or simplify it.
• As a feature extractor: wav2vec 2.0 features feed speech-quality predictors such as UTMOS (UTMOS) and the phoneme forced aligner in WhisperX (WhisperX).
• On your laptop: BASE with a CTC head runs comfortably on CPU or a 4 GB GPU, and has no decoder loop, unlike Whisper.

What it got wrong, and what came next

• The quantizer is fiddly. Gumbel temperature, diversity weight and codebook collapse all need tuning. HuBERT (HuBERT) swaps learned codebooks for offline k-means targets; BEST-RQ (data2vec / w2v-BERT / BEST-RQ) uses a frozen random quantizer.
• The contrastive task depends on how negatives are sampled. HuBERT's plain cross-entropy over cluster IDs is simpler and matches or beats it.
• It learns content but tends to discard speaker and noise information. That suits ASR, but on tasks like speaker verification and separation it is weaker than WavLM (WavLM), which adds denoising.
• Offline. Full attention needs the whole utterance; streaming needs other designs (Cache-aware streaming Conformer).
• CTC has no internal LM, so best WER needs an external n-gram LM. Whisper (Whisper) learns one implicitly in its decoder.

Check yourself

When you're back at a laptop

• pip install transformers torchaudio, then load facebook/wav2vec2-base-960h with Wav2Vec2ForCTC and Wav2Vec2Processor.
• Record 5 s of yourself at 16 kHz and get logits of shape (1, T, 32). Check that T ≈ 50 × seconds.
• Decode with your own greedy CTC decoder from CTC: argmax per frame, merge repeats, drop the blank (the pad token), and map | to a space. Compare with processor.batch_decode.
• Try Indian-accented English and a Hindi sentence; note where it fails, and why a 960 h English fine-tune would.

ids = logits.argmax(-1)[0].tolist()
out, prev = [], None
for i in ids:
    if i != prev and i != blank_id:
        out.append(vocab[i])
    prev = i
print("".join(out).replace("|", " "))