data2vec / w2v-BERT / BEST-RQ

Why this paper exists

By 2021 there were two working recipes, and both had awkward target machinery:

• wav2vec 2.0 (wav2vec 2.0) learns a Gumbel-softmax quantizer jointly with the model, held together by a contrastive loss with sampled negatives and a diversity loss.
• HuBERT (HuBERT) needs offline k-means and several full pretraining rounds.

These papers ask: what is the simplest target that still works, and does the target even need to be learned? Two of them come from Google, which wanted SSL recipes for its Conformer (Conformer) encoders at very large scale.

The core idea

Masked prediction needs targets that are (a) consistent, so the same sound gets the same target, and (b) informative, so predicting them from context requires modelling speech. The three papers sit at different points:

How it works

data2vec (Meta, 2022)

• One recipe for speech, images and text; only the input encoder changes.
• Student: gets the masked input. Teacher: gets the unmasked input. The teacher has the same architecture, and its weights are an EMA (exponential moving average) of the student's: θ_teacher ← τ θ_teacher + (1 - τ) θ_student, with τ raised toward 1 during training so the teacher changes more slowly as training goes on.
• Target: for each masked time step, take the outputs of the teacher's top K Transformer blocks (K = 8 for speech), normalise each one, and average them. The target is a continuous vector that depends on context, not a discrete ID.
• Loss: regression (smooth L1) between the student's prediction at masked steps and that target.
• Result: matched or beat wav2vec 2.0 and HuBERT on Libri-Light low-resource fine-tuning at the same size. A later data2vec 2.0 made it much faster to train.

w2v-BERT (Google, 2021)

• Conformer-based. The stack is a convolutional feature encoder over log-mel features, then a contrastive module (Conformer blocks), then a masked-prediction module (more Conformer blocks).
• The contrastive module does wav2vec 2.0's job: a quantizer turns the unmasked features into discrete codes, and the module's outputs at masked positions must identify the right code among distractors.
• The masked-prediction module takes the contrastive module's output and predicts the IDs of those same codes at masked positions with cross-entropy, as HuBERT does, but with targets produced inside the same network on the fly.
• Loss: contrastive + masked-prediction (+ the diversity term), trained end to end in one stage, so no k-means rounds.
• Its successor, w2v-BERT 2.0, is the speech encoder of SeamlessM4T (SeamlessM4T). AudioLM (AudioLM) uses k-means of w2v-BERT features as its "semantic tokens", and so does SPEAR-TTS (SPEAR-TTS).

BEST-RQ (Google, 2022)

BERT-based Speech pre-Training with Random-projection Quantizer.

log-mel frames (stacked to encoder rate)
        |                       |
     mask spans            x: unmasked
     (fill with noise)          |
        |                 A x   (A random, frozen)
   Conformer encoder            |
        |                 nearest of C random
   softmax over IDs       codebook vectors (frozen)
        |                       |
   cross-entropy  <------  target ID y_t

• The target for frame t is y_t = argmin_i || norm(c_i) - norm(A x_t) ||, where x_t is the (stacked) log-mel vector, A is a randomly initialised projection matrix, c_i are random codebook vectors, and norm is l2-normalisation. Neither A nor the codebook is ever trained.
• The encoder sees masked log-mels (masked frames filled with small random noise) and predicts y_t at masked frames with cross-entropy.
• Since the quantizer is fixed and separate from the encoder, it works with any encoder architecture, and the encoder can be designed for ASR, including streaming.
• The codebook is a few thousand entries of a low dimension (8192 × 16 in the paper's setup, if I recall correctly; check the paper). The paper reports that using several codebooks improves results.

Build the intuition

Why would a random quantizer work? A random projection roughly preserves distances (the Johnson-Lindenstrauss idea), so similar spectra land on the same nearest codeword and different spectra usually do not. That makes the targets consistent, which is the property HuBERT showed matters most. The targets are not phonetic at all, but predicting them behind a mask still requires the model to infer what the hidden spectrum looked like from context, and to do that it has to model speech. The intelligence is in the encoder, not in the targets.

Analogy: data2vec is a student copying the notes of a slightly older version of itself who saw the whole page. w2v-BERT is one student who first makes flashcards and then quizzes itself on them. BEST-RQ uses flashcards printed by a random machine: meaningless, but always the same for the same picture, which is enough to drill on.

Numbers worth remembering

• data2vec: target is the average of the top 8 teacher blocks (speech); loss is regression, not classification.
• w2v-BERT: reported large Conformer models at ~0.6B and ~1B parameters; better than Conformer-based wav2vec 2.0 and HuBERT on LibriSpeech with Libri-Light pretraining (check the paper's tables for the exact WERs).
• BEST-RQ: the quantizer has zero trainable parameters; it is the pretraining method behind Google USM (Google USM), a ~2B Conformer pretrained on millions of hours of YouTube audio.

Where it shows up today

• BEST-RQ is behind USM (Google USM) and, as I understand it, NVIDIA's NeMo SSL recipe for Fast Conformer (Fast Conformer) uses the same random-projection idea (check the NeMo docs).
• w2v-BERT 2.0 encodes speech in SeamlessM4T (SeamlessM4T), and w2v-BERT semantic tokens drive AudioLM (AudioLM) and SPEAR-TTS (SPEAR-TTS).
• data2vec's EMA-teacher regression idea carries on in later multimodal SSL work.

What it got wrong, and what came next

• data2vec's continuous targets give no discrete units, so it is less useful than HuBERT (HuBERT) as a tokenizer for speech LMs.
• w2v-BERT's two coupled losses keep wav2vec 2.0's codebook-collapse risks.
• BEST-RQ's targets cannot be used as tokens either, because they are not semantic, and the method does not by itself handle noise or speaker robustness the way WavLM (WavLM) does.
• SSL in general still needs labelled fine-tuning. Whisper (Whisper) took the other route: skip SSL and use huge weakly labelled data.

Check yourself

When you're back at a laptop

• Implement BEST-RQ's quantizer in ~10 lines: compute log-mels with torchaudio, stack 4 frames, set A = torch.randn(4*80, 16) and a codebook of torch.randn(8192, 16), normalise both, and take the argmin distance. Check that the same word spoken twice gives overlapping ID sequences.
• Skim data2vec's Figure 1 and w2v-BERT's architecture figure, and redraw each from memory.
• Read the BEST-RQ section of the USM paper (Google USM) to see how it scales.