WavLM

Why this paper exists

wav2vec 2.0 (wav2vec 2.0) and HuBERT (HuBERT) were trained on clean, single-speaker audiobooks and tuned for content. On the SUPERB benchmark (a standard suite that tests one frozen SSL model on many tasks: ASR, speaker ID and verification, diarization, emotion, separation), they were strong on content tasks but weaker on speaker-related and multi-talker ones. Real audio such as calls and meetings has noise and overlapping speech.

The core idea

Masked speech denoising and prediction. Take an utterance, mix in either noise or a segment of another utterance, mask spans, and train the model to predict the HuBERT-style pseudo-labels of the original, clean, main speaker at the masked positions. To succeed, the model has to tell the main speaker apart from the interferer (speaker information) and still recover the content behind the mask (content information).

How it works

• Backbone: a CNN feature encoder on the waveform (20 ms frames) and a Transformer, as in HuBERT.
• Utterance mixing: for each training utterance, sometimes overlay a random chunk of another utterance from the batch (or noise) at a random energy ratio. The overlap is kept to less than half the utterance so the main speaker remains clearly defined.
• Targets: k-means cluster IDs computed from clean audio using an earlier trained model, as in HuBERT's later iterations. The loss is cross-entropy on masked frames only.
• Gated relative position bias: instead of only a convolutional positional embedding, each attention layer adds a learned bias that depends on the distance between frames, scaled by a gate computed from the query. This lets the model adapt how much it relies on relative position.
• More and more varied data: Base+ and Large are pretrained on ~94k hours of English (Libri-Light, GigaSpeech, VoxPopuli) instead of audiobooks alone.

Analogy: HuBERT learns to understand speech in a quiet room. WavLM learns at a crowded tea stall: someone talks over the speaker, and it still has to report what the main speaker said, which means it has to know who the main speaker is.

Build the intuition

What would happen without the mixing? The pseudo-labels encode content, and audiobooks have one speaker, so nothing rewards keeping speaker identity. Upper layers discard it. With mixing, the model must separate two voices to predict the right labels, so speaker cues become useful and are kept. This is why WavLM features do well on speaker verification while HuBERT features are mediocre there.

Numbers worth remembering

• Sizes: Base ~95M and Large ~317M parameters, the same shapes as HuBERT.
• Pretraining data for Base+ and Large: ~94k hours (Libri-Light 60k, GigaSpeech 10k, VoxPopuli 24k).
• WavLM Large topped the SUPERB leaderboard at release across most tasks; check the paper's SUPERB table for per-task numbers.

Where it shows up today

• Speaker similarity metric in TTS: papers from VALL-E (VALL-E) onward score voice cloning with cosine similarity of embeddings from a WavLM-based speaker verification model (WavLM-TDNN). When a TTS paper reports "SIM", it is usually this.
• Semantic distillation into codecs: Moshi's Mimi codec (Moshi) distils WavLM features into its first codebook so that tokens carry content.
• Features for voice conversion (kNN-VC) and diarization.

What it got wrong, and what came next

• English-only pretraining. For Indian languages you need multilingual SSL models such as XLS-R, MMS (MMS) or IndicWav2Vec (IndicWav2Vec).
• Still HuBERT's iterative k-means. The cost remains. BEST-RQ (data2vec / w2v-BERT / BEST-RQ) drops learned targets entirely.
• Offline, full-attention encoder. It is not designed for streaming (Cache-aware streaming Conformer).

Check yourself

When you're back at a laptop

• Load microsoft/wavlm-base-plus-sv (WavLM with an x-vector head) in transformers, then embed two clips of you and one of a friend. Compare cosine similarities.
• Add background noise (traffic or a fan recording) to one of your clips and check how much the similarity drops.