WaveGrad + DiffWave
WaveGrad: Estimating Gradients for Waveform Generation / DiffWave: A Versatile Diffusion Model for Audio Synthesis
In one sentence
WaveGrad and DiffWave were the first diffusion vocoders: start from Gaussian noise the length of the waveform and denoise it step by step, conditioned on the mel, trading speed (number of steps) for quality.
Why this paper exists
In 2020 the options were autoregressive vocoders (high quality, slow), flows (large and memory-hungry, Parallel WaveNet + WaveGlow) and GANs (fast but tricky to train and prone to artifacts, MelGAN + Parallel WaveGAN). Denoising diffusion probabilistic models (DDPM, Ho et al. 2020) had just shown that a simple denoising regression loss could produce excellent images. Both papers, released within weeks of each other, asked whether the same works for waveforms.
The core idea
Diffusion: define a forward process that gradually adds Gaussian noise to real audio until it is pure noise. Train a network to predict the noise that was added at any noise level. At inference, start from noise and repeatedly subtract the predicted noise. Each step is a parallel pass over the whole waveform; the number of steps is the cost knob.
How it works
- Forward (noising): for step
twith cumulative noise scheduleᾱ_t(from ~1 at t=0 down to ~0):
x_t = sqrt(ᾱ_t) · x_0 + sqrt(1 - ᾱ_t) · ε, ε ~ N(0, I)
L = E || ε - ε_θ(x_t, mel, t) ||x_0 is the clean waveform, x_t the noisy one, ε the true noise and ε_θ the network's guess. DiffWave uses an L2 loss (squared); WaveGrad found L1 more stable.
- Reverse (sampling): start at
x_T ~ N(0, I)and fort = T … 1compute a slightly cleanerx_{t-1}fromx_tandε_θ, adding a little fresh noise each step (except the last). - WaveGrad: conditions on the continuous noise level (
sqrt(ᾱ)) instead of the integer step, so you can train once and choose a different, shorter noise schedule at inference. The network is a convolutional upsampler from mel to waveform rate with feature-wise modulation from the noisy signal. A searched schedule with 6 steps gets close to the 1000-step quality. - DiffWave: a non-causal WaveNet-style network (residual layers of bidirectional dilated convs), conditioned on the upsampled mel and a step embedding. Uses ~50 training steps and a fast 6-step schedule at inference. Also does unconditional generation (spoken digits without any mel), which GAN and flow vocoders of the time could not do well.
Build the intuition
Analogy: A sculptor starts with a rough block (noise) and removes a bit of marble at each pass, guided by a sketch (the mel). More passes mean finer detail. A GAN is a sculptor who must carve the statue in one stroke.
Why can diffusion work with few steps for a vocoder, but not always for images? The mel already determines most of the signal; the model only has to fill in phase and fine detail. The conditional distribution is narrow, so a handful of big denoising steps lands close to a good waveform.
Numbers worth remembering
- Both: around 6 inference steps reach quality close to their many-step versions (check each paper's tables).
- DiffWave: a few million parameters (~2.6M for the base model, check the paper), MOS comparable to a strong WaveNet vocoder on LJSpeech.
- WaveGrad: trained with ~1000 noise levels; inference schedules of 6 to 1000 steps compared.
Where it shows up today
- As standalone vocoders they lost to HiFi-GAN (HiFi-GAN) and Vocos (iSTFTNet + Vocos) on speed. The value is in the idea: diffusion moved into acoustic models, first Grad-TTS (Grad-TTS), then latent diffusion (NaturalSpeech 2, NaturalSpeech 2 → NaturalSpeech 3) and Seed-TTS (Seed-TTS).
- The "predict the noise / velocity with a regression loss" recipe evolved into flow matching (Flow Matching), used by Matcha-TTS (Matcha-TTS), Voicebox (Voicebox), F5-TTS (F5-TTS) and CosyVoice (CosyVoice → CosyVoice 2).
- StyleTTS 2 (StyleTTS → StyleTTS 2) uses diffusion only for a small style vector, a cheap use of the same idea.
What it got wrong, and what came next
- Many network passes per utterance, each over the full waveform at 22-24 kHz: too slow for a CPU voice agent. Flow matching (Flow Matching) straightens the paths so fewer steps suffice, and consistency or drifting objectives (CALM (Pocket TTS), Drifting Models) aim for one step.
- Few-step outputs can carry residual hiss. GAN vocoders give cleaner single-pass audio, so later systems run diffusion or flow matching on mels or latents and leave the waveform to a GAN or iSTFT vocoder.
Check yourself
Q1What does the network in a diffusion vocoder predict?
The noise ε that was mixed into the clean waveform at a given noise level, given the noisy waveform, the mel and the noise level. Subtracting a scaled version of that prediction moves the sample toward clean audio.
Q2Why does WaveGrad condition on the continuous noise level rather than the step index?
So the trained network works for any noise schedule. You can then choose a short schedule (like 6 steps) at inference without retraining, because the network only cares how noisy its input is, not which step number it is.
Q3What happens if you reduce DiffWave from 50 steps to 1 step?
Quality collapses: one denoising step from pure noise gives roughly the average of plausible waveforms, which sounds noisy and blurred. Diffusion relies on many small steps; reaching one step needs a different training objective (distillation, consistency, or a GAN).
When you're back at a laptop
- Implement the forward noising and the noise-prediction loss on 1-second sine waves with a tiny 1D conv net, conditioned on the frequency; sample with 6 and 50 steps.
- Run a pretrained DiffWave (e.g.
lmnt-com/diffwave) and time it against HiFi-GAN on the same mel on your 3050.