Reading Room

TTS · Phase 3: Vocoders

Parallel WaveNet + WaveGlow

Parallel WaveNet: Fast High-Fidelity Speech Synthesis / WaveGlow: A Flow-based Generative Network for Speech Synthesis

van den Oord et al. (DeepMind); Prenger, Valle, Catanzaro (NVIDIA) · DeepMind / NVIDIA · 2017

Skim11 minDay 6 of 10#47 of 86

In one sentence

Both papers make WaveNet-quality audio in one parallel pass by using a normalizing flow (an invertible network that turns Gaussian noise into audio): Parallel WaveNet gets there by distilling an autoregressive WaveNet teacher into a flow student, WaveGlow by training a single flow directly with maximum likelihood.

Why this paper exists

WaveNet (WaveNet) generates one sample at a time: 16,000-24,000 sequential network calls per second of audio. Training is parallel (teacher forcing: all past samples are known), but generation is not. That made it far too slow for Google Assistant or any real-time product. The question was how to keep WaveNet's quality while producing all samples at once.

The core idea

A flow maps noise z (one Gaussian number per audio sample) to audio x through invertible layers. If every layer is invertible with a cheap Jacobian determinant, you can both compute the exact likelihood of real audio (for training) and turn noise into audio in one parallel pass (for inference).

  • Parallel WaveNet uses an inverse autoregressive flow (IAF): sampling is parallel, but computing the likelihood of a given waveform is sequential, so you cannot train it by maximum likelihood efficiently. Instead it is trained by probability density distillation: the student generates audio, and the pretrained WaveNet teacher (which can score audio in parallel) tells it how likely that audio is.
  • WaveGlow uses affine coupling layers (from Glow): both directions are parallel, so it trains with plain maximum likelihood. No teacher, one loss.

How it works

Parallel WaveNet

  • Teacher: a normal autoregressive WaveNet (with a mixture-of-logistics output instead of a 256-way softmax).
  • Student: a stack of IAF blocks, each a WaveNet-like network. Each block shifts and scales the noise: x_t = z_t · s(z_<t) + μ(z_<t). Because z is known in full, all t are computed at once.
  • Main loss: KL(P_student || P_teacher) = cross-entropy of student samples under the teacher minus the student's entropy. Minimizing it makes the student produce audio the teacher finds likely, without collapsing to one sound.
  • Extra losses were needed: a power loss (match the energy in each frequency band, otherwise the output whispers), a perceptual loss from a phone classifier, and a contrastive loss (be likely under the teacher conditioned on the right text, not the wrong text).

WaveGlow

  • Squeeze groups of 8 samples into channels, then apply 12 flow steps. Each step: an invertible 1x1 convolution (mixes channels) and an affine coupling layer: split channels in half, a WaveNet-like network looks at half A plus the mel and outputs scale s and shift b for half B: x_B' = s ⊙ x_B + b. Half A passes through unchanged, so inversion is trivial.
  • Loss (negative log-likelihood):
-log p(x) = z·z / (2σ²)
          - Σ_j log s_j
          - Σ_k log|det W_k| + const

z is the noise the flow maps x to, σ the Gaussian's std, s_j the coupling scales, W_k the 1x1 conv matrices. The last two terms are the log-determinants.

  • Inference: sample z ~ N(0, σ²) (a lower σ at inference sounds cleaner), run the flow in reverse, conditioned on the mel.

Build the intuition

Analogy: WaveNet is a writer who must write each letter after reading everything before it. Parallel WaveNet hires a fast typist who writes the whole page at once and has the slow writer grade it. WaveGlow trains the fast typist directly from good pages, no grader needed, but the typist has to be bigger to learn alone.

Why is IAF fast to sample but slow to train? In an IAF, x_t depends on z_<t. Given z, you know everything up front, so generation is parallel. But training by likelihood starts from a real x and needs z: recovering z_t requires z_<t, which is sequential again. Coupling layers avoid this asymmetry by only conditioning on the half that passes through unchanged.

Numbers worth remembering

  • Parallel WaveNet: over ~20x faster than real time on GPU (check the paper), deployed in Google Assistant at 24 kHz.
  • WaveGlow: ~88M parameters (check the paper), 12 flow steps, trained on LJSpeech (~24 hours, one speaker).
  • WaveGlow: over ~500 kHz generation on a V100 (that is, much faster than real time at 22 kHz), with MOS close to WaveNet.

Where it shows up today

  • Normalizing flows came back in acoustic models: Glow-TTS (Glow-TTS) uses the same 1x1 conv + coupling design, and VITS (VITS) puts a flow inside its prior.
  • WaveGlow was NVIDIA's default vocoder for Tacotron 2 (Tacotron 2) in early NeMo releases.
  • Distillation of a slow sampler into a fast student returns in one-step diffusion and consistency models (CALM, CALM (Pocket TTS)).

What it got wrong, and what came next

  • Parallel WaveNet's training is complex (a teacher plus four losses) and fragile. GAN vocoders (MelGAN + Parallel WaveGAN, HiFi-GAN) got similar quality with simpler training.
  • WaveGlow is big and memory-hungry because flows must keep every layer invertible (no downsampling, same dimensionality everywhere). HiFi-GAN (HiFi-GAN) matched or beat it with a fraction of the parameters.
  • Neither runs well on a CPU. LPCNet (LPCNet) took the opposite approach: DSP plus a tiny network.

Check yourself

Q1Why can't you train an inverse autoregressive flow efficiently with maximum likelihood?

Likelihood needs you to map the real audio x back to noise z. In an IAF each z_t depends on earlier z values, so recovering z is sequential, one sample at a time, the same cost as WaveNet sampling. Sampling is the cheap direction, which is why Parallel WaveNet trains by having the student sample and the teacher score.

Q2What does an affine coupling layer do and why is it easy to invert?

It splits the channels into two halves, leaves one half unchanged, and uses a network on that half (plus the mel) to compute a scale and shift for the other half. To invert, you recompute the same scale and shift from the unchanged half and undo them. The Jacobian is triangular, so its log-determinant is just the sum of log scales.

Q3What would happen to Parallel WaveNet without the power loss?

The student finds a shortcut: audio that is likely under the teacher but too quiet or whispery, since low-energy outputs can still score well per sample. The power loss forces the energy in each frequency band to match real speech, which fixes this.

Q4Why is WaveGlow so large compared with HiFi-GAN?

A flow must be invertible, so it cannot compress or discard information; every layer keeps the full dimensionality and must model the whole distribution exactly. A GAN generator has no such constraint and only needs to produce convincing samples.

When you're back at a laptop

  • Load NVIDIA's pretrained WaveGlow from torch.hub (nvidia_waveglow) and vocode a mel. Try sigma 0.6, 0.8, 1.0 and listen to the noise floor change.
  • Write one affine coupling layer in PyTorch, check inverse(forward(x)) == x and that the log-determinant equals log(s).sum().
  • Count WaveGlow's parameters and compare with HiFi-GAN V1 and V2.
Parallel WaveNet arXiv ↗WaveGlow arXiv ↗