MelGAN + Parallel WaveGAN
MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis / Parallel WaveGAN: A Fast Waveform Generation Model Based on GANs with Multi-Resolution Spectrogram
In one sentence
MelGAN and Parallel WaveGAN were the first practical GAN vocoders: a convolutional generator turns a mel spectrogram into a waveform in one parallel pass, trained against discriminators, with MelGAN contributing multi-scale discriminators + feature matching and Parallel WaveGAN the multi-resolution STFT loss that made GAN audio training stable.
Why this paper exists
By 2019 there were two ways to make fast vocoders, and both hurt. Parallel WaveNet (Parallel WaveNet + WaveGlow) needed a trained WaveNet teacher and four losses. WaveGlow (Parallel WaveNet + WaveGlow) was simple but ~88M parameters and heavy. Earlier attempts at GANs for raw audio produced buzzy, artifact-filled output. People wanted a small, parallel model trained without a teacher.
The core idea
A GAN (generative adversarial network) trains a generator against a discriminator that tries to tell real audio from generated audio. You do not need likelihoods or invertibility, so the generator can be a plain, small upsampling convnet. The trick is making training stable:
- MelGAN: several discriminators at different time resolutions, plus a feature-matching loss (match the discriminator's internal activations on real vs fake audio).
- Parallel WaveGAN: add a non-adversarial multi-resolution STFT loss, so the generator gets a dense, stable signal about the spectrum from step one, and the GAN only has to fix the fine detail.
How it works
MelGAN
- Generator: mel (80 bins, hop 256) → conv → upsampling blocks (transposed convs, 8x, 8x, 2x, 2x = 256x) each followed by residual stacks of dilated convs → tanh → waveform. No noise input; the mel alone drives it.
- Discriminators: 3 identical conv discriminators on raw audio, 2x and 4x average-pooled audio. Each outputs a sequence of real/fake scores over windows (patch-style), not one score.
- Loss: hinge GAN loss + feature-matching loss (L1 between discriminator feature maps of real and generated audio, weight ~10). Weight normalization everywhere.
Parallel WaveGAN
- Generator: a non-causal WaveNet (dilated convs, no autoregression) that takes Gaussian noise at the sample rate plus the upsampled mel and outputs the waveform in one pass.
- Discriminator: a simple stack of 1D dilated convs.
- Multi-resolution STFT loss, averaged over M STFT settings (e.g. FFT sizes 512, 1024, 2048 with matching hops and windows):
L_sc = || |S(x)| - |S(x̂)| ||_F / || |S(x)| ||_F
L_mag = (1/N) || log|S(x)| - log|S(x̂)| ||_1
L_G = (1/M) Σ_m (L_sc + L_mag) + λ_adv · L_advS(·) is the STFT, x the real and x̂ the generated waveform, ||·||_F the Frobenius norm, N the number of elements. L_sc (spectral convergence) emphasizes large spectral components, L_mag (log magnitude) the quieter ones. Several resolutions prevent the generator from fitting one time-frequency trade-off and producing artifacts at others.
Build the intuition
Analogy: A pure GAN is like learning to paint only from a critic who says "fake" or "real". The STFT loss is a teacher who also shows you the reference photo. You get the shapes right fast, and the critic only polishes texture.
Why several STFT resolutions? A short window (512) has sharp timing but blurry frequency; a long one (2048) has sharp frequency but blurry timing. A generator matched at only one setting can hide errors in the other domain (for example smeared transients). Matching all three closes those loopholes.
Numbers worth remembering
- Both upsample mel to waveform by the hop size, typically 256x at 22.05 kHz or 300x at 24 kHz.
- MelGAN: ~4M parameters, faster than real time on CPU (check the paper's speed table).
- Parallel WaveGAN: ~1.4M parameters, MOS close to a WaveNet vocoder in a Japanese TTS test (check Table 2).
- Parallel WaveGAN: three STFT resolutions in the auxiliary loss.
Where it shows up today
- The multi-resolution STFT loss (or its cousin, a multi-scale mel loss) is in almost every later vocoder and codec: HiFi-GAN (HiFi-GAN) uses a mel L1, and SoundStream (SoundStream), EnCodec (EnCodec) and DAC (DAC) use multi-scale spectral losses.
- MelGAN's multi-scale discriminator became HiFi-GAN's MSD (HiFi-GAN); feature matching is in VITS (VITS) and every codec.
- The
kan-bayashi/ParallelWaveGANrepo became the standard toolkit for training these vocoders in ESPnet.
What it got wrong, and what came next
- Quality was still a step below WaveNet, with noticeable buzz or metallic artifacts on some voices. HiFi-GAN (HiFi-GAN) added the multi-period discriminator to model pitch periodicity and closed the gap.
- Neither generalized well to unseen speakers, singing or music. BigVGAN (BigVGAN) addressed that.
- Time-domain upsampling is still most of the compute. iSTFTNet and Vocos (iSTFTNet + Vocos) replace it with an inverse STFT.
Check yourself
Q1Why does adding an STFT loss stabilize GAN vocoder training?
The adversarial loss alone gives a noisy, moving target, especially early when the discriminator is weak. The STFT loss is a fixed, dense regression target that immediately pushes the generator toward the right spectrum, so the GAN only needs to add realistic fine structure.
Q2Why do MelGAN's discriminators run on downsampled audio?
Each scale sees a different frequency range and a longer context per receptive field. The raw-audio discriminator focuses on high-frequency detail, the 4x-pooled one on lower-frequency, longer-range structure. Together they cover more of the signal than one discriminator could.
Q3What would happen if you trained Parallel WaveGAN with only the STFT loss and no discriminator?
You would get intelligible but slightly muffled or phasey audio: magnitude losses ignore phase and reward the average of plausible outputs, which loses crisp detail. The adversarial loss is what pushes toward a sharp, realistic sample rather than an average.
Q4What does feature matching compare?
The intermediate activations of the discriminator on real audio and on generated audio for the same mel, with an L1 loss summed over layers. It gives the generator a richer target than the single real/fake score.
When you're back at a laptop
- Implement a multi-resolution STFT loss in ~20 lines with
torch.stftat (512, 1024, 2048); check it is zero for identical signals and grows as you add noise. pip install parallel_wavegan, download a pretrained LJSpeech model, vocode a mel and measure RTF on CPU and on your RTX 3050.