HiFi-GAN
HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis
In one sentence
HiFi-GAN is a GAN vocoder whose multi-period discriminator (looking at the waveform folded at prime periods) captures pitch periodicity, and together with a multi-scale discriminator and a mel-spectrogram L1 loss it reached WaveNet-level quality while running far faster than real time, even on CPU.
Why this paper exists
Speech is mostly periodic: voiced sounds repeat at the pitch period (2.5-12 ms for typical voices). Earlier GAN vocoders, MelGAN and Parallel WaveGAN (MelGAN + Parallel WaveGAN), were fast but audibly worse than WaveNet (WaveNet) and WaveGlow (Parallel WaveNet + WaveGlow): buzzy, metallic, unstable harmonics. Their discriminators looked at audio at different time scales, but nothing in them was built to notice whether the waveform repeated cleanly at a given period. The two-stage pipelines (Tacotron 2 Tacotron 2, FastSpeech 2 FastSpeech 2) needed a vocoder that was both high quality and cheap.
The core idea
Give the discriminator a view in which periodic structure is obvious. Reshape the 1D waveform into a 2D grid with width p (the period), so samples exactly p apart line up in the same column, then run 2D convs that move only along columns. Do this for several prime periods. Add a mel L1 loss so the generator gets a stable target from the start.
How it works
mel (80 x T frames)
│
▼ Conv1d (512 ch)
[ConvTranspose x8] → MRF ─┐
[ConvTranspose x8] → MRF │ 256x upsampling
[ConvTranspose x2] → MRF │ (hop 256 @ 22.05 kHz)
[ConvTranspose x2] → MRF ─┘ channels halve each step
│
▼ Conv1d → tanh
waveform (1 x 256T)
│
├──► MPD: 5 sub-discriminators, p = 2,3,5,7,11
└──► MSD: 3 sub-discriminators, raw, /2, /4Generator
- A fully convolutional upsampler (no noise input). Each transposed conv upsamples time; the product of the strides equals the hop size (8·8·2·2 = 256 in V1).
- After each upsampling, a Multi-Receptive Field Fusion (MRF) module: several residual blocks in parallel with different kernel sizes (3, 7, 11 in V1) and dilations (1, 3, 5), whose outputs are summed. This lets one layer model patterns at several lengths at once.
- Three configs: V1 (large, best quality), V2 (tiny), V3 (fast, different block design).
Multi-Period Discriminator (MPD)
For period p: pad the waveform of length T to a multiple of p, reshape to (T/p, p), apply 2D convs with kernels shaped (k, 1) so each column is processed separately (weights shared across columns). Periods 2, 3, 5, 7, 11 are primes, so the sub-discriminators overlap as little as possible (a period-4 view would be largely covered by period 2).
Multi-Scale Discriminator (MSD)
From MelGAN: three 1D conv discriminators on raw audio, 2x and 4x average-pooled audio. It catches consecutive, longer-range structure that the MPD (which only looks at strided samples) misses.
Losses
LSGAN (least squares) adversarial losses, for each sub-discriminator D:
L_adv(D) = E[(D(x) - 1)² + (D(G(s)))²]
L_adv(G) = E[(D(G(s)) - 1)²]
L_fm(G) = E[Σ_i (1/N_i) || D_i(x) - D_i(G(s)) ||_1]
L_mel(G) = E[|| φ(x) - φ(G(s)) ||_1]
L_G = L_adv(G) + λ_fm·L_fm + λ_mel·L_mel
(λ_fm = 2, λ_mel = 45)x is real audio, s its mel, G(s) generated audio, D_i the i-th layer's features of a discriminator (with N_i elements), and φ the function that computes a mel spectrogram. The adversarial and feature-matching losses are summed over all 8 sub-discriminators. The large mel weight (45) tells you the mel loss does most of the early work.
Build the intuition
The MPD reshape by hand
Take 12 samples x0 … x11 and period p = 3. Reshape row by row into a 4 x 3 grid:
col0 col1 col2
row0: x0 x1 x2
row1: x3 x4 x5
row2: x6 x7 x8
row3: x9 x10 x11Column 0 holds x0, x3, x6, x9: every third sample. A (k, 1) kernel slides down a column, so it compares samples exactly 3 apart. If the true pitch period is 3 samples, each column should be nearly constant for a clean voiced sound; jitter or buzz shows up as variation down the column, which the discriminator easily spots. With T = 13 you would pad 2 samples (reflect padding) to reach 15, then reshape to 5 x 3.
Real pitch periods are much longer than 11 samples (at 22.05 kHz, a 200 Hz voice has a period of ~110 samples), so the MPD is not literally matching the pitch. It looks at the signal's structure at several strides, which exposes periodic patterns and their harmonics at many scales.
Analogy: Reading a long string of digits for a repeating pattern is hard; writing them in rows of width 3 makes a repeat of 3 jump out as identical columns. The MPD writes the waveform in rows of 2, 3, 5, 7 and 11.
Why the mel loss
A GAN alone starts with a weak discriminator and gives noisy gradients. The mel L1 is a fixed target that immediately teaches the right spectral envelope, and because the mel is on a perceptual (log, mel-scaled) axis, it focuses effort on what people hear. The adversarial losses then add sharpness and realistic phase.
Numbers worth remembering
- V1: ~13.9M parameters, MOS 4.36 vs 4.45 for ground truth on LJSpeech (check Table 1).
- V1: ~168x faster than real time on a V100 GPU at 22.05 kHz; V2 (~0.9M parameters) ~13x faster than real time on CPU (the abstract's figures).
- Upsampling 256x: 80-bin mel at ~86 frames/s → 22,050 samples/s.
- 5 MPD periods (2, 3, 5, 7, 11) + 3 MSD scales = 8 discriminators.
Where it shows up today
- The default vocoder for mel-based pipelines: Tacotron 2 Tacotron 2, FastSpeech 2 FastSpeech 2, SpeechT5, CosyVoice (CosyVoice → CosyVoice 2, a HiFi-GAN-family vocoder) and many AI4Bharat Indian-language systems (Indic TTS (AI4Bharat)).
- VITS (VITS) uses the HiFi-GAN generator as its decoder and its MPD/MSD and feature matching for training; so does Piper.
- Its discriminators and losses are the template for codecs: SoundStream (SoundStream), EnCodec (EnCodec), DAC (DAC) and Mimi (in Moshi, Moshi) all use MPD-style or multi-scale discriminators plus feature matching.
- XTTS (XTTS) decodes with a HiFi-GAN-style decoder.
What it got wrong, and what came next
- Trained per dataset, it generalizes poorly to unseen speakers, noisy conditions, singing and music. BigVGAN (BigVGAN) scaled it with Snake activations and anti-aliasing to make a universal vocoder.
- Time-domain transposed-conv upsampling is expensive and can produce aliasing and tonal artifacts. iSTFTNet and Vocos (iSTFTNet + Vocos) predict an STFT and let the inverse STFT do the upsampling.
- At batch 1 on a GPU it can still be a real share of latency in a voice agent (bookmark #39's SpeechT5 setup).
- Still a separate stage after an acoustic model; VITS (VITS) trains acoustic model and vocoder end to end.
Check yourself
Q1What does the MPD's reshape do, and why use a (k, 1) kernel?
It folds the waveform into rows of width p, so samples p apart sit in the same column. A (k, 1) kernel only slides down columns, so it only ever compares samples spaced exactly p apart, making periodic structure at that stride easy to judge.
Q2Why are the MPD periods prime?
To minimize overlap between sub-discriminators. A period-4 view contains much of what a period-2 view sees (every 4th sample is also every 2nd sample), so non-prime periods add redundant views; primes give more distinct strides for the same compute.
Q3Why keep the MSD if the MPD exists?
The MPD only looks at strided, disjoint subsets of samples, so it never sees consecutive samples together. The MSD looks at the continuous waveform at several smoothing levels, catching smoothness and longer-range envelope problems.
Q4What would happen if you dropped the mel L1 loss?
Training becomes slower and less stable, and quality drops: early on the discriminators give weak, noisy gradients and nothing anchors the spectrum. The paper's ablation shows the mel loss improves MOS noticeably (check the ablation table).
Q5How does the generator reach the sample rate from the mel frame rate?
Transposed convolutions with strides whose product equals the hop size (8, 8, 2, 2 = 256 in V1), each followed by MRF residual blocks. So each mel frame becomes exactly 256 samples.
Q6What does MRF add over a single residual block?
Several residual blocks with different kernel sizes and dilations run in parallel and their outputs are summed, so each stage models patterns at multiple time lengths at once. It is a cheap way to widen the receptive field and its variety.
When you're back at a laptop
- Implement the MPD reshape:
x = F.pad(x, (0, pad), 'reflect'); x = x.view(B, C, T // p, p)and check by printing that column 0 holds samples0, p, 2p, .... - Implement
L_melwithtorchaudio.transforms.MelSpectrogram(80 bins, hop 256, n_fft 1024) and log compression. - Run a pretrained HiFi-GAN (e.g. from the
jik876/hifi-ganrepo or SpeechBrain) on a mel you computed in the fundamentals guide. Time V1 vs V2 on CPU and your RTX 3050 and compute RTF.