Reading Room

TTS · Phase 1: Foundations

Tacotron 2

Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions

Shen, Pang, Weiss, Schuster, Jaitly, Yang, Chen, Zhang, Wang, Skerry-Ryan, Saurous, Agiomyrgiannakis, Wu · Google · 2017

Deep read18 minDay 5 of 10#39 of 86

In one sentence

Tacotron 2 predicts an 80-band mel spectrogram from characters with an attention seq2seq network, then turns it into audio with a WaveNet vocoder conditioned on that mel, reaching naturalness close to recorded speech and fixing the "acoustic model → mel → vocoder" design that TTS used for the next six years.

Why this paper exists

Two 2016-17 results had complementary gaps:

  • WaveNet (WaveNet) produced near-human audio but needed a complicated linguistic front-end (phoneme features, durations, F0 from other models).
  • Tacotron (Tacotron) learned everything from characters but used Griffin-Lim, which sounds metallic, and its CBHG blocks were complicated.

Tacotron 2 joins them: let a seq2seq model do the linguistics and timing, and let a neural vocoder do the waveform, with a mel spectrogram as the interface. The mel is low-dimensional (80 numbers per 12.5 ms), easy to compute from any audio, smooth enough to regress with an L2 loss, and keeps what matters for intelligibility and voice.

The core idea

Split TTS at a well-chosen intermediate representation. Stage 1 solves the hard alignment and prosody problem on a short, smooth sequence (mel frames at 80 per second). Stage 2 solves the "fill in the waveform detail" problem, which needs no linguistic knowledge at all. Each stage can be trained, debugged and swapped independently. That separation is why HiFi-GAN (HiFi-GAN) could later replace WaveNet without touching the acoustic model.

How it works

characters
  -> 512-d embedding
  -> 3 x conv (5 wide, 512 ch, BN, ReLU)
  -> BiLSTM (256 per direction)  = encoder states h_j
                  |
   location-sensitive attention (sees prev alignment)
                  |  context vector
prev mel -> pre-net (2 x FC 256, dropout)
         -> 2 x LSTM (1024) -> linear -> mel frame
                             -> linear -> stop prob
mel frames -> post-net (5 conv, 512 ch) -> + residual
           -> WaveNet vocoder (mixture of logistics)
           -> 24 kHz waveform

Mel targets. STFT with 50 ms frames, 12.5 ms hop, Hann window; 80 mel bands from 125 Hz to 7.6 kHz; log compression (with a floor to avoid log 0). 12.5 ms hop = 80 frames per second.

Encoder. Character embeddings, three conv layers (each spans 5 characters, so the stack sees local spelling context like "ough"), then one bidirectional LSTM. Output: one 512-d vector h_j per character.

Location-sensitive attention. Plain content attention scores each encoder state only by similarity to the decoder state, so nothing stops it from jumping. Location-sensitive attention also looks at where it attended before:

f_i   = F * α_cum_{i-1}   (1-D conv over past weights)
e_i,j = w^T tanh(W s_{i-1} + V h_j + U f_i,j + b)
α_i   = softmax_j(e_i,j)
c_i   = Σ_j α_i,j h_j
  • i is the decoder step, j the character index.
  • s_{i-1} is the decoder LSTM state.
  • α_cum_{i-1} is the sum of attention weights over all previous steps, and F is a learned conv (32 filters, width 31), so f_i,j says "how much have I already used character j and its neighbours".
  • W, V, U, w, b are learned; c_i is the context vector fed to the decoder.

This nudges the attention to move forward steadily.

Decoder. Autoregressive: the previous predicted mel frame goes through a pre-net (two 256-unit ReLU layers with dropout that stays on even at inference, which adds variation and helps generalization), is concatenated with the context, and passes through two 1024-unit LSTMs. A linear layer outputs the next mel frame (one frame per step). A second linear layer plus sigmoid outputs a stop token probability; generation ends when it exceeds 0.5.

Post-net. Five conv layers predict a residual added to the decoder's mel. The decoder is causal, the post-net sees the whole utterance, so it can sharpen frames using future context.

Loss. MSE between target mel and both the pre-post-net and post-post-net predictions, plus a binary cross-entropy on the stop token. Training uses teacher forcing (feed the ground-truth previous frame).

Vocoder. A WaveNet (WaveNet) with 30 dilated conv layers in 3 cycles, conditioned on the mel upsampled to 24 kHz (one mel frame = 300 samples). Instead of a 256-way μ-law softmax it outputs a 10-component mixture of logistics, giving 16-bit-quality samples. It is trained on ground-truth-aligned mels predicted by the stage-1 model, so it learns to tolerate stage 1's slightly blurry outputs.

Build the intuition

Shapes for a 2-second utterance

  • Text: "Printing, in the only sense" ≈ 28 characters → encoder output 28 × 512.
  • Mel: 2 s × 80 frames/s = 160 frames → target 160 × 80. The decoder runs 160 sequential steps.
  • Audio: 2 s × 24,000 = 48,000 samples. WaveNet runs 48,000 sequential steps. That is why stage 2 dominated latency until GAN vocoders.

Why the mel works as an interface

The mel compresses each 300-sample chunk into 80 numbers on a perceptual frequency scale. It throws away phase and fine high-frequency detail; those are exactly the things that are hard to regress and easy for a generative vocoder to invent plausibly. The acoustic model is left with what it is good at: which sound, how loud, what pitch, how long.

When attention fails

Try "the the the the the the" or a 400-character paragraph. Location features help, but the attention can still lock onto one repeated word, jump to the end, or never fire the stop token. In production, people added hacks: forced monotonic attention, attention windowing, splitting text into sentences. All signal that learned soft attention is the wrong tool for an alignment that is known to be monotonic.

Analogy: stage 1 is an architect who draws a floor plan (mel); stage 2 is a builder who lays every brick (samples). The plan is coarse, but any competent builder produces a solid house from it, and you can change builders without redrawing the plan.

Numbers worth remembering

  • Data: ~24.6 hours, single female US English speaker (internal Google data).
  • MOS: Tacotron 2 4.53 vs ground-truth recordings 4.58, on that single-speaker data.
  • Mel: 80 bands, 50 ms window, 12.5 ms hop (80 frames/s), 125 Hz-7.6 kHz.
  • Vocoder at 24 kHz; 300 samples per mel frame.
  • Ablation worth knowing: WaveNet conditioned on Tacotron 2's predicted mels beat one conditioned on linguistic features; check the paper's tables for the vocoder and post-net ablations.

Where it shows up today

What it got wrong, and what came next

Check yourself

Q1What problem does location-sensitive attention solve, and how?

Content-only attention can jump to any character that looks similar to the current decoder state, causing skips and repeats. Location-sensitive attention adds features computed by a conv over the cumulative past attention weights, so the score for each character depends on whether it and its neighbours were already attended. That biases the alignment to move forward smoothly.

Q2Why predict a mel spectrogram instead of a waveform or a linear spectrogram?

The mel is short (80 frames per second), low-dimensional and smooth, so an autoregressive regression model can predict it with an L2 loss. It keeps the information that matters perceptually and drops phase and fine detail, which a generative vocoder can fill in. It is also easy to compute from any audio, so the vocoder can be trained separately on lots of data.

Q3What is the stop token and what happens if it fails?

A per-frame sigmoid output that predicts whether the utterance has ended; decoding stops when it exceeds 0.5. If it never fires, the decoder keeps generating silence or babble until a maximum length; if it fires early, the sentence is cut off.

Q4Why is the WaveNet vocoder trained on mels predicted by the acoustic model instead of ground-truth mels?

At inference the vocoder only ever sees predicted mels, which are smoother and slightly wrong. Training on ground-truth-aligned predictions (teacher-forced stage-1 outputs aligned to the real audio) makes the vocoder robust to that mismatch and improves quality.

Q5Why does the pre-net keep dropout on at inference?

The decoder is fed its own previous output, and with deterministic inputs it tends to copy them and get stuck. Dropout at inference acts as a bottleneck and adds noise, which the authors found necessary to learn good attention and generalize; it also introduces some variation between runs.

Q6What would happen if you swapped the WaveNet vocoder for HiFi-GAN without retraining stage 1?

It works as long as both use the same mel configuration (bands, hop, frequency range, log scaling). That is the benefit of the two-stage split: synthesis becomes much faster with similar quality, and only a vocoder fine-tune on predicted mels might be needed to remove small artefacts.

When you're back at a laptop

  • Run NVIDIA's pretrained Tacotron 2 + HiFi-GAN (or WaveGlow) from torch.hub on CPU; measure seconds of compute per second of audio for each stage.
  • Plot the attention matrix for a short sentence and for a 60-word paragraph; find where the diagonal breaks.
  • Feed the vocoder a ground-truth mel computed with the same settings (the NVIDIA model uses 22.05 kHz audio with its own hop and window; read them from its config) and compare with the predicted mel; this isolates acoustic-model error from vocoder error.
arXiv ↗PDF ↗