Tacotron
Tacotron: Towards End-to-End Speech Synthesis
In one sentence
Tacotron is a single attention-based sequence-to-sequence network that reads characters and writes a spectrogram, replacing the whole hand-built linguistic pipeline of earlier TTS, with the classic Griffin-Lim algorithm turning the spectrogram into a waveform.
Why this paper exists
Statistical parametric TTS in 2016 was a chain of separately built parts: a text analysis front-end (normalization, part-of-speech tags, phonemes, stress, syllable features), a duration model, an acoustic model, and a signal-processing vocoder. Each part needed expert design and its own training data, and errors added up. WaveNet (WaveNet) improved the last step but still needed all the linguistic features in front of it. The question: can one network learn the mapping from (text, audio) pairs alone?
The core idea
Use the same encoder-attention-decoder recipe that worked for machine translation and for Listen, Attend and Spell in ASR (Listen, Attend and Spell), but in the reverse direction: characters in, spectrogram frames out. Attention learns which characters to "read" while producing each frame, so the network learns pronunciation and timing (alignment) itself.
How it works
characters -> embedding -> pre-net -> CBHG -> encoder states
|
attention <--+
|
prev frame -> pre-net -> attention GRU -> decoder GRUs
|
r mel frames per step (r = 2)
|
post-net CBHG -> linear spectrogram
|
Griffin-Lim -> waveform- Encoder: character embeddings, a small "pre-net" (two fully connected layers with dropout), then a CBHG module: a bank of 1-D convolutions of widths 1 to K (like n-gram detectors), max pooling, highway layers, and a bidirectional GRU.
- Attention decoder: a GRU attention RNN with content-based (Bahdanau-style) attention over encoder states, then GRU decoder layers with residual connections. It predicts 80-band mel frames.
- Reduction factor r: each decoder step outputs
rframes at once (they user = 2). This shortens the decoder sequence, speeds training and, they found, made the attention learn faster. - Post-net: another CBHG that sees the whole predicted mel sequence and converts it to a linear-frequency magnitude spectrogram.
- Griffin-Lim: an iterative algorithm that guesses phase, runs inverse STFT and STFT, keeps the new phase and resets the magnitude, repeating until consistent. The spectrogram only gives magnitudes; Griffin-Lim invents a phase.
- Loss: L1 on both the mel and the linear spectrogram.
Build the intuition
Think about what attention must learn for "read". At each output frame the decoder asks "which characters am I saying now?" Early frames should attend to "r", then move to "ea", then "d", and never go back. When it works, the attention matrix (characters × frames) is a clean diagonal staircase. When it fails, the attention jumps ahead (skipped words), gets stuck (repeated sounds), or wanders (babbling). Nothing in the model forces the staircase; it is just what the gradient tends to find on good data. That fragility drives the next three years of TTS research.
Why does Griffin-Lim sound metallic? Phase carries the fine timing of each harmonic. Griffin-Lim's phase is only self-consistent, not natural, so harmonics smear and you hear a phasey, robotic buzz. It is a "this works at all" vocoder.
Analogy: Tacotron is a sight-reader with a finger on the score. The finger is the attention. Most of the time it slides smoothly left to right; on a long or unusual line it can slip, and the singer repeats a bar or skips a line.
Numbers worth remembering
- About 24.6 hours of single-speaker US English (internal Google data).
- 5-scale MOS: Tacotron ~3.82, ahead of a production parametric system (~3.69) and behind a production concatenative one (~4.09).
- Reduction factor
r = 2; 80-band mel inside the decoder, linear spectrogram after the post-net; Griffin-Lim with about 50 iterations.
Where it shows up today
- Tacotron 2 (Tacotron 2) is the direct successor: same idea, simpler blocks, mel output and a WaveNet vocoder.
- The reference-encoder and style-token work (Global Style Tokens) and SV2TTS (SV2TTS) are all built on Tacotron-family models.
- Griffin-Lim survives as a cheap baseline and debugging tool: if Griffin-Lim on your predicted mel already sounds intelligible, your acoustic model is fine and any remaining problem is the vocoder.
What it got wrong, and what came next
- Griffin-Lim quality: metallic artefacts. Tacotron 2 (Tacotron 2) swapped in a neural vocoder conditioned on mel; later vocoders (HiFi-GAN, iSTFTNet + Vocos) made it fast.
- Fragile attention: skipping, repeating, failure on long sentences. Location-sensitive attention in Tacotron 2 helps; explicit durations in FastSpeech (FastSpeech) and Monotonic Alignment Search in Glow-TTS (Glow-TTS) remove the problem.
- Slow autoregressive decoding and no control of speed or style: fixed by non-autoregressive models (FastSpeech, FastSpeech 2) and style tokens (Global Style Tokens).
- Characters only: pronunciation of unusual words is guessed; most later systems feed phonemes from a G2P (grapheme-to-phoneme) tool.
Check yourself
Q1What does the reduction factor r do and why does it help?
The decoder emits r consecutive frames per step instead of one, so the decoder runs r times fewer steps. Neighbouring frames are highly correlated, so little is lost, and the shorter sequence makes training faster and helps attention converge.
Q2Why does Tacotron need Griffin-Lim, and why does it sound metallic?
The network predicts only spectrogram magnitudes, and a waveform also needs phase. Griffin-Lim estimates a phase that is consistent with the magnitudes by alternating STFT and inverse STFT, but that phase is not the natural one, so harmonics smear and the result sounds buzzy or metallic.
Q3What would happen at inference if the attention never learned a monotonic alignment?
The decoder would not know which characters it is speaking, so it would skip words, repeat syllables, or produce babble, and it may never emit an ending. Nothing in the architecture forces monotonic alignment; it must be learned from data.
Q4What does "end-to-end" replace compared with parametric TTS?
It replaces the hand-designed linguistic feature extraction, the separate duration model and the acoustic model with one network trained on text-audio pairs. Only text normalization and the waveform reconstruction (Griffin-Lim) remain outside it.
When you're back at a laptop
- Compute a mel and a linear spectrogram of an LJSpeech clip with
librosa, then reconstruct withlibrosa.griffinlimat 5, 30 and 100 iterations; listen to the metallic quality. - Plot an attention matrix from any open Tacotron 2 checkpoint on a short and a very long sentence and look for the diagonal breaking down.