Transformer TTS
Neural Speech Synthesis with Transformer Network
In one sentence
Transformer TTS keeps Tacotron 2's overall design but replaces its recurrent encoder, decoder and attention with Transformer multi-head self-attention, which makes training much faster and matches Tacotron 2's quality, while inference stays autoregressive and alignment stays fragile.
Why this paper exists
Tacotron 2 (Tacotron 2) trains slowly because LSTMs process one step at a time even when the targets are known. They also struggle with long-range dependencies: prosody in a long sentence depends on words far away. The Transformer (2017) had just shown, in machine translation, that self-attention trains in parallel over all positions and connects any two positions in one step. The obvious experiment was to bring it to TTS.
The core idea
Swap every recurrent part of Tacotron 2 for Transformer blocks: a self-attention encoder over phonemes, a masked self-attention decoder over mel frames, and multi-head encoder-decoder attention for alignment. During training with teacher forcing, every mel frame's prediction is computed in parallel, as in WaveNet's training (WaveNet).
How it works
phonemes -> embedding -> conv pre-net -> + α·PE
-> N x (self-attn + FFN) encoder
|
mel (shifted) -> 2-layer FC pre-net -> + α·PE
-> N x (masked self-attn + enc-dec attn + FFN)
-> linear -> mel frame -> post-net (5 conv)
-> linear -> stop token- Input: phonemes (from a G2P tool) instead of characters, which removes the pronunciation-guessing burden.
- Encoder pre-net: a 3-layer conv network (as in Tacotron 2's encoder) gives each phoneme local context before self-attention.
- Decoder pre-net: two fully connected layers project the 80-d mel frame into the model's hidden space.
- Scaled positional encoding: sinusoidal position vectors are multiplied by a trainable scalar α before being added, because text and mel embeddings live in different spaces and need different position strengths.
- Masked decoder self-attention: frame
tcan only see frames< t, so the model is still autoregressive. - Multi-head encoder-decoder attention replaces location-sensitive attention. Different heads learn different alignment patterns; some heads come out clearly diagonal.
- Outputs, post-net, loss: as in Tacotron 2 (mel frame, stop token, conv post-net residual, MSE on mel plus a stop loss). They up-weight positive stop tokens because only one frame per utterance is a "stop".
- Vocoder: WaveNet.
Build the intuition
Training versus inference cost is the key asymmetry.
- Training (teacher forcing): the whole ground-truth mel is known. All 800 frames of a 10-second utterance go through the decoder in one batched pass. An LSTM would need 800 sequential steps.
- Inference: frame 1 must be generated before frame 2 can attend to it. You still run 800 decoder passes, and each one attends over all previous frames, so the total cost of a naive implementation grows quadratically with length (key/value caching helps).
So the Transformer fixes training time but not synthesis time. And because alignment is still learned soft attention, all of Tacotron 2's failure modes (skips, repeats, missing stop) remain, sometimes worse because no location features push the attention forward.
Analogy: it is like upgrading a typist's practice sessions so they can grade a whole page at once, without making them type any faster during the exam.
Numbers worth remembering
- Data: ~25 hours of professional US English single-speaker recordings (internal).
- Training was roughly 4x faster per step than Tacotron 2 (the paper reports ~4.25x).
- MOS about equal to Tacotron 2 (~4.39 vs ~4.38) and close to recordings (~4.44); treat these as "parity", and check the paper's table for the exact values.
Where it shows up today
- FastSpeech (FastSpeech) used an autoregressive Transformer TTS as its teacher: its encoder-decoder attention gave the phoneme durations, and its outputs became training targets.
- Transformer blocks became the default text encoder and decoder in most later TTS: FastSpeech 2 (FastSpeech 2), Glow-TTS's encoder (Glow-TTS), VITS (VITS), and the decoder-only LMs of VALL-E (VALL-E) and CosyVoice (CosyVoice → CosyVoice 2).
- SpeechT5's TTS (used in a bookmark experiment) is a Transformer encoder-decoder in the same spirit.
What it got wrong, and what came next
- Still autoregressive at inference, so latency grows with utterance length. FastSpeech (FastSpeech) generates all frames in parallel.
- Alignment still learned by attention, so robustness problems remain. Explicit durations (FastSpeech, FastSpeech 2) and Monotonic Alignment Search (Glow-TTS, One TTS Alignment) fix them.
- Quadratic attention cost on long utterances; later work used relative position encodings, windowed attention or convolutions to keep costs down.
Check yourself
Q1Why does Transformer TTS train faster than Tacotron 2 but not synthesize faster?
With teacher forcing, the ground-truth previous frames are known, so masked self-attention computes all frames in parallel. At inference each frame depends on the previously generated one, so the decoder still runs once per frame.
Q2Why add a trainable scale α to the positional encoding?
Phoneme embeddings and mel-frame projections have different scales and statistics, so a fixed-size positional encoding could dominate one and be negligible in the other. A learned α lets each side choose how strongly position information is mixed in.
Q3What would happen to robustness compared with Tacotron 2, and why?
Roughly the same or slightly worse on hard inputs: alignment is still soft attention learned from data, and plain multi-head attention has no location features pushing it forward, so skips and repeats on long or repetitive text remain.
Q4How did this model help FastSpeech?
FastSpeech trained an autoregressive Transformer TTS as a teacher, read phoneme durations off its most diagonal encoder-decoder attention head, and used its generated mels as distillation targets for the non-autoregressive student.
When you're back at a laptop
- In PyTorch, build a 2-layer Transformer decoder over 80-d mel frames with a causal mask; time a teacher-forced forward pass vs a step-by-step generation loop for 800 frames.
- Visualize the encoder-decoder attention of each head in an open Transformer TTS (ESPnet has recipes) and find the most diagonal head.