Glow-TTS

Why this paper exists

By 2020 there were two ways to get alignment for TTS, and both were unsatisfying:

• Learn it with attention (Tacotron 2, Tacotron 2; Transformer TTS, Transformer TTS): end to end, but soft attention can skip, repeat or fail to stop, and decoding is autoregressive.
• Get it from outside (FastSpeech, FastSpeech, from a teacher's attention; FastSpeech 2, FastSpeech 2, from a forced aligner): robust and parallel, but you need an extra trained model or an aligner with a lexicon for each language.

Glow-TTS asks: since we know the alignment is monotonic and every token covers a contiguous run of frames, why not search over exactly those alignments, inside training, with no external help?

The core idea

Turn alignment into a hard, discrete latent variable and do the classic hard-EM / Viterbi training loop from HMM speech recognition:

1. With the current model parameters, find the single best monotonic alignment by dynamic programming (MAS).
2. With that alignment fixed, update the parameters to maximize the likelihood.

To make step 1 well defined you need a likelihood of the mel given each text token. A normalizing flow decoder provides it: it maps the mel spectrogram invertibly to a latent z whose prior is a simple Gaussian per text token, and the change-of-variables formula gives the exact log-likelihood.

How it works

text -> encoder (Transformer) -> μ_i, σ_i per token (N)
                                   |
             TRAIN: MAS over log N(z_j; μ_i, σ_i)
                    -> alignment A -> durations d_i
                                   |
mel x (T x 80) <-> flow decoder <-> z (T x 80)
       (actnorm, invertible 1x1 conv, affine coupling)

             INFER: duration predictor -> d_i
                    expand μ, σ by d_i
                    z ~ N(μ, (σ·τ)²) -> flow -> mel

Model pieces

• Text encoder: a Transformer encoder (with relative position encodings) over phonemes. A final projection outputs, for every token i, a mean μ_i and standard deviation σ_i (each 80-d, one per mel channel). Together they define the prior for frames aligned to that token.
• Flow decoder: a stack of invertible blocks (Glow-style): activation normalization, invertible 1×1 convolution (mixes channels), and an affine coupling layer (half the channels predict a scale and shift for the other half using a non-causal WaveNet-like network). Every block is invertible with a cheap log-determinant.
• Duration predictor: the same conv architecture as FastSpeech's (FastSpeech), trained with MSE on log durations extracted from MAS. Its input is the encoder output with gradients stopped, so it does not disturb the encoder.

Likelihood

z = f^{-1}(x)                   (decoder run backwards)
log p(x | c) = log p_Z(z | c, A) + Σ_k log |det J_k|
log p_Z(z | c, A) = Σ_j log N(z_j ; μ_{A(j)}, σ_{A(j)})

• x: the mel spectrogram, frames j = 1..T.
• c: the text; μ_i, σ_i come from the encoder.
• A(j): which text token frame j is aligned to.
• J_k: Jacobian of flow block k; the log-determinant terms account for how the flow stretches space.

Monotonic Alignment Search

Define L[i, j] = log N(z_j ; μ_i, σ_i): how well frame j fits token i. Valid alignments start at (1, 1), end at (N, T), never go back, never skip a token, and give every token at least one frame. Let Q[i, j] be the best total log-likelihood of aligning frames 1..j with tokens 1..i such that frame j is on token i:

Q[1, 1] = L[1, 1]
Q[i, 1] = -inf                         for i > 1
Q[i, j] = max(Q[i-1, j-1], Q[i, j-1]) + L[i, j]

Either frame j-1 was on the previous token (move on) or on the same token (stay). The answer is Q[N, T]. Backtrack from (N, T): at each column choose whichever predecessor gave the max. Cost O(N × T). Durations are the run lengths.

In training, MAS runs on the current z with no gradient (it is a discrete argmax), and the loss is the negative log-likelihood under the alignment it found.

Inference

Predict durations, round them, expand μ and σ to T frames (a length regulator, as in FastSpeech), sample z = μ + σ · ε · τ with ε ~ N(0, I) and a temperature τ below 1 (lower τ gives more stable, less varied speech; check the paper for the value used), then run the flow forward to get the mel. A HiFi-GAN or WaveGlow vocoder makes the waveform. Everything is parallel.

Build the intuition

MAS by hand: 2 tokens, 4 frames

Log-likelihoods L[i, j], tokens a and b:

Fill Q:

• Row a: Q[a,1] = -1, Q[a,2] = -1 + -1 = -2, Q[a,3] = -2 + -4 = -6, Q[a,4] = -6 + -6 = -12 (row a can only "stay").
• Row b: Q[b,1] = -inf (b cannot start).
• Q[b,2] = max(Q[a,1], Q[b,1]) + L[b,2] = max(-1, -inf) - 3 = -4
• Q[b,3] = max(Q[a,2], Q[b,2]) + L[b,3] = max(-2, -4) - 1 = -3
• Q[b,4] = max(Q[a,3], Q[b,3]) + L[b,4] = max(-6, -3) - 1 = -4

Backtrack from (b, 4):

• Q[b,4] came from Q[b,3] (-3 beats -6): frame 3 is on b.
• Q[b,3] came from Q[a,2] (-2 beats -4): frame 2 is on a.
• Then frame 1 on a.

Alignment a a b b, durations [2, 2], total -1 -1 -1 -1 = -4. Check the alternatives: a b b b scores -1 -3 -1 -1 = -6, a a a b scores -1 -1 -4 -1 = -7. MAS found the best one without enumerating them.

Compared with CTC

The recurrence has the same shape as the CTC forward algorithm in ASR (CTC): a lattice of (token, frame) with "stay" and "advance" moves. Differences:

• CTC sums over paths (log-sum-exp) to get a total probability; MAS takes the max (Viterbi) to get one path.
• CTC has a blank symbol and many frames to few labels with optional emissions; MAS has no blanks and every token must own at least one frame.
• CTC's output is a loss; MAS's output is an alignment that a separate loss uses. NVIDIA's aligner (One TTS Alignment) combines both: a CTC-style sum as a loss and a MAS-style max for hard durations.

Why the flow matters

MAS needs a per-frame, per-token score that reflects the true data likelihood. With a plain L2 decoder you could use squared error as a score, which is roughly what later models (Grad-TTS, VITS) do with a Gaussian in a latent space. The flow makes it exact: a better alignment always means a higher likelihood, so the EM-style loop is principled and converges.

Analogy: MAS is a GPS route planner for a road with no turns back. Each (word, moment) cell has a cost; the planner finds the cheapest route from start to finish in one sweep, remembering at each cell only the best way to get there.

Numbers worth remembering

• Data: LJSpeech (~24 hours, single speaker), plus multi-speaker experiments on LibriTTS.
• Synthesis about 15.7x faster than autoregressive Tacotron 2 in their measurement, with comparable MOS (check Table 1).
• MAS is O(N × T) per utterance: for 100 phonemes and 800 frames, 80,000 cells, cheap on CPU (usually implemented in Cython or Numba).
• Robust on long utterances where Tacotron 2's attention breaks down (see their character-error-rate vs length plot).

Where it shows up today

• MAS is everywhere: VITS (VITS) and so Piper and MMS-TTS; Grad-TTS (Grad-TTS); Matcha-TTS (Matcha-TTS); YourTTS (YourTTS); and NVIDIA's aligner (One TTS Alignment) used in FastPitch.
• The text encoder outputting a per-token Gaussian prior is the template for VITS's prior encoder and Grad-TTS's "mean mel".
• AI4Bharat benchmarked Glow-TTS-style models for 13 Indian languages (Indic TTS (AI4Bharat)); MAS is attractive there because no language-specific aligner is needed.

What it got wrong, and what came next

• Flows are large and heavy for the quality they give; the decoder must be invertible, which limits architecture. Diffusion (Grad-TTS, Grad-TTS) and flow matching (Matcha-TTS, Matcha-TTS) keep MAS but use more expressive decoders.
• Still two-stage with a separate vocoder. VITS (VITS) merges Glow-TTS's prior and MAS with a HiFi-GAN decoder, end to end.
• Deterministic durations give monotonous rhythm; VITS's stochastic duration predictor samples them.
• Hard alignment from scratch can be slow to converge; One TTS Alignment adds a soft attention with a diagonal prior to stabilize it.
• Later large models (E2, F5-TTS, E2 TTS, F5-TTS) drop explicit alignment entirely and let attention learn it implicitly at scale.

Check yourself

When you're back at a laptop

• Implement MAS in NumPy on a random log-likelihood matrix and plot the path:

def mas(L):                     # L: N x T log-liks
    N, T = L.shape
    Q = np.full((N, T), -np.inf); Q[0, 0] = L[0, 0]
    for j in range(1, T):
        for i in range(min(N, j + 1)):
            stay = Q[i, j-1]
            move = Q[i-1, j-1] if i > 0 else -np.inf
            Q[i, j] = max(stay, move) + L[i, j]
    path, i = [], N - 1
    for j in range(T - 1, -1, -1):
        path.append(i)
        if i > 0 and (j == i or Q[i-1, j-1] > Q[i, j-1]):
            i -= 1
    return path[::-1]           # token index per frame

• Verify it on the 2 × 4 example above (expect [0, 0, 1, 1]), then compare with the monotonic_align module in the VITS repo.
• Swap max for logsumexp and see that you now have a forward algorithm; this is the bridge to One TTS Alignment.