Reading Room

TTS · Phase 7: Flow matching and diffusion at scale

E2 TTS

E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTS

Eskimez, Wang, Thakker, et al. · Microsoft · 2024

Skim9 minDay 9 of 10#74 of 86

In one sentence

E2 TTS takes Voicebox's flow-matching infilling model and throws away phonemes, durations and the aligner: it just pads the character sequence with filler tokens up to the mel length, and the model learns the alignment on its own.

Why this paper exists

Voicebox (Voicebox) and NaturalSpeech (NaturalSpeech 2 → NaturalSpeech 3) worked well but carried a heavy front end:

  • a grapheme-to-phoneme (G2P) converter per language,
  • a forced aligner to get phone durations for training,
  • a separate duration model at inference.

Each piece is language-specific, needs its own data, and fails on things like names, code-mixed text or new languages. The question: is all of that necessary, or can a big enough model learn alignment from raw characters?

The core idea

Make the text sequence the same length as the audio by padding, and let attention sort out which characters go with which frames.

text:  "hello" -> h e l l o <F> <F> <F> ... <F>
                  |<------- N mel frames ------>|
mel:   [prompt frames][  masked frames to fill  ]

<F> is a filler token. The padded character sequence, the masked mel context and the noisy mel are stacked frame by frame and fed to a Transformer trained with the Voicebox infilling flow-matching loss (Flow Matching). Nothing tells the model that frame 37 is the letter "l"; it learns that from data.

At inference you choose only the total duration of the output (in practice from a simple rule such as scaling the prompt's speaking rate by the number of characters).

How it works

  1. Training: take an utterance's mel x (N frames) and its character transcript (M < N characters). Pad the characters to length N with filler tokens. Mask a random span of the mel.
  2. Feed [noisy mel x_t ; masked mel context ; padded character embeddings] per frame into a Transformer with U-Net-style skips (Voicebox's backbone).
  3. Loss: flow-matching MSE on masked frames, as in Voicebox.
  4. Inference: concatenate prompt transcript + new text, pad to the chosen total length, set the prompt's frames as context, integrate the ODE, vocode.

The paper also describes extensions (E2 TTS X1, X2): one removes the need for the prompt's transcript, the other lets you specify pronunciation for some words (check the paper for details).

Build the intuition

Analogy: instead of telling a choir conductor exactly how many beats each syllable gets, you hand them the lyrics and the total length of the piece. After enough rehearsal (data), they distribute the syllables naturally.

What would happen if you chose a total duration that is far too short? The model must squeeze all characters into fewer frames: speech gets fast and may slur or drop words. Too long: slow speech or long pauses. Duration is now your only knob for speaking rate.

Why this helps Indian languages: no Hindi or Tamil G2P and no aligner are needed, only characters and transcribed audio. That is why the F5-TTS descendant IndicF5 was feasible (F5-TTS).

Numbers worth remembering

  • Trained on Libriheavy (~50k hours of English audiobook speech; check the paper).
  • The paper reports zero-shot results comparable to or better than Voicebox and NaturalSpeech 3 on its LibriSpeech-based test set (check Table 1 for exact WER/SIM).
  • No phonemes, no duration model, no aligner: one model plus a total-length choice.

Where it shows up today

  • F5-TTS (F5-TTS) is the direct successor and is what people actually run; F5-TTS's repo ships an E2 TTS configuration too.
  • The "pad text to audio length" trick sits in many open NAR cloning models.
  • MaskGCT (MaskGCT) also avoids explicit alignment, with a masked token model instead.

What it got wrong, and what came next

  • Slow convergence and robustness issues. Learning alignment from scratch with characters glued directly onto audio frames takes long training and can fail on hard text. F5-TTS adds a ConvNeXt text encoder and a DiT backbone to fix this (F5-TTS).
  • Total duration is a guess. A bad estimate hurts naturalness; there is still no learned duration control.
  • Not streaming: whole utterance at once; see Delayed Streams Modeling (TTS).

Check yourself

Q1What three components does E2 TTS remove compared to Voicebox?

The grapheme-to-phoneme front end (it uses characters), the forced aligner used to get training durations, and the duration model at inference. Alignment is learned implicitly.

Q2What are the filler tokens for?

They pad the character sequence to the same length as the mel spectrogram so text and audio can be combined frame by frame. The model learns which frames correspond to which characters.

Q3What do you still have to choose at inference?

The total output duration. It sets speaking rate; a common rule scales the prompt's frames-per-character to the new text's length.

Q4Why is this design attractive for a new Indian language?

It needs only transcribed audio and characters: no language-specific G2P, phone set or aligner, which are often missing or weak for Indian languages and code-mixed text.

When you're back at a laptop

  • In the F5-TTS repo, run the E2 TTS checkpoint and the F5-TTS checkpoint on the same prompt and text; compare WER with Whisper.
  • Vary the total duration by ±30% and listen to rate and clarity.
arXiv ↗PDF ↗