Reading Room

TTS · Phase 7: Flow matching and diffusion at scale

F5-TTS

F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching

Chen, Niu, Ma, Deng, Wang, Zhao, Yu, Chen · Shanghai Jiao Tong University (X-LANCE) and collaborators · 2024

Deep read20 minDay 9 of 10#75 of 86

In one sentence

F5-TTS is E2 TTS made practical: the same "pad characters to the mel length and infill with flow matching" recipe, but with a ConvNeXt text encoder, a Diffusion Transformer backbone and sway sampling of ODE steps, giving a fast, robust, open zero-shot voice-cloning model.

Why this paper exists

E2 TTS (E2 TTS) showed you can drop phonemes, aligners and duration models. But in practice:

  • It converged slowly. Characters were concatenated directly onto audio frames, so the network had to learn text representations and alignment and acoustics in one go.
  • It was fragile. Without enough training, it mispronounced or skipped words, especially on hard text.
  • It was not open. Like Voicebox (Voicebox), no public weights.

The community needed a simple, open, fast NAR cloning model trained on diverse real-world data.

The core idea

Keep E2's implicit alignment, but give the text its own small encoder before it meets the audio, so the main network receives "text that already knows its local context". Use a modern DiT (Diffusion Transformer: a Transformer whose layers are modulated by the flow time via adaptive layer norm). And at inference, spend ODE steps where they matter: near the noise end.

How it works

characters + filler <F> to N frames
        |
  [ConvNeXt V2 blocks]   text refinement
        |
        v
concat per frame: [ noisy mel x_t | masked mel ctx | text ]
        |
   linear + conv position embedding
        |
  [DiT blocks x L]  <- t via adaLN-zero
        |
   predicted velocity (N x 100)
        |
   Euler ODE, ~32 NFE, sway sampling, CFG
        |
   mel -> [Vocos] -> 24 kHz waveform

Inputs

  • Mel: 100-band log mel at 24 kHz with hop 256, so ~94 frames per second. Shape for a 10 s clip: ~940 × 100.
  • Text: characters (Chinese uses pinyin in the released model; check the repo), padded with filler tokens to N frames, embedded, then passed through ConvNeXt V2 blocks. Convolutions give each character a view of its neighbours, which helps the model find the alignment.
  • Context: the real mel with the target span masked (zeros), as in Voicebox.

Loss

The masked flow-matching loss (Flow Matching):

x_t = (1 − t)·x_0 + t·x_1
L   = E || m ⊙ ( v_θ(x_t, ctx, text, t) − (x_1 − x_0) ) ||²

x_1 the real mel, x_0 Gaussian noise, m the mask (1 on frames to generate), t ∈ [0, 1]. Text and context are dropped at random during training so classifier-free guidance works at inference.

Sway sampling

Plain Euler uses uniform times t = 0, 1/N, 2/N, .... Sway sampling warps them:

t' = t + s · ( cos(π/2 · t) − 1 + t )

With s < 0 (e.g. −1), t' stays close to 0 for longer, so more steps land near the noise end, where the model decides the rough structure of the utterance (alignment, prosody). The paper reports it improves results at a fixed NFE and lets you cut NFE with less loss. It needs no retraining: it only changes the inference schedule, and the paper shows it also helps E2 TTS.

Total duration

You must give the output length in frames. The repo's default rule: frames_total = frames_prompt × (len(text_prompt + text_new) / len(text_prompt)), i.e. assume the same characters-per-second as the prompt (check the repo for the exact current rule; a speed factor scales it).

Build the intuition

Sway sampling is easiest to see in the 1D flow-matching example from Flow Matching: with two data points at −2 and +2, the sample stays near the middle at first, and the field "commits" to one side around t ≈ 0.5. Steps spent before that commitment decide which mode you get; steps after it just slide to the endpoint. For speech, the "mode" is the coarse layout: where each word sits in time, the overall intonation. Get that right and the fine detail is easy.

Analogy: sketching a portrait. Spend most of your strokes on the proportions (where the eyes go). Shading is quick once the layout is right. Sway sampling front-loads the strokes.

What would happen if you remove the ConvNeXt text encoder? You are back to E2 TTS: the DiT must build text features and alignment from raw character embeddings, which the paper found slows convergence and increases failures.

What would happen if your prompt is 15 s of fast Hinglish and your new text is formal Hindi? The duration rule assumes the same character rate, so the output may be rushed. Override the duration or speed.

Numbers worth remembering

  • Training data: Emilia (LibriTTS → Emilia), ~95k hours of in-the-wild English and Chinese speech (check the paper's data section).
  • Base model ~336M parameters: DiT with 22 layers, 16 heads, dim 1024 (approximate; check the config).
  • Abstract: inference RTF ~0.15 (real-time factor, compute time / audio duration) on a GPU, much faster than prior diffusion TTS.
  • Default inference: 32 NFE Euler with sway sampling and CFG; 16 NFE is usable.
  • Vocoder: Vocos (iSTFTNet + Vocos) at 24 kHz.

Where it shows up today

  • One of the most-used open zero-shot cloning models, with many community fine-tunes, including IndicF5 from AI4Bharat for Indian languages (IndicVoices-R for Indian data).
  • The DiT-over-mel + flow matching + Vocos stack is the "NAR cloning" row in today's model zoo; CosyVoice 2's acoustic stage is a close cousin (CosyVoice → CosyVoice 2).
  • Sway-style timestep schedules are a cheap trick worth trying in any flow-matching model.

What it got wrong, and what came next

  • Not streaming. It generates the whole utterance, so time to first audio (TTFA) equals the time to synthesize the full sentence. For voice agents see CosyVoice 2's chunk-aware flow matching (CosyVoice → CosyVoice 2) and Kyutai's DSM (Delayed Streams Modeling (TTS)).
  • Total length is a heuristic. Bad estimates cause rushed or dragged speech.
  • Multi-step and heavy for CPU. 32 DiT passes over the whole mel is fine on a GPU but slow on a laptop CPU. CALM (CALM (Pocket TTS)) with a one-step head trained by drifting (Drifting Models) is Pocket TTS's answer.
  • Non-commercial weights. The released checkpoints inherit Emilia's non-commercial license (check the model card); the code is permissive.
  • Cloning misuse. Like any zero-shot cloner, a few seconds of audio clones a voice.

Check yourself

Q1List the three changes F5-TTS makes to E2 TTS and what each fixes.

A ConvNeXt V2 text encoder refines the padded characters before they meet audio, which speeds convergence and reduces alignment failures. A DiT backbone with adaLN-zero time conditioning replaces the U-Net-skip Transformer. Sway sampling puts more ODE steps near noise, improving quality at a fixed or reduced NFE.

Q2Write the sway sampling formula and say what s < 0 does.

t' = t + s·(cos(π/2·t) − 1 + t). For s < 0 the warped times stay small longer, so steps cluster near t = 0, the noise end, where coarse structure is decided.

Q3Why does sway sampling need no retraining?

The network learns velocity at every t in [0, 1]. Sway sampling only changes which t values the ODE solver visits at inference.

Q4How does F5-TTS decide how long the output is?

You give the total frame count. The default rule assumes the new text is spoken at the same characters-per-second as the prompt, optionally scaled by a speed factor.

Q5What happens to quality and latency if you drop NFE from 32 to 8?

Latency drops roughly 4× (each step is one DiT forward pass). Quality degrades: more blur and artifacts, and occasionally alignment errors, because the coarse ODE misses the early "commitment" phase. Sway sampling softens this.

Q6Why is F5-TTS a poor fit as-is for a real-time voice agent?

It needs the full sentence text and produces the full utterance in one go, so speech cannot start until the LLM has finished the sentence and the model has run all steps. Streaming designs start speaking after a few words.

Q7Why was a fine-tune like IndicF5 practical?

The model uses characters, not phonemes, so adding a language only needs transcribed audio in that script: no G2P or aligner. The architecture is unchanged; only the vocabulary and data change.

When you're back at a laptop

  • pip install f5-tts, record a clean 5-10 s prompt of yourself with its transcript, and synthesize three sentences.
  • Sweep NFE in {8, 16, 32} and time each on your RTX 3050; compute RTF.
  • Sweep the speed / total duration by ±25% and listen.
  • Score outputs: WER with Whisper, speaker similarity with a WavLM speaker model, UTMOS (UTMOS).
  • Try IndicF5 on a Hindi sentence and a Hinglish sentence; note where it mispronounces English words.
arXiv ↗PDF ↗