TTS · Phase 7: Flow matching and diffusion at scale
NaturalSpeech 2 → NaturalSpeech 3
NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers; NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models
In one sentence
NaturalSpeech 2 generates the continuous latents of a neural codec with a non-autoregressive diffusion model (avoiding VALL-E's long token sequences), and NaturalSpeech 3 goes further by building a codec (FACodec) that splits speech into separate content, prosody, timbre and acoustic-detail codes, then generates each part with its own diffusion stage.
Why this paper exists
VALL-E (VALL-E) showed zero-shot cloning by predicting discrete codec tokens with a language model. Microsoft's own group saw its weaknesses:
- Long sequences. Residual vector quantization (RVQ: several codebooks per frame, each coding the residual of the previous) gives many tokens per second, and modelling them autoregressively is slow.
- Instability. Sampling tokens left to right can repeat or skip words.
- Everything tangled. One token stream mixes what is said, how it is said and who says it, so the model must learn all factors at once.
NS2 attacks the first two; NS3 attacks the third.
The core idea
NS2: keep the codec (EnCodec-style RVQ codec trained by the authors), but model the continuous vector before quantization instead of the discrete tokens. One continuous vector per frame is short and dense, so a diffusion model can generate all frames in parallel. A phoneme encoder with duration and pitch predictors gives alignment and prosody, and a speech prompt (a few seconds of the target speaker) is attended to for in-context cloning.
NS3: make the representation itself disentangled. FACodec (factorized codec) uses separate quantizers for content, prosody and acoustic detail plus a global timbre vector, trained with supervision and adversarial tricks so each subspace carries only its own factor. A factorized diffusion model then generates prosody, content and detail codes in turn, each prompted by the matching part of the reference speech.
How it works
NaturalSpeech 2
phonemes -> phoneme encoder -> duration + pitch predictors
| (prompted)
v
latent diffusion (WaveNet-style net, NAR)
^ speech prompt via attention
v
continuous codec latents z -> codec decoder -> wav- The diffusion model predicts the clean latent
z(rather than noise) and is trained with a latent loss plus an RVQ-based regularizer (check the paper for the exact loss). - Because the target is continuous, there is no codebook choice per frame and no AR loop.
- Trained on ~44k hours of English speech (and also does singing with singing data added).
NaturalSpeech 3
wav -> FACodec encoder
-> content codes (phone-like)
-> prosody codes (pitch/energy-like)
-> detail codes (residual acoustics)
-> timbre vector (speaker, global)
text -> durations -> [diffusion: prosody]
-> [diffusion: content]
-> [diffusion: detail]
+ timbre from prompt -> FACodec decoder -> wav- The diffusion is discrete (a masked/absorbing-style process over codes), non-autoregressive.
- Scaled to ~1B parameters and ~200k hours of data (per the paper; check the scaling section).
Build the intuition
Analogy: NS2 hands the painter one rich colour photo to reproduce. NS3 hands over separate layers (a line drawing, a lighting map, a colour palette) and lets a specialist fill each one. If the layers are cleanly separated, each specialist's job is easy, and you can swap one layer (a different speaker's timbre) without touching the others.
What would happen if FACodec's disentanglement leaked, e.g. some timbre stayed in the content codes? Then a voice prompt could not fully override the speaker: cloned speech would carry traces of the training speaker. That is why NS3 spends most of its effort on the codec's training objectives.
Numbers worth remembering
- NS2 training data: ~44k hours (English, from MLS); NS3: ~200k hours, up to ~1B parameters.
- Both are non-autoregressive and both still use explicit duration prediction.
- NS3 reported quality on par with human recordings on LibriSpeech test sets in its evaluation (check the tables for exact CMOS/SIM/WER).
Where it shows up today
- Continuous latents instead of discrete tokens returns in MELLE (MELLE) and CALM (CALM (Pocket TTS)), which model continuous frames autoregressively.
- Factorized/disentangled tokens show up in Spark-TTS's global + semantic tokens (Llasa + Spark-TTS) and in the semantic vs acoustic split of AudioLM-style systems (AudioLM).
- FACodec was released through Amphion, the same toolkit as MaskGCT (MaskGCT).
What it got wrong, and what came next
- Complex pipelines. Phonemes, duration and pitch predictors, a custom codec, several diffusion stages. E2 TTS (E2 TTS) and F5-TTS (F5-TTS) show a much simpler flow-matching model on characters matches or beats this.
- Weights not broadly released for the full systems, so the community built on F5-TTS and CosyVoice (CosyVoice → CosyVoice 2) instead.
- Not streaming, like all whole-utterance diffusion models; see DSM (Delayed Streams Modeling (TTS)).
Check yourself
Q1Why does NS2 model continuous codec latents instead of RVQ tokens?
RVQ tokens give many symbols per second (several codebooks per frame), which makes autoregressive modelling slow and unstable. One continuous vector per frame is short and can be generated for all frames in parallel by diffusion.
Q2What four factors does FACodec separate, and why bother?
Content, prosody, acoustic detail and timbre. Each generator then has a simpler job, and each can be prompted separately (e.g. timbre from the reference speaker), which improves cloning and controllability.
Q3What alignment mechanism do NS2 and NS3 still need that E2 TTS removes?
An explicit duration predictor on phonemes. E2 TTS pads the character sequence to the mel length and lets the model learn alignment implicitly, so only a total duration is needed.
Q4What would go wrong if timbre leaked into FACodec's content codes?
The content stream would carry some speaker identity from training data, so cloned output would not fully match the prompt speaker, and voice conversion by swapping timbre would be incomplete.
When you're back at a laptop
- Install Amphion and run FACodec on a clip of your voice: encode, swap the timbre vector with another speaker's, decode, and listen for leakage.
- Compare token counts: tokens per second for an 8-codebook RVQ codec vs one continuous latent vector per frame.