TTS track
Speech synthesis
From WaveNet and Tacotron 2 to HiFi-GAN, VITS, Kokoro, neural codecs, VALL-E, CosyVoice, F5-TTS, Kyutai's streaming TTS, Pocket TTS and Indian-language TTS.
0 of 51 read · about 12.0 hours of reading
The map: how every model fits together
Almost every TTS system is:
text → normalization + G2P → TEXT ENCODER
→ ALIGNMENT / DURATION → ACOUSTIC MODEL
→ mel / tokens / latents → VOCODER → waveform| Choice | Options |
|---|---|
| Representation | mel spectrogram → discrete codec tokens → continuous latents |
| Alignment | learned attention → explicit durations (FastSpeech, MAS) → implicit (codec LMs, E2 / F5-TTS) |
| Generation | autoregressive vs non-autoregressive |
| Objective | regression → GAN → flow → diffusion → flow matching → language modelling → one-step (consistency, drifting) |
| Voice control | speaker ID → speaker embedding → style tokens → audio prompt → text description |
| Streaming | whole sentence vs audio starting before the text is finished |
TTS went from "predict a mel, then vocode it" to "treat audio as tokens or latents and model them like a language". Unlike ASR, TTS has many valid outputs for one input, so the model must be generative and evaluation needs humans or learned metrics.
Phase 1
Foundations
Text has a few tokens per second, audio has 16,000-48,000 samples per second, and you don't know how long each sound should last. The first neural systems: WaveNet and Tacotron.
Phase 2
Non-autoregressive TTS and alignment
Explicit durations and Monotonic Alignment Search: how TTS stopped skipping and repeating words, and got fast.
Phase 3
Vocoders
The mel spectrogram throws away phase, so something has to generate a believable waveform from it. Often the latency bottleneck in production.
Phase 4
End-to-end and expressive TTS
VITS, StyleTTS 2 (the base of Kokoro) and Matcha-TTS: small, fast, natural-sounding models.
Phase 5
Neural audio codecs
Codecs turn audio into a short sequence of tokens and back. They are what lets a language model speak.
Phase 6
Codec language-model TTS
TTS as language modelling over audio tokens, and zero-shot voice cloning from a few seconds of audio.
Phase 7
Flow matching and diffusion at scale
Non-autoregressive zero-shot TTS: Voicebox, NaturalSpeech, E2 and F5-TTS.
Phase 8
Continuous and streaming TTS
A voice agent needs audio within a couple hundred milliseconds while the LLM is still writing. Kyutai's CALM, drifting and delayed streams.
Phase 9
Data, evaluation and Indian languages
Where TTS data comes from, how TTS is measured, and AI4Bharat's Indian-language work.