TTS · Phase 6: Codec language-model TTS
Seed-TTS
Seed-TTS: A Family of High-Quality Versatile Speech Generation Models
In one sentence
Seed-TTS is ByteDance's large-scale zero-shot TTS family (an autoregressive LM over speech tokens plus a diffusion acoustic model, and a fully non-autoregressive variant) that reported speech close to human recordings, and it introduced Seed-TTS-eval, the WER + speaker-similarity benchmark most TTS papers now report.
Why this paper exists
By mid-2024 codec-LM TTS (VALL-E, VALL-E; CosyVoice, CosyVoice → CosyVoice 2) worked, but quality still fell short of real recordings in naturalness and expressiveness, and every paper evaluated on its own test set, so results were hard to compare. ByteDance had the data and compute to push scale, and wanted to show what an industrial system could do, including emotion control, voice conversion and speech editing. The paper describes the system at a high level (architecture details are sparse, no weights were released), but its benchmark is public.
The core idea
Scale the "LLM over speech tokens + strong acoustic model" recipe with far more data and model capacity, then add the tricks text LLMs use after pretraining: fine-tuning for speaker and instruction following, and reinforcement learning on rewards like intelligibility and speaker similarity.
How it works
- Speech tokenizer turns audio into tokens.
- Autoregressive Transformer LM generates speech tokens from text, conditioned on a prompt's tokens (as in VALL-E).
- Diffusion acoustic model turns tokens into continuous acoustic representations, then a vocoder produces the waveform.
- Seed-TTS_DiT: a fully non-autoregressive variant using a diffusion transformer directly, with no explicit phoneme durations (similar in spirit to E2 TTS, E2 TTS).
- Speech factorization by self-distillation for voice conversion (separating timbre from content).
- RL fine-tuning (the paper compares methods such as PPO, REINFORCE and DPO) with rewards from ASR WER, speaker similarity and emotion classifiers, to reduce errors and improve control.
Seed-TTS-eval (the part you will use):
- test-en: English prompts and texts from Common Voice.
- test-zh: Chinese from DiDiSpeech; test-hard: tricky Chinese sentences (tongue twisters, repetitions).
- Metrics: WER from an ASR model (Whisper-large-v3 for English, Paraformer for Chinese) and speaker similarity (SIM) as cosine similarity of WavLM-based speaker-verification embeddings between prompt and output.
Build the intuition
Analogy: Seed-TTS-eval is to TTS what a standard exam is to schools. Each school's own tests always made its students look good; a shared exam lets you compare.
Reading a results table. A row like "WER 2.2, SIM 0.76" on test-en means the Whisper transcript of the output differs from the input text by about 2 words in 100, and the output voice is close to the prompt voice (the ground-truth recordings themselves score around the mid-0.7s for SIM on this set, so values near that are near-human). Low WER with low SIM means a clear but generic voice; high SIM with high WER means a good clone that mispronounces or skips.
What would happen if you built an Indian-language version? You would need prompts and texts for Hindi, Tamil and Hinglish, an ASR model good on them (for example IndicWhisper from Vistaar, Vistaar), and the same WavLM similarity model. That is the evaluation you would run for an Indian customer-support agent.
Numbers worth remembering
- Seed-TTS reported test-zh CER around 1.1 and test-en WER around 2.2, with SIM around 0.76-0.80 (check the paper's tables), near the ground-truth recordings.
- Subjective tests: CMOS close to 0 against human recordings, i.e. listeners had no clear preference (check the paper).
- Training data is described only as very large-scale (orders of magnitude beyond earlier TTS); exact hours are not given in a form you should quote.
Where it shows up today
- Seed-TTS-eval is in the results tables of CosyVoice 2 (CosyVoice → CosyVoice 2), F5-TTS (F5-TTS), MaskGCT (MaskGCT), Llasa and Spark-TTS (Llasa + Spark-TTS).
- RL with WER and SIM rewards is now common in TTS post-training (check recent papers such as CosyVoice 3), and the same kind of reward models drive Llasa's inference-time search (Llasa + Spark-TTS).
What it got wrong, and what came next
- Closed: no weights, few architecture details. Open models (CosyVoice 2, F5-TTS, Llasa) reproduce the recipe.
- The benchmark is narrow: read sentences in English and Chinese, WER and SIM only. It says nothing about naturalness, streaming latency or other languages; UTMOS and human preference tests (UTMOS) fill some of that gap.
- AR LM robustness is still a concern, addressed by RL and by non-autoregressive alternatives.
Check yourself
Q1What two metrics does Seed-TTS-eval report, and how is each computed?
WER (or CER for Chinese): transcribe the generated audio with a fixed ASR model (Whisper-large-v3 for English, Paraformer for Chinese) and compare to the input text. SIM: cosine similarity between speaker-verification embeddings (a WavLM-based model) of the prompt and the output.
Q2Why does a low WER alone not show that a cloning TTS is good?
A model can speak every word clearly in a generic voice and get a low WER while failing at cloning. You need SIM for voice similarity, and a naturalness measure (MOS or UTMOS) because neither metric captures prosody.
Q3Why use RL after pretraining a TTS model?
Likelihood training does not directly penalize the failures users notice: skipped words, wrong voice, flat emotion. With an ASR model, a speaker-verification model and an emotion classifier as reward functions, RL pushes the model toward outputs that score well on exactly those.
When you're back at a laptop
- Clone the public
seed-tts-evalrepo, generate test-en outputs with F5-TTS or CosyVoice 2, and compute WER and SIM. - Build a 50-sentence Hindi / Hinglish version with your own prompts, and see how much WER changes when you swap the ASR model.