TTS · Phase 6: Codec language-model TTS
SPEAR-TTS
Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervision
In one sentence
SPEAR-TTS splits TTS into "reading" (text → semantic tokens, a translation task that needs paired data) and "speaking" (semantic → acoustic tokens, which needs only untranscribed audio), so a multi-speaker, voice-cloning TTS can be trained with as little as 15 minutes of transcribed speech.
Why this paper exists
Most TTS needs many hours of transcribed, clean speech per language. That is the bottleneck for low-resource languages, including most Indian languages. AudioLM (AudioLM) had shown that untranscribed audio alone can teach a model to produce realistic speech in any voice, given semantic tokens. The missing piece was getting from text to those semantic tokens cheaply.
The core idea
Semantic tokens (k-means clusters of w2v-BERT features, data2vec / w2v-BERT / BEST-RQ) are close to phonetic content and nearly speaker-independent. So:
- Reading (text → semantic tokens) is like machine translation between two "languages". It needs parallel data, but because the target carries no voice information, very little is enough, especially with pretraining and back-translation.
- Speaking (semantic → acoustic tokens, SoundStream codes, SoundStream) needs no text at all: tokenize any audio both ways and learn the mapping. Voice comes from an audio prompt.
How it works
text --[reading: seq2seq Transformer]--> semantic
semantic + 3 s prompt --[speaking: decoder LM]-->
SoundStream tokens --> decoder --> waveform- Reading model: an encoder-decoder Transformer. It is first pretrained as a denoising autoencoder on semantic tokens alone (corrupt a token sequence, reconstruct it), from untranscribed audio. Then it is fine-tuned on the small parallel set. Back-translation grows the parallel data: train a semantic → text model on the small set, run it on lots of untranscribed audio, and use the synthetic (text, semantic) pairs for more training.
- Speaking model: a decoder-only Transformer, AudioLM-style, trained on LibriLight audio only. Voice prompting: prefix with the semantic and acoustic tokens of a ~3 s clip, and it continues in that voice. No speaker labels anywhere.
Build the intuition
Analogy: a news anchor and a script writer. The writer (reading) turns facts into a script; the anchor (speaking) reads any script in their own voice. You only need the two to agree on a script format, so you can train the anchor on hours of talk without ever showing them a newspaper.
What would happen if you applied this to Bhojpuri, with 30 minutes of transcribed speech and 500 hours of radio? The speaking model learns from all 500 hours (assuming w2v-BERT-style semantic tokens work on that language, which may need a multilingual SSL model), and only the reading model depends on the 30 minutes. That is the low-resource argument.
Numbers worth remembering
- 15 minutes of parallel (text, audio) data was enough for a character error rate competitive with fully supervised systems, per the abstract (check the paper for which dataset the 15 minutes came from).
- Voice prompt: ~3 s, no speaker labels in training.
- Speaking model trained on LibriLight (~60k hours, untranscribed).
- Check the paper's tables for CER and MOS against the baselines.
Where it shows up today
- The text → semantic → acoustic pipeline is the shape of CosyVoice (CosyVoice → CosyVoice 2), MaskGCT (MaskGCT) and Spark-TTS (Llasa + Spark-TTS).
- The idea that the acoustic stage can be trained on untranscribed audio motivates the large in-the-wild datasets like Emilia (LibriTTS → Emilia) and IndicVoices-R (IndicVoices-R).
What it got wrong, and what came next
- Two-stage hierarchy, slow AR decoding: SoundStorm (Google, 2023) made the speaking stage parallel; flow matching (CosyVoice, F5-TTS, F5-TTS) replaced it entirely.
- Unsupervised semantic tokens are not perfectly speaker-free or phoneme-aligned. CosyVoice uses ASR-supervised tokens.
- English experiments on read speech: the low-resource setting was simulated by limiting English parallel data, not tested on a real low-resource language.
Check yourself
Q1Why does the reading model need so little parallel data?
Its output, semantic tokens, carries mostly content and almost no voice or recording information, so mapping text to it is a constrained, translation-like problem. Pretraining on semantic tokens and back-translation from untranscribed audio supply most of what the model needs.
Q2What data does the speaking model train on?
Only untranscribed audio. Each clip is tokenized into semantic and acoustic tokens and the model learns to produce the acoustic tokens from the semantic ones, with a prefix from the same speaker acting as the voice prompt.
Q3How is back-translation used here?
A semantic → text model is trained on the small parallel set and run on large amounts of untranscribed audio to create synthetic text-semantic pairs. The reading model is then trained on these, which greatly increases its training data.
When you're back at a laptop
- Extract HuBERT k-means tokens for the same sentence spoken by three speakers and measure how similar the token sequences are (edit distance). This tells you why reading needs little data.
- Sketch how you would train a SPEAR-style speaking model for Hindi using only IndicVoices audio: which SSL model for semantic tokens, which codec for acoustic tokens.