Reading Room

TTS · Phase 6: Codec language-model TTS

XTTS

XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Model

Casanova, Davis, Gölge, et al. · Coqui · 2024

Skim9 minDay 8 of 10#66 of 86

In one sentence

XTTS is Coqui's production zero-shot TTS: a GPT-2-style autoregressive model predicts discrete audio tokens from text, conditioned on speaker latents pooled from a few seconds of reference audio, and a HiFi-GAN-style decoder turns the model's hidden states straight into a waveform, in 16+ languages including Hindi.

Why this paper exists

VALL-E (VALL-E) showed codec-LM voice cloning but was English-only and never released. Open multilingual cloning models (YourTTS, YourTTS) were weaker. Coqui wanted one open model that clones a voice from ~6 seconds, speaks many languages (including low-resource ones), streams with low latency, and runs on one consumer GPU. XTTS builds on Tortoise TTS (an earlier open GPT + diffusion model) and simplifies it for speed.

The core idea

Keep the "LLM over audio tokens" idea but make every piece cheap:

  • Coarse audio tokens from a small VQ-VAE over mel spectrograms, at a low frame rate (about 21.5 Hz, check the paper), so the GPT generates few tokens per second.
  • Speaker conditioning from a Perceiver-style resampler that turns any-length reference audio into a fixed set of latent vectors.
  • Skip a separate token-to-mel stage: a HiFi-GAN-style decoder (HiFi-GAN) is trained to decode the GPT's hidden states (not the tokens) directly to audio, also conditioned on a speaker embedding.

How it works

reference clip(s) -> mel -> Perceiver resampler
                              -> 32 speaker latents
text (BPE) ----------+
                     v
            GPT-2-style decoder (AR)
            -> audio tokens (VQ-VAE codes)
            -> last hidden states
                     v
   HiFi-GAN decoder (+ speaker embedding)
                     v
               24 kHz waveform
  1. VQ-VAE (small, ~13M parameters per the paper) encodes mel frames into codes from a large codebook; these are the GPT's targets.
  2. GPT-2-style decoder-only LM (~443M parameters, check the paper): input = speaker latents + text tokens; output = audio codes, one at a time.
  3. Decoder: takes the GPT's final hidden states for each audio token, upsamples to 24 kHz waveform with HiFi-GAN blocks; a speaker-verification embedding is added to keep the voice consistent.
  4. Streaming: since the decoder works on chunks of GPT latents, audio can be emitted while the GPT is still generating.

Build the intuition

Analogy: a dubbing artist who listens to a few seconds of the original actor, writes themselves short notes on the voice (the 32 speaker latents), then reads the new script while keeping those notes in view.

Why decode hidden states instead of tokens? The VQ codes are coarse (few per second, one codebook), so decoding them directly loses detail. The GPT's hidden state at each step contains more information than the chosen token (it "knows" context and prosody), so decoding from it recovers detail the token alone does not hold.

What would happen if you fed XTTS Hinglish text like "aapka EMI 5,000 rupees hai"? It has to choose a language tag per request, the tokenizer may not handle mixed scripts well, and numbers must be normalized first (Text normalization challenge). Code-mixing is a known weak spot; test before shipping.

Numbers worth remembering

  • 16 languages in the paper; the released XTTS-v2 lists 17 including Hindi (check the model card).
  • Cloning from about 6 seconds of reference audio.
  • GPT backbone in the hundreds of millions of parameters (~443M per the paper); output at 24 kHz.
  • Streaming with low first-chunk latency on a GPU (Coqui reported under ~200 ms; check the paper or docs).

Where it shows up today

  • One of the most deployed open cloning models (XTTS-v2), including in community voice-agent stacks; Coqui shut down in early 2024 and maintenance continues in community forks. Note its non-commercial Coqui Public Model License.
  • Its pattern (AR LM over coarse tokens + strong decoder + speaker conditioning) parallels CosyVoice (CosyVoice → CosyVoice 2) and Seed-TTS (Seed-TTS).

What it got wrong, and what came next

  • AR failure modes: occasional hallucination, repetition or truncation, especially on short or unusual text (shared with VALL-E). Non-autoregressive F5-TTS (F5-TTS) and supervised semantic tokens (CosyVoice, CosyVoice → CosyVoice 2) are more robust.
  • Similarity and naturalness fall behind 2024-25 models trained on more data (CosyVoice 2, F5-TTS, Llasa, Llasa + Spark-TTS).
  • Uneven languages: quality tracks training data per language; low-resource languages (including Hindi) are weaker than English.

Check yourself

Q1Why does XTTS use a low-frame-rate VQ-VAE rather than EnCodec's 8 codebooks?

The GPT generates tokens one at a time, so its cost scales with tokens per second. One coarse codebook at around 20 Hz means far fewer steps than 75 Hz × 8 codebooks. The detail lost in coarse tokens is recovered by decoding the GPT's hidden states rather than the tokens.

Q2What does the Perceiver resampler do?

It turns a reference mel of any length into a fixed number of latent vectors via cross-attention from learned queries. That gives the GPT a fixed-size speaker and style prompt and lets you average several reference clips.

Q3Why can XTTS stream?

The decoder converts chunks of the GPT's hidden states to audio independently of the rest, so audio for the first chunk can be played while the GPT continues generating later tokens.

When you're back at a laptop

  • Install the community coqui-tts package, load XTTS-v2, and clone your voice from a 6 s clip in English and Hindi.
  • Measure time to first audio with streaming on your RTX 3050 and on CPU.
  • Try Hinglish and Indian-format numbers with and without your own text normalizer.
arXiv ↗PDF ↗