CosyVoice → CosyVoice 2

Why this paper exists

By 2024 codec-LM TTS (VALL-E, VALL-E) cloned voices well but had well-known problems:

• Acoustic tokens are hard targets. EnCodec-style tokens (EnCodec) mix content, speaker, prosody and noise. An LM predicting them from text has to model all of it, and errors show up as mispronunciations, skipped words and hallucinations.
• Unsupervised semantic tokens (HuBERT / w2v-BERT k-means, as in AudioLM AudioLM and SPEAR-TTS SPEAR-TTS) help, but they are not trained to align with text, so they still carry speaker and prosody leakage and can be ambiguous about content.
• Token-to-waveform quality. Decoding discrete tokens directly loses detail; you need a strong generative model to fill it in.
• Streaming. A voice agent needs audio before the sentence is done. Most high-quality zero-shot systems were offline.

The core idea

Use ASR supervision to define the tokens: put a vector quantizer in the middle of a speech recognition encoder and train the whole thing with the ASR loss. The tokens are then forced to carry whatever is needed to recognize the words, and little else. Predicting them from text is a cleaner, easier problem for an LLM.

Then let a different kind of model do the acoustics: flow matching (Flow Matching) generates a mel spectrogram conditioned on the tokens, the speaker embedding and a prompt mel, and a vocoder makes the waveform. LLM for content and prosody; flow matching for voice and acoustic detail.

How it works

text (BPE)            prompt audio (3-10 s)
   |                     |         |
   |              S3 tokenizer  speaker emb
   |              (ASR encoder   (x-vector)
   |               + VQ / FSQ)       |
   v                     v           |
  [ text LLM: text -> speech tokens ] (AR)
                  |
                  v  speech tokens (25-50 Hz)
  [ conditional flow matching ]  <- prompt mel,
     tokens -> mel (50 Hz)          speaker emb
                  |
                  v
  [ HiFi-GAN-family vocoder ] -> waveform

CosyVoice 1

1. Supervised semantic speech (S3) tokenizer. Take a multilingual ASR encoder (SenseVoice), insert a vector quantizer after the first few encoder layers, and train with the ASR loss. Codebook size 4,096; about 50 tokens per second (check the paper; a 25 Hz variant was also released). At inference you keep only the encoder part up to the quantizer.
2. Text-to-token LLM (~300M parameters). Sequence:

[start] [spk x-vector] [text tokens] [turn]
        [speech tokens ...] [end]

Trained with next-token cross-entropy on the speech tokens only:

L_LM = - Σ_t log p(μ_t | μ_<t, text, v)

μ_t is the speech token at position t, v the speaker embedding. For zero-shot cloning, the prompt's text and tokens are placed before the target, as in VALL-E.

1. Conditional flow matching (an optimal-transport CFM model, as in Matcha-TTS Matcha-TTS). Speech tokens are embedded and aligned to the mel frame rate; the model learns a velocity field from Gaussian noise x_0 to the target mel x_1:

x_t = (1 - t)·x_0 + t·x_1
L_FM = || v_θ(x_t, t | tokens, v, prompt_mel)
         - (x_1 - x_0) ||²

v_θ is the network's predicted velocity, t in [0, 1] is flow time, prompt_mel is the prompt's mel with the target region masked. At inference you integrate the ODE from noise in about 10 steps, using classifier-free guidance (mix the conditional and unconditional velocities) to strengthen conditioning.

1. Vocoder: mel → 22-24 kHz waveform (a HiFi-GAN-family model, HiFi-GAN; check the paper for the exact variant).
2. Instruction variant (CosyVoice-Instruct): fine-tuned with style instructions (emotion, speed, role) in the text input.

CosyVoice 2 changes

• FSQ tokens. Replace VQ with finite scalar quantization: project to a few dimensions, round each to a small set of integer levels, and the code is the tuple of levels. No learned codebook, so no dead entries: near 100% codebook utilization at 25 tokens/s (codebook size 6,561 = 3^8, check the paper).
• Pretrained text LLM backbone: the LLM is initialized from Qwen2.5-0.5B. The separate text encoder and the speaker embedding in the LLM are removed (the embedding leaked information and hurt cross-lingual use); voice comes from the prompt and the flow-matching stage.
• Bistream text-speech LM for streaming: in training, text and speech tokens are interleaved in a fixed ratio (5 text tokens, then 15 speech tokens, repeated), so at inference the LLM can consume text as it arrives from an upstream LLM and emit speech tokens in step.
• Chunk-aware causal flow matching: the flow-matching model is trained with several attention masks (non-causal, fully causal, chunk-M, chunk-2M), so the same model runs offline with full context or streaming with limited look-ahead. Tokens are upsampled 2x to the 50 Hz mel rate.
• Result: streaming quality close to offline, with low first-packet latency (~150 ms reported; check the paper).

Build the intuition

Analogy: the LLM is a script writer who also marks the pauses and emphasis; the flow-matching model is a voice actor who reads that marked-up script in a voice you show them. The script contains no information about the actor, which is why any voice can read it.

Why ASR supervision makes good tokens. Imagine two speakers, a Tamil-accented man and a Delhi woman, saying "your EMI is due". HuBERT k-means tokens differ quite a lot between them, because clusters also pick up speaker and accent. S3 tokens are trained so an ASR head can read the words from them; anything that does not help recognition is not rewarded, so the two token sequences end up more similar. The LLM's target is then mostly "what is said and roughly how it is paced", which it can learn from text.

Token budget, 10 s of speech:

Streaming arithmetic. At 25 tokens/s, a chunk of 15 speech tokens is 0.6 s of audio. The flow-matching model processes a chunk with its causal mask, the vocoder runs, and audio plays while the LLM produces the next chunk. Time to first audio = text for the first chunk + 15 LLM steps + one flow-matching chunk (a few ODE steps) + vocoder.

What would happen if you swapped the voice prompt but kept the same speech tokens? You would get the same words with the same rhythm in the new voice. That is the content / acoustics split working, and it is how CosyVoice does cross-lingual cloning.

Numbers worth remembering

• CosyVoice 1: S3 tokens from a SenseVoice ASR encoder with VQ, 4,096 codes, ~50 Hz; LLM ~300M; trained on ~170k hours of multilingual speech (check the paper).
• CosyVoice 2: FSQ, 25 Hz, near-full codebook use; LLM initialized from Qwen2.5-0.5B.
• Bistream ratio 5 text : 15 speech tokens.
• Flow matching to 50 Hz mel (tokens upsampled 2x in CosyVoice 2); about 10 ODE steps with classifier-free guidance.
• CosyVoice 2 reports results on Seed-TTS-eval (Seed-TTS) close to human recordings on content consistency and speaker similarity (check its tables).

Where it shows up today

• "LLM for tokens, flow matching for mel" is the dominant open-source TTS design of 2025-26. Chatterbox and several other open models use a CosyVoice-style token-to-mel stage (check each model card); CosyVoice 3 continues the line.
• Supervised semantic tokens: compare with SSL-distilled codec tokens in Mimi (Moshi) and SpeechTokenizer (SpeechTokenizer + SNAC), and single-codebook FSQ codecs in Llasa / Spark-TTS (Llasa + Spark-TTS).
• Flow-matching acoustic models: Matcha-TTS (Matcha-TTS), Voicebox (Voicebox), F5-TTS (F5-TTS).
• Streaming: compare CosyVoice 2's chunk-aware design with Kyutai's delayed streams (Delayed Streams Modeling (TTS)).

What it got wrong, and what came next

• Three models in a chain (LLM, flow matching, vocoder) means three places to add latency and three things to deploy. Single-model streaming designs: Kyutai TTS (Delayed Streams Modeling (TTS)), Sesame CSM (Sesame CSM), Pocket TTS (CALM (Pocket TTS)).
• Still autoregressive over tokens: occasional hallucinations or skips remain, though fewer than VALL-E. Seed-TTS (Seed-TTS) and later work use reinforcement learning on WER and similarity rewards to reduce them.
• The flow-matching stage needs several ODE steps per chunk. One-step generators (drifting, Drifting Models) and consistency heads (CALM, CALM (Pocket TTS)) aim to remove that.
• Language coverage is centred on Chinese and English (plus Japanese, Korean, Cantonese and some dialects); Indian languages need their own data (IndicVoices-R, IndicVoices-R).
• CosyVoice 1's speaker embedding in the LLM leaked information and hurt cross-lingual use; CosyVoice 2 removed it.

Check yourself

When you're back at a laptop

• Clone the CosyVoice repo and run CosyVoice2-0.5B zero-shot with a 5-10 s clip of your voice, in English and with a Hindi sentence (expect weaker Hindi).
• Print the speech token sequence for one sentence and confirm about 25 tokens per second.
• Compare streaming and non-streaming inference: measure time to first audio and listen for chunk-boundary artifacts.
• Swap prompts while keeping the same tokens to hear the content / voice split.
• Measure WER with Whisper and speaker similarity with a WavLM verification model (UTMOS for the metrics).