Llasa + Spark-TTS

Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesis; Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokens

Ye, Zhu, Chan, et al. (Llasa); Wang, Jiang, Ma, et al. (Spark-TTS) · HKUST and collaborators · 2025

Skim12 minDay 8 of 10#69 of 86

In one sentence

Llasa and Spark-TTS make TTS a plain text LLM emitting one stream of speech tokens (no RVQ hierarchy, no separate acoustic model stage): Llasa puts a Llama over a single-codebook codec and studies scaling train-time and inference-time compute, and Spark-TTS puts a Qwen2.5 over a codec that splits speech into global (speaker) tokens and semantic tokens for controllable voices; Orpheus TTS is the same idea with SNAC tokens.

Why this paper exists

Earlier LLM-based TTS needed extra machinery that standard LLM tooling does not support:

Multiple codebooks per frame (VALL-E's AR + NAR split, VALL-E; Moshi-style depth decoders, Moshi).
Multi-stage pipelines (CosyVoice's LLM, then a flow-matching model, then a vocoder, CosyVoice → CosyVoice 2).

Each extra stage means custom training code, custom serving and more latency. Meanwhile text LLMs had well-understood scaling behaviour and mature serving stacks (vLLM, SGLang, quantized inference). The question both papers ask: if speech is just one token stream, can you reuse all of that unchanged?

The core idea

Design the codec so that one token per frame is enough, then treat TTS as ordinary next-token prediction: text tokens in, speech tokens out, with the speech codes added to the LLM's vocabulary. The codec's decoder turns the tokens back into audio.

Llasa: a single-layer vector quantizer codec (X-codec2) plus a standard Llama. Then ask LLM-style questions: does more training compute help (yes: more natural speech and better prosody), and can inference-time compute help, by sampling many candidates and searching with speech understanding models as verifiers (yes: better emotion, timbre consistency and content accuracy, per the abstract)?
Spark-TTS: a codec (BiCodec) whose single stream has two parts: a fixed-length set of global tokens for speaker attributes, and low-bitrate semantic tokens for content. On a Qwen2.5 backbone with chain-of-thought-style generation, it can produce a voice from attribute labels (gender, pitch level, speed) rather than only cloning a reference.

How it works

Llasa

[text tokens] -> Llama (1B / 3B / 8B)
                    |
             [X-codec2 tokens]
                    |
       X-codec2 decoder -> audio

X-codec2 encodes 16 kHz audio into one token stream at about 50 tokens/s from a large single codebook (65,536 entries via FSQ, check the paper), fusing semantic SSL features with acoustic features before quantization.
Zero-shot cloning works the usual way: prompt text + prompt speech tokens as a prefix.
Inference-time search: generate several candidates (or do beam-style search) and score them with verifiers such as a speaker-similarity model, an ASR model (content accuracy) and an emotion recognizer; keep the best. Same idea as best-of-N with a reward model for text LLMs.
Released 1B, 3B and 8B checkpoints, training code and the codec.

Spark-TTS

text + optional attributes (gender, pitch, speed)
   -> Qwen2.5-0.5B
   -> [attribute tokens] [fine pitch / speed values]
   -> [global tokens: speaker]  (fixed count)
   -> [semantic tokens: content] (~50/s)
   -> BiCodec decoder -> audio

BiCodec: semantic tokens carry linguistic content at low bitrate; a fixed number of global tokens (independent of utterance length) carry speaker identity and style. Check the paper for the exact counts and rates.
Chain-of-thought generation: the model first writes coarse attribute labels, then fine-grained values, then global tokens, then semantic tokens, so you can control the voice at any of these levels or skip them by supplying a reference clip's global tokens.
VoxBox: a released 100,000-hour dataset with attribute annotations for controllable TTS research.

Orpheus (no paper)

Canopy Labs fine-tuned Llama-3.2-3B to emit SNAC tokens (SpeechTokenizer + SNAC), flattened 7 per coarse frame (about 84 tokens/s). Same philosophy: an unmodified LLM plus a codec decoder, served with vLLM.

Build the intuition

Analogy: instead of building a new kind of printing press for audio, these papers redesign the alphabet. Once speech is written as one line of letters, any existing press (LLM training and serving stack) can print it.

Why one stream matters for serving. With a single token per step, the whole TTS model is a standard decoder-only LLM: you get KV caching, continuous batching, paged attention, speculative decoding and 4-bit quantization for free. For a voice agent you can even serve the text LLM and the TTS LLM from the same vLLM cluster.

Real-time budget. At 50 tokens/s (Llasa, Spark-TTS) the model must decode faster than 50 tokens/s per stream to be real time; at Orpheus's ~84 tokens/s, faster than 84. A 0.5B Qwen does that easily on a modest GPU and plausibly on your RTX 3050; an 8B Llasa needs a much bigger GPU or heavy quantization.

What would happen if you used Llasa's verifier search in a voice agent? Each extra candidate multiplies the compute, and you cannot score a candidate until it is (at least partly) generated, so it adds latency. It suits offline generation (audiobooks, dubbing) more than live calls.

Numbers worth remembering

Llasa: 1B / 3B / 8B Llama models; X-codec2 single codebook, ~50 tokens/s (check the paper for codebook size and training hours).
Spark-TTS: Qwen2.5-0.5B backbone; BiCodec global + semantic tokens; VoxBox 100k hours.
Orpheus: Llama-3.2-3B + SNAC, ~84 tokens/s.
Both papers report results on Seed-TTS-eval (Seed-TTS); check their tables.

Where it shows up today

Orpheus, Llasa and Spark-TTS are popular open models because they drop into LLM tooling; many community fine-tunes (including Indian-language ones) follow this recipe.
Verifier-guided sampling ties TTS to the reward-model ideas used in Seed-TTS's RL (Seed-TTS).
Global-plus-content token designs relate to the speaker / content split in SpeechTokenizer (SpeechTokenizer + SNAC) and FACodec (NaturalSpeech 2 → NaturalSpeech 3).

What it got wrong, and what came next

One codebook is lossy: single-stream codecs at 50 Hz trade some fidelity for simplicity. Continuous-latent models (CALM / Pocket TTS, CALM (Pocket TTS)) avoid quantization.
Still AR token generation, so hallucinations and skips are possible, and long outputs are slow at 50-84 sequential steps per second.
Streaming needs care: the model reads the whole text before speaking; Kyutai's delayed streams (Delayed Streams Modeling (TTS)) and CosyVoice 2's bistream design (CosyVoice → CosyVoice 2) handle text that arrives word by word.
Large models (3B-8B) are heavy for CPU or a 4 GB GPU.

Check yourself

Q1What problem does a single-codebook codec solve for LLM-based TTS?

With one token per frame, generation is ordinary next-token prediction over one sequence, so a standard LLM architecture, training code and serving stack (vLLM, quantization, batching) work unchanged. Multi-codebook codecs need an extra NAR stage or a depth decoder.

Q2What is inference-time compute in Llasa?

Spending more compute at generation time by producing several candidates and choosing among them with verifiers: speech understanding models that score speaker similarity, content accuracy (ASR) and emotion. It is the TTS analogue of best-of-N sampling with a reward model.

Q3Why does Spark-TTS use a fixed number of global tokens?

Speaker identity and style do not change with utterance length, so a fixed-size set of tokens can describe them once per utterance. This separates "who is speaking" from "what is said", and lets the model generate a new voice from attributes or copy one from a reference clip by reusing its global tokens.

Q4Orpheus emits about 84 tokens per second of audio. What decode speed does real-time streaming need?

More than about 84 tokens/s per stream, plus headroom for the SNAC decoder and network. With batching, total throughput must exceed 84 times the number of concurrent calls.

When you're back at a laptop

Run Spark-TTS-0.5B on your RTX 3050: clone your voice, then generate a voice from attributes only (female, low pitch, slow).
Serve Orpheus or Llasa-1B with vLLM, measure tokens/s and time to first audio, and check it against the real-time threshold.
Implement best-of-4 sampling with Whisper WER as the verifier and measure how often it fixes a skipped word.

Llasa arXiv ↗Spark-TTS arXiv ↗