SpeechTokenizer + SNAC

SpeechTokenizer: Unified Speech Tokenizer for Speech Language Models; SNAC: Multi-Scale Neural Audio Codec

Zhang, Zhang, Li, Zhou, Qiu (SpeechTokenizer); Siuzdak, Grötschla, Lanzendörfer (SNAC) · Fudan University (SpeechTokenizer) / ETH Zurich (SNAC) · 2023

Skim12 minDay 8 of 10#62 of 86

In one sentence

Two fixes to "RVQ codec tokens are awkward for language models": SpeechTokenizer (2023) makes the first RVQ codebook carry content by distilling HuBERT into it, and SNAC (2024) runs its coarse codebooks at lower frame rates so there are fewer tokens per second, which is what Orpheus TTS generates.

Why this paper exists

By 2023 there were two kinds of speech tokens, each with a flaw:

Semantic tokens (k-means clusters of HuBERT or w2v-BERT features, HuBERT, data2vec / w2v-BERT / BEST-RQ) capture what is said and are easy to predict from text, but throw away the voice, so you cannot reconstruct audio from them alone.
Acoustic tokens (RVQ codes from SoundStream / EnCodec, SoundStream, EnCodec) reconstruct audio well, but every codebook mixes content, speaker and recording conditions, and there are many tokens per second.

AudioLM (AudioLM) used both, in a three-stage pipeline with two separate tokenizers. SpeechTokenizer asks: can one codec give you both in one hierarchy? SNAC asks a different question: why should every codebook run at the same frame rate when coarse information changes slowly?

The core idea

SpeechTokenizer: train an RVQ codec as usual, plus a loss that makes the output of the first quantizer (RVQ-1) match HuBERT's representations. Codebook 1 is pushed to hold the linguistic content; since later codebooks only quantize the residual, they end up holding what HuBERT does not care about: timbre, prosody detail, channel. Content and voice are separated by construction.

SNAC: make the RVQ multi-scale. Codebook 1 quantizes a heavily downsampled version of the residual (a few frames per second), codebook 2 a less downsampled one, codebook 3 the full frame rate. Coarse structure costs few tokens; only the finest level runs fast.

How it works

SpeechTokenizer

Architecture: EnCodec-like conv encoder/decoder (with a BiLSTM in place of the LSTM), 8 RVQ codebooks of 1,024 entries, 16 kHz audio at 50 frames/s.
Semantic distillation: a projection of RVQ-1's quantized output is trained to match a HuBERT layer's features (the paper uses a cosine-similarity loss per feature dimension; it also tries predicting HuBERT k-means labels). Check the paper for which HuBERT layer.
Usual codec losses otherwise: reconstruction, adversarial, feature matching, commitment.
The result: decode RVQ-1 alone and you get the right words in a flattened voice; decode RVQ-2..8 with someone else's RVQ-1 and you get that content in this voice (a simple voice conversion).
The paper also builds USLM, a VALL-E-style TTS (VALL-E): an AR model predicts RVQ-1 tokens from text, a NAR model fills RVQ-2..8 from a speaker prompt. Because RVQ-1 is content-only, the AR model has an easier job; they report better zero-shot results than VALL-E (check the paper's table).
It also proposes SLMTokBench, a benchmark for how well tokens align with text and preserve speech information.

SNAC

residual r
  |-- downsample x4 -> VQ (codebook 1) -> upsample
  |      subtract from r
  |-- downsample x2 -> VQ (codebook 2) -> upsample
  |      subtract from r
  '-- full rate     -> VQ (codebook 3)

The 24 kHz speech model has 3 codebooks at about 12, 23 and 47 Hz, codebook size 4,096 (12 bits), for about 0.98 kbps. Check: (11.7 + 23.4 + 46.9) × 12 ≈ 984 bits/s.
There are also 32 kHz and 44 kHz general-audio models with 4 codebooks.
Other choices (from the paper): noise blocks, depthwise convolutions and local windowed attention in the encoder/decoder.

Build the intuition

Analogy: a newspaper has headlines, paragraphs and fine print. SNAC writes the headline (codebook 1) once per page, the paragraph (codebook 2) twice, and the fine print (codebook 3) four times. EnCodec writes all three at the same rate, so most of what it repeats is redundant.

Token budget, per second of speech:

Codec	Layout	Tokens/s
EnCodec 6 kbps	8 × 75 Hz	600
SpeechTokenizer	8 × 50 Hz	400 (50 for RVQ-1)
SNAC 24 kHz	12 + 23 + 47 Hz	~82
Mimi	8 × 12.5 Hz	100

How Orpheus flattens SNAC. For each coarse frame (~12 per second) it writes 1 token from codebook 1, 2 from codebook 2 and 4 from codebook 3: 7 tokens per frame, about 84 tokens per second, all in one stream that a plain Llama can predict left to right (Llasa + Spark-TTS). A 3B model at ~84 tokens/s needs to decode faster than about 100 tokens/s to stay real time, which an LLM server like vLLM can do on a decent GPU but your 4 GB RTX 3050 would struggle with at full precision.

Numbers worth remembering

SpeechTokenizer: 16 kHz, 50 Hz, 8 codebooks × 1,024; codebook 1 distilled from HuBERT.
SNAC 24 kHz: 3 codebooks, ~12 / 23 / 47 Hz, 4,096 entries, ~0.98 kbps.
Orpheus: 7 SNAC tokens per ~12 Hz frame, ~84 tokens/s.

Where it shows up today

Mimi, the codec in Moshi (Moshi), uses SpeechTokenizer's idea with WavLM (WavLM) distillation into codebook 1, at 12.5 Hz. Sesame CSM (Sesame CSM) and Kyutai TTS (Delayed Streams Modeling (TTS)) use Mimi.
Later single-stream codecs such as X-codec2 and Spark-TTS's BiCodec (Llasa + Spark-TTS) also inject semantic (SSL) features into their tokens, a close relative of this idea.
Orpheus TTS (Canopy Labs) is a Llama fine-tuned to emit SNAC tokens (Llasa + Spark-TTS).

What it got wrong, and what came next

Distillation is imperfect: some speaker information still leaks into codebook 1 and some content into later ones. CosyVoice (CosyVoice → CosyVoice 2) trains tokens with ASR supervision instead; NaturalSpeech 3's FACodec (NaturalSpeech 2 → NaturalSpeech 3) factorizes further into content, prosody, timbre and detail.
Still several codebooks per frame for SpeechTokenizer, so TTS needs a VALL-E-like AR + NAR split. Single-codebook codecs (X-codec2, BiCodec, Llasa + Spark-TTS) remove the hierarchy.
SNAC's flattened stream is still long compared to text, and its frames are not aligned to anything linguistic. CALM (CALM (Pocket TTS)) and Pocket TTS sidestep tokens with continuous latents.

Check yourself

Q1What does SpeechTokenizer's first codebook carry, and how is that enforced?

Mostly linguistic content. A distillation loss pushes the RVQ-1 output to match HuBERT features, which are known to encode phonetic content and largely ignore speaker identity. Because later codebooks only quantize the residual, they pick up what remains: timbre and acoustic detail.

Q2Why does a content-only first codebook make VALL-E-style TTS easier?

The AR model maps text to codebook-1 tokens. If those tokens are content-like, the mapping is closer to a pronunciation model and has less to do with voice or noise, so it is easier to learn and less error-prone. Voice comes in later through the NAR stage and the prompt.

Q3How does SNAC reduce tokens per second without lowering the finest time resolution?

Only the last codebook runs at the full frame rate (~47 Hz); the coarse codebooks quantize downsampled residuals at ~23 and ~12 Hz. Slowly changing information is coded once over several frames instead of being repeated, for about 82 tokens/s total instead of 3 × 47 = 141.

Q4If you wanted to serve a SNAC-based TTS with an ordinary LLM stack, how would you lay out the tokens?

Flatten each coarse frame into a fixed pattern (1 coarse, 2 medium, 4 fine tokens = 7 tokens), add the codec tokens to the LLM's vocabulary, and train the model to emit them after the text. The decoder can start producing audio after a few frames, so you can stream. This is exactly what Orpheus does.

When you're back at a laptop

Load SpeechTokenizer from its repo; encode two speakers, swap their RVQ-1 tokens and decode. Listen for content vs voice.
pip install snac; encode 10 s of speech with hubertsiuzdak/snac_24khz and print the three code tensors' lengths (they should be in a 1:2:4 ratio).
Write the Orpheus-style 7-token flattening and its inverse, and check round-trip decoding.

SpeechTokenizer arXiv ↗SNAC arXiv ↗