SoundStream

Why this paper exists

Classic speech codecs (Opus, EVS) are hand-designed signal processing: they model the vocal tract with linear prediction and quantize the parameters. They work well at moderate bitrates but sound muffled or robotic at very low ones (around 3-6 kbps), and they do not improve with data.

Earlier neural codecs either needed a heavy autoregressive decoder (a WaveNet-style vocoder fed with codec parameters) or were trained for a single bitrate. Two problems were open:

• Quality at low bitrate with a decoder cheap enough to run in real time on a phone.
• Quantizing a learned embedding with enough bits. A plain vector quantizer (VQ: replace a vector with its nearest entry in a learned table, the codebook) at 6 kbps and 75 frames/s needs 80 bits per frame, which means a codebook with 2^80 entries. That cannot exist.

SoundStream solves the second problem with RVQ, and the first with a GAN-trained conv decoder in the style of MelGAN and HiFi-GAN (MelGAN + Parallel WaveGAN, HiFi-GAN).

The core idea

Quantize in stages. Codebook 1 gives a coarse approximation of the encoder's output vector; codebook 2 quantizes what codebook 1 missed (the residual); codebook 3 quantizes what is still left; and so on. With 8 codebooks of 1,024 entries you get 8 × 10 = 80 bits per frame using only 8 × 1,024 stored vectors.

Then train with quantizer dropout: on each training example use only the first n codebooks, with n random. The model learns to make codebook 1 carry the most important information and each later codebook add detail, so at inference you can stop after any number of codebooks and get a lower bitrate from the same model.

How it works

waveform 24 kHz (24,000 samples/s)
      |
  conv ENCODER (strides 2,4,5,8 = x320)
      |
  z: 75 frames/s x D dims
      |
  RVQ: codebook 1 -> residual -> codebook 2 -> ...
      |      (keep first n_q codebooks)
  tokens: 75/s x n_q indices (each 0..1023)
      |
  z_hat = sum of chosen code vectors
      |
  conv DECODER (transposed convs, x320)
      |
waveform_hat 24 kHz
      |
  losses: multi-scale spectral recon
          + adversarial (wave + STFT discriminators)
          + feature matching

Step by step:

1. Encoder. A stack of 1D conv blocks downsamples by strides 2, 4, 5 and 8 (product 320). At 24 kHz that is 24,000 / 320 = 75 frames per second, each a D-dimensional vector z_t.
2. Residual vector quantization. For each frame:

r_0 = z_t
for i = 1..n_q:
    k_i = argmin_k || r_{i-1} - C_i[k] ||
    q_i = C_i[k_i]
    r_i = r_{i-1} - q_i
z_hat_t = q_1 + q_2 + ... + q_{n_q}

C_i is codebook i (1,024 vectors of size D), k_i is the token (an integer index) emitted by stage i, r_i is the residual left after stage i. The tokens k_1..k_{n_q} are what you transmit or what a language model predicts.

1. Training the codebooks. Codebooks are initialized with k-means on the first batch and updated with an exponential moving average of the vectors assigned to each entry; entries that stop being used are re-initialized from random frames of the current batch. Gradients pass through the argmin with the straight-through estimator: in the backward pass, pretend z_hat = z so the encoder still gets gradient.
2. Quantizer dropout. For each training example, sample n_q uniformly from 1 to the maximum (N_q) and use only the first n_q stages.
3. Decoder. Mirrors the encoder with transposed convs back to 24 kHz samples.
4. Losses.
   • A multi-scale spectral reconstruction loss: L1/L2 between mel spectrograms of input and output at several STFT window sizes.
   • Adversarial loss from two discriminators: a wave-based multi-resolution discriminator (raw and downsampled waveform, as in MelGAN) and an STFT-based discriminator that works on the complex spectrogram.
   • Feature matching: L1 between discriminator internal activations on real and generated audio.
5. Optional denoising. A conditioning flag (applied with FiLM, a learned per-channel scale and shift) tells the model whether to keep or remove background noise, so compression and enhancement happen in one model.
6. Streaming. All convolutions are causal (they only look at past samples), so the codec can encode and decode frame by frame with low latency.

The bitrate formula you should be able to write from memory:

bitrate = frame_rate x n_q x log2(codebook_size)
6 kbps  = 75 Hz      x 8   x 10 bits

Build the intuition

Analogy: RVQ is like paying with coins. The first codebook is a 500-rupee note that gets you close, the second a 50, the third a 5. Each stage pays off what is left. Quantizer dropout is training a cashier who has to give a sensible total even if you stop after the first one or two coins.

A tiny RVQ by hand

Use 2D vectors and three tiny codebooks of 4 entries each (2 bits per stage). The encoder outputs z = (5.3, -2.1).

The tokens for this frame are [2, 0, 0]. Reconstruction with all three: (5,-2) + (0.25,0) + (0.06,-0.06) = (5.31, -2.06). Stop after one codebook and you get (5, -2); after two, (5.25, -2.0). Each stage cuts the error by roughly 8x here. Notice that the later codebooks hold small vectors: they only ever see residuals, which is exactly why later tokens carry fine detail and why dropping them degrades quality gracefully.

The token budget

A codec-based TTS model has to generate these tokens, so frame rate × codebooks is its workload:

For comparison, the text of 10 s of speech is about 25 words, roughly 30-40 LLM text tokens. If you flatten all 8 codebooks into one sequence, the audio is ~150x longer than the text, and attention cost grows with the square of length. This is why VALL-E (VALL-E) generates only codebook 1 autoregressively, why Moshi (Moshi) uses a separate small depth transformer per frame, and why later codecs push the frame rate down.

What would happen if you trained without quantizer dropout and then used only 2 codebooks at inference? The decoder has never seen a z_hat built from just 2 codebooks, and the information is spread evenly across all 8, so the output would be badly distorted rather than just lower quality.

Numbers worth remembering

• 24 kHz input, encoder stride 320, so 75 frames/s.
• Codebooks of 1,024 entries (10 bits each); bitrates from 3 to 18 kbps with one model (18 kbps = 24 codebooks).
• 6 kbps = 75 × 8 × 10. Be able to derive this.
• Headline result: SoundStream at 3 kbps was rated better than Opus at 12 kbps in their listening tests, and close to EVS at a much higher bitrate (check the paper's MUSHRA figure for exact comparisons).
• Runs in real time on a single smartphone CPU core, with causal (streaming) convolutions.

Where it shows up today

• EnCodec (EnCodec) is SoundStream's design with better losses; DAC (DAC) improves codebook usage; Mimi in Moshi (Moshi) and SpeechTokenizer (SpeechTokenizer + SNAC) add semantic distillation to the first codebook.
• AudioLM (AudioLM) and SPEAR-TTS (SPEAR-TTS) use SoundStream tokens as their acoustic tokens; MusicLM does too.
• RVQ plus quantizer dropout is in nearly every codec used for TTS and speech LLMs: VALL-E (VALL-E), Sesame CSM (Sesame CSM), Kyutai TTS (Delayed Streams Modeling (TTS)).
• Google's Lyra v2 codec is built on SoundStream.

What it got wrong, and what came next

• Codebook collapse. Many codebook entries go unused, wasting bits. DAC (DAC) fixes this with factorized, L2-normalized lookups; CosyVoice 2 (CosyVoice → CosyVoice 2) and X-codec2 (Llasa + Spark-TTS) switch to finite scalar quantization.
• Too many tokens per second for language models (600 tokens/s at 6 kbps). Mimi drops to 12.5 Hz; SNAC (SpeechTokenizer + SNAC) uses coarse codebooks at lower frame rates; CALM (CALM (Pocket TTS)) drops quantization entirely and models continuous latents.
• Tokens mix content and voice. Nothing forces codebook 1 to carry what is said rather than how it sounds, which makes them hard for an LM to learn from text. AudioLM (AudioLM) adds separate semantic tokens; SpeechTokenizer and Mimi distill SSL features into codebook 1.
• Balancing many losses by hand. EnCodec (EnCodec) adds a loss balancer.

Check yourself

When you're back at a laptop

• Write RVQ in about 30 lines of PyTorch: a loop of nearest-neighbour lookups with torch.cdist, subtract the chosen vectors, sum them, and apply z + (z_hat - z).detach().
• Fit the codebooks with k-means on 10,000 random 64-dim vectors, then plot mean squared error against number of codebooks (1 to 8). You should see it fall at every stage.
• Add quantizer dropout to a toy autoencoder and compare the error curve with and without it when truncating codebooks.
• Compute the token budget for a 20 s voice-agent reply at 6 kbps and at Mimi's 12.5 Hz × 8.