TTS · Phase 5: Neural audio codecs
EnCodec
EnCodec: High Fidelity Neural Audio Compression
In one sentence
EnCodec is Meta's open SoundStream-style codec (conv + LSTM encoder, RVQ, conv decoder) with a multi-scale STFT discriminator and a loss balancer, giving 24 kHz audio at 1.5-24 kbps from 75 frames/s × 2-32 codebooks, and its tokens are what VALL-E and the first wave of codec-LM TTS were built on.
Why this paper exists
SoundStream (SoundStream) showed that an end-to-end neural codec with residual vector quantization (RVQ: quantize, then quantize the leftover, repeatedly) beats classic codecs at low bitrate. But it was not released, and training it is fiddly:
- Several losses (time-domain, spectral, adversarial, feature matching) with very different gradient scales need hand-tuned weights, and the right weights change during training.
- Quality on music and general audio, and at high sample rates (48 kHz stereo), was less explored.
- Bits could still be saved: RVQ tokens are not uniformly distributed, so plain fixed-length codes waste bits.
EnCodec is the engineering paper that made a neural codec robust, general (speech, music, noisy audio) and open source, which is why so many later systems use it.
The core idea
Keep SoundStream's encoder → RVQ → decoder design, and fix the training:
- A multi-scale STFT discriminator (MS-STFTD) that judges the complex spectrogram at several resolutions, giving the GAN a good view of both harmonics and transients.
- A loss balancer so each loss contributes a fixed fraction of the total gradient, regardless of its raw scale. Loss weights then mean "share of the gradient" and stop needing retuning.
- An optional small Transformer language model over the tokens plus arithmetic coding, which squeezes out extra bits losslessly.
How it works
audio 24 kHz mono
|
ENCODER: 1D convs, strides 2,4,5,8 (x320)
+ 2-layer LSTM
|
z: 75 frames/s x 128 dims
|
RVQ: up to 32 codebooks x 1024 entries
(bandwidth picks how many are used)
|
tokens [75/s x n_q] --(optional)--> small
| Transformer LM +
| arithmetic coding
DECODER: mirror of encoder (transposed convs)
|
audio_hat 24 kHz
|
losses: L1 waveform + L1/L2 multi-scale mel
+ MS-STFT adversarial + feature matching
+ VQ commitment, combined by the balancerBandwidths. Each codebook adds 75 × 10 = 750 bits/s:
| Bandwidth | Codebooks | Tokens/s |
|---|---|---|
| 1.5 kbps | 2 | 150 |
| 3 kbps | 4 | 300 |
| 6 kbps | 8 | 600 |
| 12 kbps | 16 | 1,200 |
| 24 kbps | 32 | 2,400 |
Like SoundStream, a random bandwidth is chosen per training batch so one model serves all of them. There is also a 48 kHz stereo model for music.
The loss balancer. Each loss l_i has a gradient g_i = ∂l_i/∂x_hat with respect to the decoder's output x_hat. Instead of summing λ_i · g_i, the balancer rescales:
g_total = Σ_i R · (λ_i / Σ_j λ_j) · g_i / <||g_i||><||g_i||> is an exponential moving average of the norm of g_i, R is a reference norm, and λ_i is the desired share. So a loss with λ_i / Σλ = 0.3 provides about 30% of the gradient magnitude, whether the loss is measured in decibels or in raw sample units. (The balancer applies only to losses that depend on the output; the VQ commitment loss is added normally.)
Discriminator. The MS-STFTD runs the same small 2D conv network on the real and imaginary parts of STFTs at several window sizes (from short windows that resolve timing to long windows that resolve pitch). It replaces SoundStream's pair of discriminators.
Entropy coding. A small Transformer predicts the distribution of each frame's tokens from the past; arithmetic coding then spends fewer bits on likely tokens. The paper reports roughly 25-40% extra compression from this (check the paper for the exact figure); TTS users ignore it because they use the token indices directly.
Build the intuition
Analogy: The loss balancer is like a manager who gives each reviewer a fixed share of speaking time, whether they talk loudly or quietly. Without it, the reviewer with the loudest voice (the loss with the largest gradient) dominates the meeting.
Token counting for TTS. VALL-E uses EnCodec at 6 kbps: 8 codebooks × 75 Hz. A 3-second voice prompt is 225 frames, so 225 tokens in codebook 1 and 1,800 tokens across all 8. A 10-second output is 750 frames. If one model generated all 8 codebooks in a flat sequence it would emit 6,000 tokens for 10 s; VALL-E instead generates only the 750 first-codebook tokens one by one and fills in the other 7 codebooks with 7 parallel passes (VALL-E).
What would happen if you encoded speech at 1.5 kbps (2 codebooks) and decoded it? Words stay intelligible but the voice gets rough and the speaker's identity drifts. That tells you where information lives: the first few codebooks carry content and coarse spectrum, later ones carry timbre detail, breathiness and room tone.
Numbers worth remembering
- 24 kHz, stride 320 → 75 frames/s; codebooks of 1,024 entries (10 bits).
- Bandwidths 1.5 / 3 / 6 / 12 / 24 kbps = 2 / 4 / 8 / 16 / 32 codebooks.
- 6 kbps = 8 codebooks is the VALL-E setting: 600 tokens/s.
- A 48 kHz stereo variant exists for music.
- Evaluated with MUSHRA listening tests against Opus, EVS and Lyra v2; EnCodec at low bitrates was preferred over these baselines at similar or higher bitrates (check the paper's tables for exact scores).
Where it shows up today
- VALL-E (VALL-E) and its follow-ups (VALL-E X, VALL-E 2) model EnCodec tokens.
- Meta's AudioCraft: MusicGen and AudioGen generate EnCodec tokens (a 32 kHz music EnCodec), using a "delay pattern" to interleave codebooks, the same idea Kyutai later uses in delayed streams (Delayed Streams Modeling (TTS)).
- Vocos (iSTFTNet + Vocos) can replace EnCodec's decoder with a faster iSTFT-based one that decodes the same tokens.
- Its MS-STFT discriminator is reused in later codecs: DAC (DAC) uses a multi-band version of it, and Mimi in Moshi (Moshi) builds on EnCodec-style training (check the Moshi paper for which losses it keeps).
What it got wrong, and what came next
- Token rate is high. 75 Hz × 8 codebooks is a heavy LM workload. Mimi (12.5 Hz), SNAC (SpeechTokenizer + SNAC) and X-codec2 (Llasa + Spark-TTS, one codebook at 50 Hz) cut it down.
- Codebook usage is uneven. DAC (DAC) gets much better utilization with factorized, L2-normalized codes and reaches 44.1 kHz at ~8 kbps.
- Purely acoustic tokens. Codebook 1 is not forced to carry linguistic content, so text-to-token mapping is harder and voice and content are entangled. SpeechTokenizer (SpeechTokenizer + SNAC) and Mimi distill HuBERT / WavLM features into codebook 1; CosyVoice (CosyVoice → CosyVoice 2) uses ASR-supervised tokens instead.
- Discrete is lossy. NaturalSpeech 2 (NaturalSpeech 2 → NaturalSpeech 3) and CALM (CALM (Pocket TTS)) model continuous latents instead.
Check yourself
Q1How many tokens per second does EnCodec produce at 6 kbps, and why?
- The frame rate is 24,000 / 320 = 75 Hz, and 6 kbps / 75 Hz = 80 bits per frame, which is 8 codebooks of 10 bits (1,024 entries). So 75 × 8 = 600 token indices per second.
Q2What problem does the loss balancer solve?
Losses like waveform L1, mel loss and adversarial loss have gradients of very different and changing magnitudes, so fixed weights make one loss dominate. The balancer normalizes each loss's gradient by a running average of its norm and rescales it so each loss contributes a chosen fraction of the total. Weights become interpretable and stop needing retuning when you add or change a loss.
Q3Why does EnCodec have an LSTM in the encoder and decoder?
The convolutions see only a local window; a small LSTM over the frame sequence adds longer temporal context cheaply, while staying causal for streaming. It is a modest addition on top of SoundStream's all-conv design.
Q4A TTS team wants fewer tokens per second. Why not just use EnCodec at 1.5 kbps?
At 2 codebooks the audio quality and speaker similarity drop noticeably, because EnCodec's fine detail lives in later codebooks. The frame rate is still 75 Hz, so you also still have 75 sequential AR steps per second. Codecs designed for LMs instead lower the frame rate (Mimi at 12.5 Hz) or put more information in one codebook (X-codec2).
Q5What does the entropy-coding Transformer add, and why do TTS models ignore it?
It predicts the next tokens' distribution so arithmetic coding can store them in fewer bits, a lossless saving of a few tens of percent. TTS models generate and consume the token indices directly, never the compressed bitstream, so the saving is irrelevant to them.
When you're back at a laptop
pip install encodec, loadEncodecModel.encodec_model_24khz(), and callmodel.set_target_bandwidth(b)for b in 1.5, 3, 6, 12, 24.- Encode a 10 s recording of your own voice; print the code tensor shape (it should be
[1, n_q, 750]) and confirm tokens/s = 75 × n_q. - Listen to each bandwidth. Note at which point your voice stops sounding like you; try a Hindi or Hinglish clip too.
- Bonus: decode the same 6 kbps tokens with a Vocos EnCodec decoder and compare speed on CPU.