TTS · Phase 5: Neural audio codecs
DAC
High-Fidelity Audio Compression with Improved RVQGAN
In one sentence
DAC (the Descript Audio Codec) is an improved SoundStream/EnCodec-style RVQ codec that fixes unused codebook entries with factorized, L2-normalized code lookups, adds Snake activations and better discriminators, and compresses 44.1 kHz audio to about 8 kbps for speech, music and general sound.
Why this paper exists
SoundStream (SoundStream) and EnCodec (EnCodec) quantize with residual vector quantization (RVQ: a cascade of codebooks, each quantizing what the previous ones missed). In practice many codebook entries are almost never chosen (codebook collapse), so a 10-bit codebook might effectively carry far fewer bits. Wasted bits mean lower quality at a given bitrate. Their quality at full sample rate (44.1 kHz) on music and general audio also had room to improve.
The core idea
Make every codebook entry useful. Instead of searching for the nearest code in the encoder's full high-dimensional space, project the residual down to a tiny space (8 dimensions in the paper), L2-normalize both the projection and the codes, and look up there; then project the chosen code back up. With normalization, nearest-neighbour search becomes cosine similarity, and the low dimension makes the codes spread out and get used. The paper calls this factorized codes, an idea borrowed from improved VQGAN work in images.
How it works
- Encoder / decoder: conv architecture like SoundStream, with Snake activations (
x + (1/α)·sin²(αx), a learnable periodic activation from BigVGAN, BigVGAN) that help model periodic signals like pitch. - RVQ with factorized codes: codebook size 1,024; lookups in a low-dimensional L2-normalized space; 9 codebooks for the 8 kbps 44.1 kHz model.
- Quantizer dropout, applied only part of the time (probability 0.5 in the paper). Always dropping codebooks, as SoundStream does, was found to hurt quality at full bitrate; dropping half the time keeps the bitrate flexibility and the full-bitrate quality.
- Discriminators: HiFi-GAN's multi-period discriminator (HiFi-GAN) plus a multi-band, multi-scale STFT discriminator (splitting the spectrogram into frequency bands so high frequencies get their own judgement).
- Losses: multi-scale mel L1, adversarial, feature matching, and the usual VQ codebook and commitment losses.
- Frame rate: 44,100 / 512 ≈ 86 frames/s. So at 9 codebooks the token budget is about 86 × 9 ≈ 775 tokens/s.
Build the intuition
Analogy: Codebook collapse is a restaurant menu with 1,024 dishes where customers only ever order 100. You pay to print 1,024, but you are really offering 100. Factorized, normalized lookups are like reorganizing the menu so that every dish is distinct and someone wants it.
Bits check: 86 frames/s × 9 codebooks × 10 bits ≈ 7.7 kbps, which is the "~8 kbps" headline. Raw 44.1 kHz 16-bit mono is about 706 kbps, so this is roughly a 90x compression.
What would happen if you used DAC tokens for a voice-agent TTS on your RTX 3050? At ~86 Hz × 9 codebooks it is an even heavier token budget than EnCodec at 6 kbps. That is fine for a model that predicts all codebooks per frame in parallel (Parler-TTS uses a delay pattern), but painful for anything that flattens the codebooks into one sequence.
Numbers worth remembering
- 44.1 kHz model at ~8 kbps, 9 codebooks of 1,024, ~86 frames/s.
- About 90x compression versus 16-bit PCM.
- Released models at 16, 24 and 44.1 kHz (check the repo for exact configurations).
- Reported better quality than EnCodec at the same bitrate on speech, music and environmental sound (check the paper's MUSHRA results).
Where it shows up today
- Parler-TTS (Parler-TTS) and Indic Parler-TTS generate DAC-style codec tokens (a DAC trained by the Parler team at 44.1 kHz).
- Its improved-RVQGAN recipe (Snake, factorized codes, multi-band STFT discriminator) is a standard starting point for later speech and music codecs (check each codec paper for what it keeps).
- A common baseline in every new codec paper (SNAC, SpeechTokenizer + SNAC; Mimi, Moshi).
What it got wrong, and what came next
- Still purely acoustic tokens with no semantic structure: SpeechTokenizer (SpeechTokenizer + SNAC) and Mimi distill SSL features into codebook 1.
- High token rate for language modelling: SNAC (SpeechTokenizer + SNAC) uses multi-rate codebooks; X-codec2 and BiCodec (Llasa + Spark-TTS) use one stream at 50 Hz.
- Quantization is still lossy: continuous-latent approaches like CALM (CALM (Pocket TTS)) avoid it.
Check yourself
Q1What is codebook collapse and why does it cost quality?
It is when most entries of a learned codebook are rarely or never chosen, so a 10-bit code effectively carries fewer bits. The bitrate is fixed by the number of tokens, so wasted entries mean less information per token and worse reconstruction at the same bitrate.
Q2How do factorized, L2-normalized codes help?
The residual is projected to a low-dimensional space and normalized before the nearest-neighbour search, so the search is over directions on a small sphere. Codes spread out and more of them get used. The chosen code is projected back up to the full dimension before decoding.
Q3Why does DAC apply quantizer dropout only with probability 0.5?
Always training with a random subset of codebooks means the model rarely trains with all of them, which hurt quality at the highest bitrate. Applying dropout half the time keeps the ability to decode from fewer codebooks while the other half of training optimizes the full-bitrate case.
Q4Compute DAC 44 kHz's token rate.
44,100 / 512 ≈ 86 frames/s, times 9 codebooks ≈ 775 tokens/s, each 10 bits, so ≈ 7.7 kbps.
When you're back at a laptop
pip install descript-audio-codec, encode a clip with the 44 kHz model, and print the code shape.- Decode with only the first 1, 2, 4 and 9 codebooks and listen.
- Count codebook usage: histogram the token indices over a few minutes of speech for EnCodec and DAC and compare how many of the 1,024 entries are used.