MELLE

Why this paper exists

VALL-E (VALL-E) made zero-shot TTS a language-modelling problem, but discrete codec tokens bring baggage:

• Lossy, coarse. A codec at a fixed bitrate throws away detail, and residual vector quantization (RVQ: several stacked codebooks per frame) means several tokens per frame, so VALL-E needed an extra non-autoregressive stage for the later codebooks.
• Sampling hacks. Token sampling needed tricks (top-k, repetition-aware sampling) to avoid loops.

Text LLMs need discrete tokens because text is discrete. Audio is not. Can an AR model predict the continuous frames directly?

The core idea

Replace "softmax over a codebook" with "predict a continuous vector". The decoder-only Transformer reads text tokens plus the previous mel frames and outputs the next mel frame (or several frames at once, a reduction factor). Pure regression on its own produces over-smoothed, averaged frames (the same mode-averaging issue as one-step flow matching, Flow Matching), so MELLE adds two fixes:

• Latent sampling module: the model predicts a mean and variance, and samples a latent from that Gaussian before decoding the frame (a VAE-style reparameterization with a KL term). This puts randomness back into each step, so outputs vary like sampled tokens do.
• Spectrogram flux loss: a term that rewards change between consecutive predicted frames, discouraging flat, repetitive, over-smoothed output.

How it works

Training loss, roughly:

L = L_reg (L1 + L2 on mel)
  + λ_KL · KL(predicted Gaussian || prior)
  + λ_flux · L_flux
  + L_stop (binary "end of speech" per frame)

The λ are weights; check the paper for the exact form of each term. A post-net refines the mel and a vocoder produces the waveform.

[text tokens][prompt mel frames][generated frames ...]
      -> decoder-only Transformer
      -> latent sampling (mean, var -> z)
      -> linear -> next mel frame(s) + stop flag

Build the intuition

Analogy: a discrete-token model is a painter limited to a 1024-colour palette, choosing one swatch per stroke. MELLE mixes any colour, but if it always picks the average colour the painting turns grey, so it jitters its choice (sampling) and is penalized for repeating the same stroke (flux loss).

What would happen without the latent sampling module? Given the same text and prompt, the model would produce the same, overly smooth output every time, losing natural variation in prosody.

Numbers worth remembering

• One stage: no separate NAR model for extra codebooks, unlike VALL-E.
• Reduction factor (frames per AR step) trades speed for quality; check the paper's ablation for values.
• The paper reports results competitive with VALL-E 2-style baselines on LibriSpeech zero-shot evaluation (check the tables for WER and SIM).

Where it shows up today

• The direct ancestor of the continuous autoregressive line: CALM (CALM (Pocket TTS)) also predicts continuous frames autoregressively, but of a VAE latent, with a consistency (few-step) head instead of a single Gaussian.
• NaturalSpeech 2 (NaturalSpeech 2 → NaturalSpeech 3) earlier argued for continuous latents, but used NAR diffusion.
• Pocket TTS (CALM (Pocket TTS), Drifting Models) is the practical descendant you can run on CPU.

What it got wrong, and what came next

• A Gaussian per step is a weak distribution. Real next-frame distributions can be multimodal; a single Gaussian plus regression still blurs. CALM uses a small generative head (consistency model) per step, and Pocket TTS swaps in a drifting-trained one-step head (Drifting Models).
• Mel + vocoder: two stages; CALM generates VAE latents decoded straight to waveform.
• Streaming of text input is not addressed; see DSM (Delayed Streams Modeling (TTS)).

Check yourself

When you're back at a laptop

• Train a tiny AR model on one speaker's mel frames (e.g. LJSpeech subset) with plain L2 loss and listen to the over-smoothing; then add a Gaussian sampling head and compare.