Voicebox

Why this paper exists

In early 2023 the exciting zero-shot TTS result was VALL-E (VALL-E): give a 3-second prompt and it continues in that voice. But codec language models had real problems:

• Autoregressive and slow. They generate codec tokens one step at a time, so latency grows with utterance length.
• Only continuation. A left-to-right model can extend audio but cannot use context on both sides, so editing a word in the middle of a sentence is awkward.
• Fragile. Sampling from a token LM can skip or repeat words, giving high word error rate (WER).

Separately, models trained on clean read speech (FastSpeech 2, FastSpeech 2) did not scale to messy, diverse data. Meta wanted the "train one big model on lots of data, prompt it for many tasks" recipe that worked for text LLMs, but for speech and non-autoregressive.

The core idea

Train on a single task: text-guided speech infilling. Take a real utterance, hide a random contiguous chunk of its mel spectrogram, and ask the model to regenerate the hidden part given (a) the full frame-aligned phone sequence and (b) the unmasked audio around it. The generator is a flow-matching model (Flow Matching), so it generates all frames in parallel in a small number of steps.

Every downstream task is then a choice of what to mask:

• Zero-shot TTS: the prompt audio is the context, mask everything after it, give the prompt's transcript plus the new text.
• Editing: mask one word's frames and change that word in the transcript.
• Denoising: mask the noisy segment and regenerate it clean from context.
• Cross-lingual transfer: prompt in one language, text in another (multilingual model).

How it works

Voicebox has two models, both trained with flow matching and both doing infilling.

transcript -> phones
     |
 [duration model]  infills durations for masked phones
     |
 frame-level phones (each phone repeated d times)
     |
     v
 [audio model]  <- masked mel context (zeros where masked)
     |          <- noisy mel x_t, time t
     v
 predicted velocity -> ODE steps -> full mel
     |
 [vocoder] -> waveform

Audio model

Inputs at each frame (concatenated, then projected):

• x_t: the noisy mel at flow time t (80-dim log mel per frame).
• x_ctx = (1 − m) ⊙ x: the real mel with masked frames zeroed. m is the binary mask (1 = hidden), ⊙ is elementwise product.
• y: the phone embedding for that frame, from forced-alignment durations.

The backbone is a Transformer with U-Net-style skip connections (layer i linked to layer N−i+1). The paper's main model has 24 layers, dim 1024 (~330M parameters; check the paper).

Loss: the flow-matching regression, applied only to masked frames:

u = x_1 − (1−σ)·x_0
L = E || m ⊙ ( v_θ(x_t, x_ctx, y, t) − u ) ||²

x_1 is the real mel, x_0 noise, σ a tiny constant. Training masks a large contiguous span most of the time (on the order of 70-100% of frames; check the paper), so the model learns both "continue from a prompt" and "fill a gap".

Duration model

A second, smaller model infills phone durations the same way (masked phones, context durations, phone sequence). The paper compares a flow-matching duration model against a simple regression one. Durations matter: they set rhythm and total length.

Inference

1. Convert text to phones; get durations for the prompt by forced alignment and for the new text from the duration model.
2. Build the frame-level phone sequence and the context mel (prompt frames real, new frames zero).
3. Solve the ODE from noise with the learned velocity. Classifier-free guidance (CFG: run the model with and without conditioning and extrapolate the difference) sharpens adherence to text and prompt.
4. Vocode the mel (HiFi-GAN in the paper, HiFi-GAN).

Build the intuition

Analogy: Voicebox is a photo inpainting tool for audio. Erase a rectangle of a photo and the tool repaints it to match the lighting and style around it. Erase "Tuesday" from a recording, type "Thursday", and Voicebox repaints those frames in the speaker's voice, room and mic.

Why one task covers so many: a speaker's identity, recording channel and speaking style are all present in the unmasked audio. The phone sequence says what to say. Infilling forces the model to combine both, which is exactly zero-shot cloning. Denoising falls out because a mostly-clean context plus a transcript is enough to regenerate the noisy part clean.

What would happen if you trained without masking (always generating the full utterance from text alone)? You would get an ordinary multi-speaker TTS: no way to inject a new voice at inference, because the model never learned to copy style from context.

What would happen if the forced-alignment durations were wrong for an accent (say, Indian English in a model trained on US audiobooks)? The audio model would still try to fit the phones into the frames it was given, giving odd rhythm. That dependence on an external aligner and phone set is exactly what E2 TTS removes (E2 TTS).

Numbers worth remembering

• Training data: ~60k hours of English audiobooks for the English model, plus a multilingual model on ~50k hours across 6 languages (check the paper's data section).
• Main audio model: ~330M parameters, 24-layer Transformer (approximate; check the paper).
• Abstract headline vs VALL-E on zero-shot TTS: WER ~1.9% vs ~5.9%, higher speaker similarity, and up to ~20× faster.
• Non-autoregressive: inference cost is set by the NFE, not by utterance length in steps.

Where it shows up today

• E2 TTS (E2 TTS) and F5-TTS (F5-TTS) keep the infilling + flow matching recipe and drop phones and durations. IndicF5 inherits it for Indian languages.
• The "mask a span, condition on context" training is now standard for prompt-based cloning in NAR models.
• CosyVoice's flow-matching acoustic stage conditions on a prompt mel in a similar way (CosyVoice → CosyVoice 2).
• Meta did not release Voicebox weights, citing misuse risk, and trained a classifier to detect its output: an early example of the safety concerns now attached to every cloning model.

What it got wrong, and what came next

• Needs phones and an aligner. Grapheme-to-phoneme tools and forced alignment are language-specific and brittle for code-mixed text like Hinglish. E2 TTS pads characters instead (E2 TTS).
• Separate duration model. Adds a component and error source; E2/F5 only need a total duration.
• Mel + vocoder. Two stages; later systems work on codec or VAE latents (NaturalSpeech 2/3 NaturalSpeech 2 → NaturalSpeech 3, CALM CALM (Pocket TTS)).
• Not streaming. It generates the whole utterance at once. Streaming designs: CosyVoice 2 (CosyVoice → CosyVoice 2), DSM (Delayed Streams Modeling (TTS)).
• Many ODE steps. Fine on a GPU, heavy on CPU. Sway sampling (F5-TTS) and one-step methods (Drifting Models) chip at this.

Check yourself

When you're back at a laptop

• No official weights, so learn the recipe through F5-TTS (F5-TTS): its training code implements the same masked-infilling flow-matching loss.
• Read the F5-TTS loss code and find the mask; confirm loss is computed only on masked frames.
• Try editing: take a 5 s clip of yourself, mask one word's frames, change the text, and infill with F5-TTS's editing script (if your version has one; otherwise do it by hand with the inference API).