Reading Room

TTS · Phase 8: Continuous and streaming TTS

Delayed Streams Modeling (TTS)

Streaming Sequence-to-Sequence Learning with Delayed Streams Modeling

Zeghidour, Kharitonov, Orsini, Volhejn, de Marmiesse, Grave, Pérez, Mazaré, Défossez · Kyutai · 2025

Deep read16 minDay 10 of 10#80 of 86

In one sentence

Delayed Streams Modeling (DSM) puts text and audio on parallel, time-aligned streams in one decoder-only model and shifts one stream by a fixed delay; delay the audio behind the text and you get a streaming TTS (Kyutai TTS) that starts speaking after reading only a few words ahead and handles arbitrarily long text.

Source note: this page uses the arXiv abstract plus the reading guides (TTS paper 45, ASR paper 31). Details beyond them are marked as interpretation; check the paper for delay values, model sizes and results. The ASR side is covered in Kyutai DSM.

Why this paper exists

A voice agent runs ASR → LLM → TTS. The guide's latency target: time to first audio (TTFA) of a couple of hundred milliseconds. The LLM emits text word by word. Two common TTS designs fail this:

  • Offline (sentence-level) TTS such as F5-TTS (F5-TTS) or Kokoro needs the whole sentence before it starts, so the agent waits for the LLM to finish the sentence and for synthesis. That adds a full sentence of delay.
  • Learned read/write policies. The abstract describes the classic streaming seq2seq alternative: learn a policy that decides when to read more input and when to write output. That is a hard extra learning problem.

DSM avoids both.

The core idea

From the abstract: instead of learning when to read and write, DSM models already time-aligned streams with a decoder-only language model. Alignment moves into preprocessing, and delays between streams give streaming inference.

Given a text stream and an audio stream on the same clock:

  • Delay the text behind the audio: the model hears audio before it writes the words, so it is ASR (Kyutai DSM).
  • Delay the audio behind the text: the model reads words before it speaks them, so it is TTS.

Same architecture, roles swapped.

How it works

Time-aligned streams

Kyutai's models run on the Mimi codec from Moshi (Moshi): 12.5 frames per second with residual vector quantization (several codebook tokens per frame). At each frame the model handles one text slot plus that frame's audio tokens.

Training data is aligned in preprocessing: each word is placed in the text stream at the frame where it is spoken (interpretation: word timestamps from an ASR aligner; check the paper for the exact pipeline). Frames with no new word get a padding token.

frame:   1   2   3   4   5   6   7   8
text:   Hi  _   how _   are _   you _
audio:  .   .   .   .   .   .   .   .   (real timing)

TTS: shift audio right by D frames (D=2 here)
text:   Hi  _   how _   are _   you _
audio:  -   -   a1  a2  a3  a4  a5  a6

_ is padding, - is "no audio yet", a_k the audio tokens for frame k of the original speech. With the shift, when the model generates audio for "how", it has already seen the text up to D frames later.

Generation

A decoder-only Transformer models the joint sequence frame by frame, as Moshi does: a large temporal Transformer across frames, and a small per-frame component for the RVQ codebooks (Moshi). For TTS, the text stream is fed from outside (the LLM's output) and the audio stream is sampled.

LLM words --> text stream ---------+
                                   v
               [decoder-only Transformer]
                 per frame (12.5 Hz)
                                   |
               audio tokens, delayed by D
                                   v
                 [Mimi decoder] -> waveform

The latency knob

The delay D is the look-ahead. Larger D: the model sees more upcoming words, so prosody can anticipate the sentence (questions, emphasis), but first audio waits longer. Smaller D: faster start, less context. TTFA is roughly D frames of text plus model compute (interpretation). Because each frame only attends to the past, the model has no fixed maximum length: the abstract says it supports arbitrarily long sequences.

Interpretation: at inference the LLM's words do not arrive with timestamps, so the system must decide in which frame to insert the next word. Check the paper for how Kyutai TTS does this (for example whether the model itself signals when it is ready for the next word).

Build the intuition

Analogy: a news anchor reading a teleprompter. The words scroll a little ahead of what she is saying; she never needs the whole script, only the next few words, and the script can be written live while she speaks. The gap between the scroll and her voice is the delay D.

A worked latency comparison (illustrative numbers, not from the paper): the LLM streams 5 words per second, and the sentence has 15 words.

  • Offline TTS waits ~3 s for the sentence, then synthesizes: first audio after 3 s or more.
  • DSM with a look-ahead of ~2 words: first audio after ~0.4 s plus compute, then speech continues as words arrive.

What would happen if D = 0? The model must speak a word at the same frame it first sees it, with no idea what follows. Pronunciation of words that depend on context and sentence-level intonation (rising for a question) would suffer.

What would happen if the LLM stalls mid-sentence? The text stream fills with padding; the model can only keep speaking up to what it has read, then must pause naturally (interpretation: behaviour on stalls depends on how inference handles text insertion).

Numbers worth remembering

  • Codec: Mimi, 12.5 frames/s, RVQ (from Moshi, Moshi).
  • One architecture, two tasks: text delayed = ASR, audio delayed = TTS (abstract).
  • Abstract: state-of-the-art performance and latency, supports arbitrarily long sequences, competitive even with offline baselines (check the TTS tables for WER, speaker similarity and latency).
  • Code, samples and demos: github.com/kyutai-labs/delayed-streams-modeling (abstract).

Where it shows up today

  • Kyutai TTS and Kyutai STT (guide), and the ASR side in Kyutai DSM.
  • Builds on Moshi's multi-stream, Mimi-token design (Moshi); Moshi's "inner monologue" is already a time-aligned text stream.
  • Pocket TTS (CALM (Pocket TTS), Drifting Models) is Kyutai's small CPU model; check its card for how it streams text.
  • The alternative streaming design is CosyVoice 2's chunk-aware flow matching (CosyVoice → CosyVoice 2).

What it got wrong, and what came next

Interpretation; check the paper's limitations section:

  • Needs aligned training data. Word timestamps come from preprocessing, so alignment errors become training errors. For Indian languages and code-mixed speech, good aligners are scarcer.
  • Fixed delay is a compromise. One D for all sentences: too small for some, too large for others.
  • Discrete tokens. Mimi's RVQ tokens are lossy; CALM (CALM (Pocket TTS)) argues for continuous latents to get quality at lower compute.
  • GPU-sized. Kyutai TTS is a large model; Pocket TTS targets CPU.

Check yourself

Q1How does DSM get both ASR and TTS from one formulation?

Text and audio are time-aligned parallel streams in one decoder-only model. Delaying the text stream behind audio gives ASR (hear, then write); delaying audio behind text gives TTS (read, then speak).

Q2What does DSM do instead of learning a read/write policy?

It moves alignment into preprocessing, so streams are already aligned in time, and uses a fixed delay between them. The model just predicts the next frame of each stream; no policy has to be learned.

Q3Why does a sentence-level TTS hurt a voice agent's latency?

It cannot start until the LLM has produced the whole sentence, so first audio waits for sentence generation plus synthesis. With a streaming TTS like DSM, speech starts after a few words.

Q4What does the delay D trade off?

Look-ahead vs latency. A larger delay gives the model more upcoming text, improving prosody and pronunciation, but postpones first audio. A smaller delay speaks sooner with less context.

Q5Why can DSM handle arbitrarily long text?

It is causal and frame by frame, consuming text as it streams in and producing audio continuously, so there is no fixed-length utterance to generate at once.

Q6Compare DSM with CosyVoice 2's chunk-aware streaming. Which can start speaking before the LLM finishes the sentence?

DSM consumes text word by word with a small fixed look-ahead, so it starts after a few words. CosyVoice 2 streams its token-to-mel stage in chunks; check that paper for how much text its LLM stage needs before the first chunk. DSM's design is built for text arriving live.

When you're back at a laptop

  • Clone github.com/kyutai-labs/delayed-streams-modeling and run Kyutai TTS on your GPU (check VRAM needs against your 4 GB 3050; fall back to a cloud GPU).
  • Feed text word by word with sleeps to simulate an LLM; log TTFA.
  • Compare with Pocket TTS on CPU (CALM (Pocket TTS)) and with F5-TTS sentence-level (F5-TTS).
  • Read the paper's TTS section and record the actual delay used and how text is inserted at inference.
arXiv ↗PDF ↗