Kyutai DSM
Streaming Sequence-to-Sequence Learning with Delayed Streams Modeling
In one sentence
Delayed Streams Modeling (DSM) puts audio tokens and text tokens on parallel, time-aligned streams, shifts one stream later by a fixed delay, and trains a single decoder-only Transformer on them, so the same recipe gives streaming ASR (text delayed behind audio) or streaming TTS (audio delayed behind text).
Why this paper exists
Sequence-to-sequence models for speech come in two flavours, and both are awkward for real-time use:
- Offline encoder-decoders like Whisper (Whisper) or Canary (Canary) read the whole input, then write the whole output. Great accuracy, no streaming.
- Streaming models like transducers (RNN-T, Cache-aware streaming Conformer) learn the alignment between input and output during training, with special losses, and are built only for one direction (audio to text).
Each task (ASR, TTS) also gets its own architecture. The DSM paper asks whether one simple, decoder-only language model can do streaming for any input/output pair, including arbitrarily long sequences.
The core idea
Do the alignment before training, not inside the model. If you already know when each word was spoken (from word-level timestamps), you can place each text token on the same time grid as the audio frames. Now audio and text are two time-aligned streams: at frame t there is one audio frame's tokens and one text slot (a word token or padding).
Then add a delay: shift the output stream later by a fixed number of frames. For ASR, the text stream is delayed, so when the model writes the text for frame t it has already heard audio up to t plus the delay. For TTS, the audio stream is delayed behind text, so the model has read a little text ahead before it speaks it. A plain causal decoder trained with next-token prediction on these streams is then a streaming model by construction.
How it works
Streams on a time grid
Audio is turned into discrete tokens by Kyutai's Mimi codec (Moshi), at 12.5 frames per second (80 ms per frame). Each frame has several codebook tokens. Text is aligned to the same grid.
ASR: text delayed by d frames behind audio
frame: t1 t2 t3 t4 t5 t6 t7
audio: a1 a2 a3 a4 a5 a6 a7 (input)
text: PAD PAD hel PAD lo PAD ... (output)
|<-d->|
text slot at t3 can use audio a1..a3,
i.e. d frames after the word started
TTS: audio delayed behind text (roles swapped)
text: hel PAD lo PAD ... (input)
audio: -- -- a1 a2 a3 (output)- PAD: a filler token for frames where no new word starts, so text and audio advance in lockstep.
- Delay d: fixed number of frames between the streams. It is a design choice: a longer delay gives more context (better accuracy) but more latency.
The model
per frame t:
[audio tokens_t] + [text token_t] -> embed, sum
|
decoder-only Transformer (causal over t)
|
predict next frame's output-stream tokens
(for multi-codebook audio, a small depth
model predicts codebooks within the frame,
as in Moshi's RQ-Transformer)- Training objective: ordinary next-token cross-entropy on the output stream, given everything earlier on all streams.
- Inference: feed input tokens as they arrive; at each frame, sample the output stream's token. No beam search over alignments, no separate alignment loss.
- Because the model is causal with a fixed per-frame step, it handles arbitrarily long inputs; you can keep a sliding attention window over the past. Check the paper for how long-form context is bounded.
Equation (the whole objective is standard language modelling):
L = - Σ_t log p( y_t | x_{<= t+d}, y_{<t} )x: input stream (audio for ASR),y: output stream (text for ASR).d: the delay in frames. The output at time t is conditioned on input up to t + d, because the output stream has been shifted d frames later.
Build the intuition
Analogy: a simultaneous interpreter who stays a fixed two seconds behind the speaker. They never wait for the end of the speech, and they never guess words that have not been said yet. The fixed lag is the delay d. A text-to-speech system is the same interpreter reading a script aloud, staying a few words behind their eye.
What would happen with d = 0 for ASR? The model would have to output a word's token at the very frame the word starts, before it has heard it. It would guess, and WER would be poor. With a large d it hears the whole word and some right context, like an offline model, but latency grows. DSM turns the latency-vs-WER trade-off into one integer.
What would happen without pre-alignment? You would not know where to put each text token on the grid; you would need CTC/RNN-T-style alignment learning. DSM sidesteps that with an external aligner (timestamps from an existing ASR model) in preprocessing.
Numbers worth remembering
- Audio stream: Mimi tokens at 12.5 Hz (80 ms frames), inherited from Moshi (Moshi).
- One architecture for both directions: text delayed for ASR, audio delayed for TTS.
- The paper reports state-of-the-art performance and latency on both ASR and TTS, competitive with offline baselines, and support for arbitrarily long sequences (abstract). For exact WERs, delays and model sizes, check the paper's tables.
- Submitted September 2025; code, samples and demos are on Kyutai's GitHub.
Where it shows up today
- Behind Kyutai's open STT and TTS models (the guide's "Kyutai STT" row). The TTS side is covered in Delayed Streams Modeling (TTS).
- Reuses Moshi's codec, multi-stream layout and depth Transformer (Moshi); Mimi descends from SoundStream (SoundStream) and EnCodec (EnCodec) style RVQ codecs.
- For a voice agent, a DSM-style STT gives a constant, known lag per word, which is easy to budget against turn-taking and LLM latency. Compare cache-aware transducers (Cache-aware streaming Conformer), where the lag is set by look-ahead.
What it got wrong, and what came next
- It depends on good word-level timestamps for preprocessing; alignment errors become training noise. Indian languages and code-mixed speech have weaker aligners, which matters if you adapt it.
- The delay is fixed per trained model; changing the latency typically means another model or training with several delays (check the paper).
- Decoder-only models with large Transformers are heavier than a small CTC/TDT model (TDT) on CPU; on a 4 GB GPU you need the smaller checkpoints or quantization.
- Discrete audio tokens limit audio quality for TTS; Kyutai's CALM (CALM (Pocket TTS)) explores continuous audio instead.
Check yourself
Q1In DSM, what is "aligned" and when is the alignment done?
Text tokens are placed on the same frame grid as audio tokens, using word timestamps computed in a preprocessing step. The model never learns alignment itself; it is given time-aligned streams and only learns to predict tokens.
Q2How does one model family do both ASR and TTS?
Both are just two aligned streams with one delayed. For ASR, audio is input and text is the delayed output; for TTS, text is input and audio is the delayed output. The decoder and objective are the same.
Q3What does the delay trade off?
Accuracy against latency. A longer delay lets the model hear (or read) more of the future before committing to an output, which improves quality, but every output arrives that much later.
Q4Why is PAD needed in the text stream?
Words are much sparser than 12.5 Hz frames. PAD fills frames with no new word so both streams step together, one slot per frame, and the model can output "nothing new yet".
Q5Why can DSM handle arbitrarily long audio while Whisper cannot natively?
DSM is causal and processes a fixed amount per frame, so it can run indefinitely (with a bounded attention window). Whisper's encoder takes fixed 30 s windows, so long audio needs chunking and stitching (WhisperX).
Q6How is DSM different from a transducer for streaming ASR?
A transducer learns alignment implicitly through a loss that sums over all alignments, and its lag depends on the encoder's look-ahead and on when it chooses to emit. DSM fixes alignment in advance and uses a fixed delay, with plain cross-entropy.
When you're back at a laptop
- Clone Kyutai's
delayed-streams-modelingrepo and run the STT example on your recordings; log the timestamp of each emitted word against the audio to measure the lag. - Try Indian-accented English (e.g. Svarah, Svarah) and note where errors come from.
- Sketch the stream layout for a 2-second clip with 3 words by hand, then compare with the repo's data format.