Reading Room

ASR · Phase 6: Streaming ASR

Cache-aware streaming Conformer

Stateful Conformer with Cache-based Inference for Streaming Automatic Speech Recognition

Noroozi, Majumdar, Kumar, Balam, Ginsburg · NVIDIA · 2023

Deep read20 minDay 4 of 10#28 of 86

In one sentence

Train a Fast Conformer encoder with the same limited left context and look-ahead it will have at streaming time, then cache attention keys/values and convolution states between chunks, so streaming inference does no recomputation and matches training exactly.

Why this paper exists

A Conformer (Conformer, Fast Conformer) trained offline sees the whole utterance: every attention layer looks at all frames, and every convolution looks both ways. The usual way to stream such a model is buffered streaming: slide a window over the audio, run the full offline model on each window (say a few seconds with overlap), and stitch the outputs. That has two problems:

  • Wasted compute. Overlapping windows re-encode the same frames many times.
  • Train/test mismatch. The model was trained on full utterances but at inference sees chopped windows with abrupt edges, so accuracy drops, especially near chunk boundaries.

The core idea

Make the encoder causal-ish by construction and train it that way:

  1. Limit every self-attention layer to a fixed left context and a fixed (small) right context.
  2. Make every depthwise convolution causal (it only looks at past frames).
  3. At inference, keep a cache per layer: the last left-context activations for attention, and the last kernel-minus-one frames for convolution. Each new chunk is computed once, using the cache, and then pushed into it.

Because training used exactly those limits, streaming outputs match what the model computed in training.

How it works

The picture to memorise

Time runs left to right. One encoder layer, one step. Frames are post-subsampling frames (Fast Conformer subsamples 8×, so one frame = 80 ms).

already processed            new audio      not yet heard
...|<-- left context L -->|<- chunk C ->|<- look R ->|...
   [ cache: K,V from past ][ compute   ][ future   ]
                            these frames  frames
                                  ^
           attention for chunk frames sees L + C (+ R)

conv cache: last (k-1) frames of this layer's input
...[c c c c c c c c][x x x x]   causal conv, kernel k
                    ^ chunk
after the step: newest L frames -> attention cache,
                newest k-1 frames -> conv cache
  • Chunk size C: how many new frames are encoded per step. Bigger chunks are more GPU-efficient but you wait longer before the first output of each chunk.
  • Left context L: how far back attention can look. Longer is more accurate but costs memory and compute per step. It adds no latency since it is the past.
  • Look-ahead R: future frames each output may depend on. This is the part that adds latency.
  • Cache: per layer, the stored activations for the L left frames (attention) and k-1 frames (conv). Total state is fixed size, so a 30-minute call costs the same per step as a 3-second one.

Two ways to get look-ahead

  • Regular look-ahead: each layer lets a frame see R future frames. Stacked over N layers, the effective look-ahead grows to N × R, which gets large fast.
  • Chunk-aware look-ahead: frames are grouped into chunks; every frame in a chunk may attend to all frames in its chunk and to the left context, but not beyond the chunk. The look-ahead does not grow with depth, and the last frame of a chunk has zero look-ahead while the first has C-1. The paper reports chunk-aware gives better accuracy for the same latency, and it is what NVIDIA's released models use.

Attention mask, as a formula

For frame i in chunk c(i) = floor(i / C), frame j is visible if:

c(i)*C - L  <=  j  <  (c(i)+1)*C
  • i, j: frame indices after subsampling.
  • C: chunk size in frames, L: left context in frames.
  • In NeMo this appears as att_context_size = [L, R] with chunk size R + 1.

Latency, roughly

algorithmic latency ~ look-ahead frames x 80 ms
(+ compute time, + any decoder delay)

NVIDIA's released streaming Fast Conformer models offer settings like [70, 0], [70, 1], [70, 6], [70, 13], which correspond to about 0, 80, 480 and 1040 ms of look-ahead with ~5.6 s of left context (70 × 80 ms). Check the model card for the exact values.

Training tricks

  • Multi-look-ahead training: sample a different context size per batch, so one model can be run at several latencies. Pick the latency at deployment time.
  • Hybrid CTC/RNN-T: one shared encoder with two heads, CTC (CTC) and RNN-T (RNN-T). The paper reports the joint training improves accuracy and makes the CTC head converge faster. At inference you choose: CTC for speed, RNN-T for accuracy.

Build the intuition

Analogy: a live subtitler who listens in 1-second pieces. Buffered streaming is a subtitler who rereads the last 5 seconds of transcript audio every time a new second arrives. Cache-aware is one who keeps notes on what was just said (the cache) and only listens carefully to the new second, glancing slightly ahead if allowed (look-ahead).

Tiny worked example: C = 2 frames (160 ms), L = 4 frames, conv kernel 9 (cache of 8 frames), 17 layers. At step t the encoder computes only 2 new frames per layer. Each layer's attention reads 4 cached + 2 new key/value vectors, and each conv reads 8 cached + 2 new inputs. Compare buffered streaming with a 4 s window: 50 frames re-encoded per step to get 2 new outputs, about 25× more work.

If you trained offline and only applied the mask at inference, every layer would see less context than it learned to use and the convs would lose their right half. WER would jump.

Numbers worth remembering

  • Frame rate after Fast Conformer 8× subsampling: 12.5 frames/s, so 80 ms per frame.
  • Released look-ahead options around 0 / 80 / 480 / 1040 ms with ~70 frames (5.6 s) of left context (model card values; check them).
  • The paper reports better accuracy, lower latency and faster inference than buffered streaming on LibriSpeech and a large multi-domain set. For exact WERs per latency, check the results tables.
  • Every frame is encoded exactly once; state per layer is constant size.

Where it shows up today

  • NVIDIA NeMo's streaming Fast Conformer models (for example the fastconformer_hybrid_large_streaming_multi family) and the streaming Parakeet / Nemotron ASR releases build on this; check each model card. NeMo ships speech_to_text_cache_aware_streaming_infer.py for it.
  • The same caching logic powers Emformer (Emformer) and streaming Zipformer (Squeezeformer / E-Branchformer / Zipformer); Google's mobile RNN-T (Streaming E2E ASR for mobile) got it for free with unidirectional LSTMs.
  • Contrast with Kyutai's delayed streams (Kyutai DSM), which streams with a decoder-only model and a fixed text delay, and with Whisper (Whisper), which is offline (30 s windows).

What it got wrong, and what came next

  • Streaming still costs WER versus the offline model with full context; the gap grows as look-ahead shrinks.
  • Look-ahead choices are discrete and trained in; you cannot pick an arbitrary latency.
  • DSM (Kyutai DSM) and Moshi (Moshi) show another design: one causal decoder that interleaves audio and text, which also makes TTS and dialogue possible with the same machinery.

Check yourself

Q1What is the train/test mismatch in buffered streaming, and how does this paper remove it?

The offline model was trained with full-utterance context but at inference sees cut windows, so its activations near window edges are unlike anything seen in training. Cache-aware training applies the exact same left-context, chunk and look-ahead limits during training, so inference computes the same function the model was trained on.

Q2What exactly is cached, per layer?

For self-attention, the keys and values (in practice the layer inputs) of the last L frames. For the causal depthwise convolution, the last k-1 input frames, where k is the kernel size. Both are fixed-size buffers updated after each chunk.

Q3Which of chunk size, left context and look-ahead add latency?

Look-ahead adds latency directly, since you must wait for future audio. Chunk size adds waiting too, because the first frame of a chunk must wait for the rest of the chunk. Left context only uses the past, so it adds compute and memory but not waiting.

Q4Why does regular look-ahead get worse with depth, and how does chunk-aware look-ahead avoid that?

If every layer looks R frames ahead, layer 2 sees layer 1's outputs, which already saw R ahead, so the effective look-ahead becomes N × R. Chunk-aware attention never lets a frame see past the end of its chunk at any layer, so the total look-ahead is bounded by the chunk.

Q5Why train a hybrid CTC/RNN-T model?

The shared encoder gets two training signals, which the paper reports helps accuracy and speeds up CTC convergence. At deployment you can use the CTC head for the cheapest decoding or the RNN-T head for better accuracy, from one checkpoint.

When you're back at a laptop

  • Draw the cache diagram above from memory, then check it against NeMo's CacheAwareStreamingAudioBuffer and the encoder's cache_last_channel / cache_last_time tensors.
  • Run a NeMo streaming Fast Conformer with att_context_size set to [70,0], [70,1], [70,13] on LibriSpeech or your own recordings. Plot WER against look-ahead latency.
  • Time per-chunk compute on your RTX 3050 and on CPU; check the real-time factor stays well under 1.
arXiv ↗PDF ↗