Emformer
Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognition
In one sentence
Emformer makes a Transformer encoder stream by processing audio in fixed-size blocks, attending to a short cached left context, a small right look-ahead, and a compact memory bank that summarises everything older.
Why this paper exists
Full self-attention looks at the whole utterance, so it cannot stream, and its cost grows with the square of the length. Earlier streaming Transformers either attended to all past frames (cost grows during a long call) or recomputed the left context every block. Meta's AM-TRF (augmented-memory Transformer) introduced memory summaries but was slow to train and decode.
The core idea
Split the input into segments. For each segment, attention sees only three things: the current block, a short slice of the past whose keys and values are cached from the previous step (not recomputed), and a few future frames as look-ahead. Long-range history is squeezed into a handful of memory vectors, one per past segment. Cost per block becomes constant, no matter how long the audio runs.
How it works
memory bank left ctx center block right ctx
[m1 m2 m3] [ cached ] [ C frames ] [ R frames ]
\ \ | /
+------- self-attention (one layer) -+
|
output for the C center frames
+ new memory vector for this block- Center block: the C frames whose outputs are produced this step.
- Left context: the last L frames before the block. Their keys and values were already computed in the previous step, so Emformer just reuses them.
- Right context: R future frames used as look-ahead. This is what sets the algorithmic latency.
- Memory bank: a summary vector per past segment (built from a pooled summary of the segment passed through attention). It gives each block a cheap view of distant history.
- Parallel training: at training time all blocks of an utterance are computed in one pass using masks and copies of the right-context frames, instead of looping block by block. That is where the training speedup over AM-TRF comes from.
- Used as the encoder of an RNN-T (RNN-T), so the whole system streams.
Build the intuition
Analogy: taking minutes in a long meeting. You remember the last few sentences word for word (left context), you hear the speaker start the next sentence (right context), and for anything older you rely on short notes from each earlier agenda item (memory bank).
What if R = 0? Latency drops to about one block, but each frame decides with no future evidence, so WER rises. The paper's 80 ms vs 960 ms results show that trade-off directly.
Numbers worth remembering
- LibriSpeech, ~960 ms average latency: 2.50% WER test-clean, 5.62% test-other.
- ~80 ms latency: 3.01% / 7.09%, a 9% / 16% relative reduction vs an LSTM baseline of the same latency and size.
- vs AM-TRF: ~4.6× faster training and 18% lower decoding real-time factor.
Where it shows up today
Emformer RNN-T is the streaming ASR model shipped in torchaudio (torchaudio.models.emformer_rnnt_base, with a LibriSpeech pipeline). The same block-plus-cache pattern appears in NVIDIA's cache-aware Fast Conformer (Cache-aware streaming Conformer) and in Zipformer streaming models (Squeezeformer / E-Branchformer / Zipformer). Kyutai's DSM (Kyutai DSM) takes a different route: a causal decoder-only model with a fixed delay instead of block look-ahead.
What it got wrong, and what came next
- Pure Transformer blocks miss the local modelling that convolution gives; Conformer-based streaming (Conformer, Cache-aware streaming Conformer) does better.
- The memory bank is a lossy summary; later systems mostly rely on a long cached left context instead.
- Look-ahead is fixed at training time; Cache-aware streaming Conformer trains one model that supports several look-aheads.
Check yourself
Q1What problem does caching the left context's keys and values solve?
Without it, each new block would recompute the encoder activations for the overlapping past frames, multiplying compute by the overlap factor. Caching means every frame is computed once and then reused by the following blocks.
Q2Which part of Emformer sets the latency, and why?
The right context. The model cannot output a block until the R future frames have arrived, so latency is roughly block size plus look-ahead. Left context and memory only use past audio and add no waiting.
Q3Why is per-block cost constant over a long call?
Each block attends to a fixed number of left-context frames, a fixed right context and a memory bank of bounded size. Nothing grows with total audio length, unlike full-history attention.
When you're back at a laptop
- Run torchaudio's
EMFORMER_RNNT_BASE_LIBRISPEECHstreaming pipeline on a WAV file, feeding it in chunks and printing partial text. - Change the chunk feed size and time per-chunk latency on your RTX 3050 vs CPU.