RNN-T

Why this paper exists

CTC (CTC) solved the alignment problem but made a strong simplification: given the audio, each frame's output is independent of every other frame's output. The model never sees what it has already written, so it cannot learn that "q" is usually followed by "u", or that it just wrote "their" and the next word is unlikely to be "there". CTC also allows at most one label per frame, so the output can never be longer than the input.

Graves wanted a general sequence transducer: any input sequence to any output sequence, of any relative length, with full dependence on previous outputs, still trained without alignments.

The core idea

Split the model into two halves that each look at one sequence, and join them per pair of positions:

• an encoder (Graves called it the transcription network) reads audio frames,
• a prediction network reads the tokens emitted so far, acting like a small language model,
• a joint network combines encoder frame t with prediction state u and outputs a distribution over tokens plus blank.

Here blank means "move to the next audio frame". A real token means "emit this, stay on the same frame". Every alignment is then a monotonic path through a 2D grid of (frame, tokens-emitted-so-far), and the loss sums over all of them.

How it works

audio x_1..x_T          previous tokens y_0..y_{u}
     |                   (y_0 = start/blank)
 ENCODER                 PREDICTION NET (LSTM)
     |                          |
    h_t                        g_u
     \                         /
      +----- JOINT NET -------+
             z = f(h_t, g_u)
                   |
      softmax over V tokens + blank
                   |
         P(k | t, u)  for every (t, u)

Shapes

• Encoder output: T × D (e.g. 25 frames/s after subsampling).
• Prediction output: (U+1) × D (one state for "nothing emitted yet", plus one per target token).
• Joint output during training: T × (U+1) × (V+1) logits per utterance. This 4D tensor (with batch) is the memory hog of transducer training.

The joint network

In the 2012 paper the encoder and prediction outputs were simply added before the softmax. The now-standard form, from Graves, Mohamed and Hinton (2013), adds a hidden layer:

z_{t,u} = W_out · tanh(W_enc · h_t + W_pred · g_u)

P(k | t, u) = softmax(z_{t,u})_k

h_t is encoder frame t, g_u is the prediction network's state after u tokens, k ranges over tokens plus blank. Note the original encoder was a bidirectional LSTM, so the 2012 model was not streaming; streaming came later by making the encoder unidirectional or limited-context.

The T × U lattice

Write ∅ for blank. Index nodes by (t, u): we are at frame t and have emitted u tokens. From each node there are exactly two moves:

• emit ∅ with probability ∅(t,u) → go to (t+1, u),
• emit the next correct token y_{u+1} with probability y(t,u) → go to (t, u+1).

Start at (1, 0). Finish by emitting a final ∅ from (T, U).

The forward recursion

α(1,0) = 1

α(t,u) = α(t-1,u) · ∅(t-1,u) + α(t,u-1) · y(t,u-1)

P(y | x) = α(T,U) · ∅(T,U)

L_RNNT = -log P(y | x)

The first term says "I arrived by finishing frame t-1"; the second says "I arrived by emitting token u while on frame t". Cost is O(T · U) joint evaluations, and a backward variable gives the gradients, just like CTC.

Greedy decoding

t = 1, state = pred_net(start)
while t <= T:
    k = argmax joint(h_t, state)
    if k == blank: t += 1
    else: emit k; state = pred_net(k, state)

Production decoders cap the number of tokens per frame so the loop cannot run forever. Because the decoder only needs frames up to t, tokens appear as audio arrives: this is why transducers are the default for streaming.

Build the intuition

Walking the lattice by hand

Take T = 2 frames and target y = "ab" (U = 2). Draw the lattice: right = blank (next frame), up = emit a token.

u=2  (1,2)----->(2,2)-----> end
       ^          ^       (final ∅)
       | b        | b
u=1  (1,1)----->(2,1)
       ^          ^
       | a        | a
u=0  (1,0)----->(2,0)
      t=1        t=2

Every complete alignment has U = 2 token moves and T = 2 blank moves, ending in a blank. There are 3 of them:

The first alignment, two tokens in one frame, is impossible in CTC. In general there are C(T+U-1, U) alignments: for T = 250, U = 50 that is astronomically many, which is why the dynamic program matters.

Now fill in numbers. Suppose at every node y(t,u) = 0.5 and ∅(t,u) = 0.4 (they need not sum to 1; the other tokens take the rest).

• α(1,0) = 1
• α(1,1) = α(1,0)·y = 0.5; α(1,2) = 0.5·0.5 = 0.25
• α(2,0) = α(1,0)·∅ = 0.4
• α(2,1) = α(1,1)·∅ + α(2,0)·y = 0.2 + 0.2 = 0.4
• α(2,2) = α(1,2)·∅ + α(2,1)·y = 0.1 + 0.2 = 0.3
• P(ab | x) = α(2,2)·∅(2,2) = 0.3 · 0.4 = 0.12

Check: each of the 3 paths has two y moves and two ∅ moves, 0.5² · 0.4² = 0.04, and 3 × 0.04 = 0.12. In a real model the probabilities differ per node, because P(k | t, u) depends on both the audio at t and the text so far at u. That dependence on u is exactly what CTC lacks.

Analogy: a court stenographer listening to a live speaker. At each moment they either write the next word (and their next choice depends on what they have written) or wait for more audio. They never go back. CTC is a stenographer who cannot see their own notes.

What would happen if...

...the prediction network were removed (the joint sees only h_t)? Then P(k | t, u) no longer depends on u, and you are back to something very close to CTC, with the same independence problem. The prediction network is where the language knowledge lives.

Numbers worth remembering

• Lattice size: T × (U+1) nodes; alignments per utterance: C(T+U-1, U).
• Training memory: a 10 s utterance at 25 frames/s (T = 250), U = 100, vocab 1024 gives 250 × 101 × 1025 ≈ 26M joint logits, about 100 MB in fp32 for one utterance before gradients. On a 4 GB RTX 3050 this, not the encoder, is what limits your batch.
• Original experiments: TIMIT phoneme recognition, with a small gain over CTC (check the results table for exact phoneme error rates).
• Modern transducer encoders typically run at 25 or 12.5 frames/s (4× or 8× subsampling).

Where it shows up today

• Google's on-device streaming recogniser (Streaming E2E ASR for mobile) and many later Google systems.
• NVIDIA Parakeet-RNNT, and Parakeet-TDT via the Token-and-Duration Transducer (TDT), on a Fast Conformer encoder (Fast Conformer); cache-aware streaming versions (Cache-aware streaming Conformer).
• Zipformer transducers in k2 / icefall / sherpa-onnx (Squeezeformer / E-Branchformer / Zipformer), which use a pruned RNN-T loss to shrink the joint tensor.
• Emformer + RNN-T in torchaudio (Emformer).
• For a voice agent, a streaming transducer is the usual way to get partial transcripts within a few hundred milliseconds.

What it got wrong, and what came next

• Memory-hungry training from the T × U × V joint tensor. Fixed by function merging, fused loss kernels and pruned RNN-T (Squeezeformer / E-Branchformer / Zipformer).
• Slow frame-by-frame decoding: the decoder visits every frame, even silent ones. TDT (TDT) predicts how many frames to skip.
• The original was not streaming (BLSTM encoder) and needed better encoders and engineering to beat hybrid systems. Google's work (Streaming E2E ASR for mobile), Conformer (Conformer) and cache-aware streaming (Cache-aware streaming Conformer) supplied that.
• The prediction network is a weak LM. It only sees paired audio-text data, so transducers still gain from external LMs or text-only adaptation.
• Attention models (Listen, Attend and Spell) are often more accurate offline because their decoder sees the whole encoded utterance, at the cost of streaming.

Check yourself

When you're back at a laptop

• Code the T = 2, U = 2 example above with real random logits: compute log_softmax of a (T, U+1, V+1) tensor, run the forward recursion in log space, and compare with torchaudio.functional.rnnt_loss (note torchaudio's argument order and that blank defaults to -1; check the docs).
• Brute-force check: enumerate all C(T+U-1, U) alignments for T = 4, U = 2 and sum their probabilities.
• Write the greedy decoding loop with a max-symbols-per-frame cap.
• Print the joint tensor size for your own data (T, U, V) and see what batch fits on 4 GB.