TDT

Why this paper exists

A standard RNN-T (RNN-T) walks through the encoder output one frame at a time. At each frame it calls the joint network (the small network that combines an encoder frame with the prediction network's state) and either emits a token, staying on the same frame, or emits blank, which means "nothing more here, move to the next frame".

Most frames are blank. Speech has far more frames than tokens: even at Fast Conformer's 80 ms per frame (Fast Conformer) a 10 s clip has 125 frames but maybe 30 subword tokens. So most joint-network calls in greedy or beam search only say "blank, next frame". Those calls are sequential (each depends on the previous decision), so they cannot be batched away, and on a GPU the decoding loop, not the encoder, often dominates latency. For a voice agent that is wasted time on every utterance.

The core idea

Let the model say how far to move. At every step the joint network outputs two things:

• a token distribution over the vocabulary plus blank, as in RNN-T, and
• a duration distribution over a small set of frame counts, for example {0, 1, 2, 3, 4}.

After emitting a token or blank with duration d, the decoder advances d frames. A long silence or a long vowel can be covered in one step. Duration 0 means "emit another token at this same frame", which plays the role of RNN-T's non-blank stay.

How it works

Architecture

audio -> encoder -> f_t (one vector per frame)
                        \
tokens y_<u -> pred net -> g_u
                        /
             joint(f_t, g_u) -> hidden h
                |                 |
          token head        duration head
        softmax(V+blank)   softmax over D
          P_T(v|t,u)         P_D(d|t,u)

• t is the encoder frame index (0 to T-1), u is how many tokens have been emitted so far (0 to U).
• V is the vocabulary; D is the set of allowed durations, e.g. {0,1,2,3,4}.
• The two heads share the joint hidden state but have separate softmaxes. The probability of one step is the product:

P(v, d | t, u) = P_T(v | t, u) * P_D(d | t, u)

Transitions in the T x U lattice

From lattice node (t, u):

• emit a real token y_{u+1} with duration d in D (d may be 0): go to (t + d, u + 1)
• emit blank with duration d in D, d >= 1: go to (t + d, u)

Blank must move at least one frame, otherwise the decoder could loop forever. Compare RNN-T: token goes to (t, u+1), blank goes to (t+1, u). RNN-T is the special case where tokens always have d = 0 and blanks always d = 1.

Training loss

As in RNN-T, the loss is the negative log of the total probability of all paths through the lattice that produce the target text. The forward variable α(t, u) is the probability of reaching node (t, u):

α(t,u) =
    Σ_{d in D, d>=1} α(t-d, u)
        * P_T(blank | t-d, u)
        * P_D(d | t-d, u)
  + Σ_{d in D} α(t-d, u-1)
        * P_T(y_u | t-d, u-1)
        * P_D(d | t-d, u-1)

L = -log α(T, U)

The first line is "arrived by a blank that jumped d frames", the second "arrived by emitting token y_u and jumping d frames". Paths must end exactly at the end of the audio with all U tokens emitted (the paper handles the final step so paths terminate at T). The same dynamic programming as RNN-T works; each node now sums over |D| predecessors instead of 1, so training is a bit more expensive. No duration labels are needed: the model learns durations by marginalising over all paths, just as CTC and RNN-T learn alignments.

Greedy decoding

t, u, hyp = 0, 0, []
while t < T:
    v, d = argmax P_T(.|t,u), argmax P_D(.|t,u)
    if v != blank:
        hyp.append(v); update pred net
    elif d == 0:
        d = 1          # blank must advance
    t += d

Build the intuition

Walking through a decode by hand

Take 8 encoder frames (t = 0..7) of someone saying "cat", tokenised as c, a, t, with D = {0,1,2,3,4}.

Five joint calls. A standard RNN-T greedy decode needs one call per emitted token plus one blank call per frame: 3 + 8 = 11 calls. With realistic data the saving grows, because real utterances have long runs of blank frames (pauses, held vowels, frames inside a long subword) that TDT crosses in jumps of up to 4.

What the jump means in time

Parakeet-TDT models run the encoder at 80 ms per frame (8x subsampling of 10 ms features). A duration of 4 skips 320 ms of audio in one step. A 2 s pause in a phone call is 25 frames: RNN-T makes 25 blank calls; TDT can make about 7 (6 jumps of 4 and one more).

Analogy: RNN-T reads a transcript by moving its finger one tick at a time along a timeline and saying "nothing" at most ticks. TDT moves its finger by up to four ticks when it can see nothing is coming, and only slows down where words are.

Numbers worth remembering

• Up to ~2.8x faster inference than a conventional RNN-T on ASR, with equal or better WER (the abstract quotes 2.82x).
• Gains also reported on speech translation and spoken language understanding (roughly 2.3x and 1.3x faster; check the abstract for exact figures).
• Typical duration set: {0, 1, 2, 3, 4}, which is also what NVIDIA's Parakeet-TDT configs use.
• At 80 ms frames, max duration 4 = 320 ms skipped per step.

Where it shows up today

• The "TDT" in Parakeet-TDT (for example parakeet-tdt-0.6b), NVIDIA's widely used English ASR models with a Fast Conformer encoder (Fast Conformer).
• NVIDIA's multilingual Parakeet / Nemotron ASR releases build on Fast Conformer + CTC/TDT (check each model card for which decoder is used).
• The idea of predicting how long each unit lasts also appears on the TTS side: duration predictors in FastSpeech 2 (FastSpeech 2) and alignment learning in One TTS Alignment. There, durations are used to expand text to frames; here, to skip frames.

What it got wrong, and what came next

• Still a frame-synchronous transducer underneath. Training needs the full T×U lattice (now with |D| predecessors per node), which is memory-hungry for long audio. Fast Conformer's 8x subsampling (Fast Conformer) keeps T small enough.
• Durations are capped. Long silences still take several steps; in practice a VAD in front of the model handles really long gaps.
• Beam search is more complex, because hypotheses can sit at different frames. NeMo has dedicated TDT beam search; greedy decoding is what most deployments use.
• Streaming requires a streaming encoder; the cache-aware Fast Conformer (Cache-aware streaming Conformer) provides that.

Check yourself

When you're back at a laptop

• Load nvidia/parakeet-tdt-0.6b-v2 in NeMo, transcribe a recording, and time it. If your 4 GB RTX 3050 runs out of memory, try a shorter clip or CPU.
• Modify a greedy TDT decode (or read NeMo's) to log each step's frame, token and duration. Count steps versus frames.
• Implement the TDT forward recursion above for a tiny lattice (T = 6, U = 2, D = {0,1,2}) with random log-probs in log-space, and check that setting D to the RNN-T special case reproduces torchaudio.functional.rnnt_loss.