CTC

Why this paper exists

Audio arrives at roughly 50-100 feature frames per second; a transcript has a few characters or tokens per second. Nobody tells you which frames belong to which letter.

Before CTC, neural networks in speech lived inside hybrid HMM systems: a hidden Markov model (HMM, a probabilistic state machine over sound units) supplied the alignment, and the network only classified frames. Training needed a label for every frame, produced by an existing HMM system via forced alignment. So you could not train a recogniser from scratch, alignment errors became training-target errors, and HMM decoding was still needed to glue frame outputs into words.

CTC removed that dependency: give it audio and text, and it trains the network end to end.

The core idea

Let the network emit one symbol per frame, from the vocabulary plus one extra symbol, blank (written _), meaning "nothing new here". Define a fixed, many-to-one collapse function from frame-level paths to text:

1. merge consecutive repeated symbols,
2. then delete blanks.

So hh_e_ll_lo → h_e_l_lo → hello. Many different paths collapse to the same text, and they differ only in timing. Since we do not know the timing, CTC's training objective is the total probability of all paths that collapse to the target. Alignment becomes a hidden variable that is summed out, not something you must supply.

How it works

audio features x (T frames)
         |
  BLSTM (or any encoder)
         |
  T x (V+1) logits  -- V tokens + blank
         |
  softmax per frame: y_t(k)
         |
  +------+-------------------+
  | train: CTC forward-      |
  | backward over all paths  |
  | decode: argmax/frame,    |
  | collapse; or beam + LM   |
  +--------------------------+

The original paper used a bidirectional LSTM (BLSTM) on TIMIT phoneme recognition. Today the encoder is usually a Conformer (Conformer) or Fast Conformer (Fast Conformer), but the loss is unchanged.

The probability of one path

For a path π = (π_1, ..., π_T), one symbol per frame:

p(π | x) = Π_{t=1..T} y_t(π_t)

where y_t(k) is the softmax probability of symbol k at frame t. This product is the conditional-independence assumption: given the audio, each frame's choice is independent of the others. The network does not see its own previous outputs.

The loss

p(l | x) = Σ_{π : B(π) = l} p(π | x)

L_CTC = -log p(l | x)

l is the target label sequence (length U), and B is the collapse function. The number of paths grows exponentially with T, so you never enumerate them.

The forward algorithm

Insert blanks around every label to get an extended sequence l' of length 2U + 1. For cat: l' = _ c _ a _ t _. Define α_t(s) = total probability of all path prefixes of length t that end at position s of l' and are consistent with the target so far.

• Start: α_1(1) = y_1(_), α_1(2) = y_1(l_1), all others 0.
• Step: α_t(s) = (α_{t-1}(s) + α_{t-1}(s-1)) · y_t(l'_s)
• plus a skip term + α_{t-1}(s-2), allowed only if l'_s is not blank and l'_s ≠ l'_{s-2}.
• End: p(l | x) = α_T(S) + α_T(S-1) with S = 2U + 1 (you may finish on the last label or on the trailing blank).

The skip rule is the heart of it: you may jump straight from a to t without a blank in between, but you may not jump from l to l, because two adjacent ls would merge. That is exactly why "hello" requires a blank between the two ls.

Cost is O(T · U). A backward variable β computed the same way from the end gives the gradients. In practice everything runs in log space, because multiplying hundreds of probabilities underflows.

Decoding

• Greedy (best path): argmax at each frame, then collapse. Fast and often good enough.
• Prefix beam search: keeps hypotheses as collapsed prefixes, and can add an external language model (LM) score. CTC benefits a lot from an LM precisely because of the independence assumption.

Build the intuition

A worked example by hand

Vocabulary {a, b} plus blank _. Target ab. Three frames (T = 3). The network outputs:

There are 27 possible 3-symbol paths. Which collapse to ab?

Sum: p(ab | x) = 0.468, so L = -ln 0.468 ≈ 0.76. Note that aba collapses to aba and a__ collapses to a, so they do not count.

Now check with the forward algorithm on l' = _ a _ b _ (positions 1-5):

• t=1: α(1) = .3, α(2) = .6.
• t=2: α(1) = .3·.4 = .12; α(2) = (.6+.3)·.3 = .27; α(3) = (0+.6)·.4 = .24; α(4) = (0+0+.6)·.3 = .18 (skip from a to b allowed); α(5) = 0.
• t=3 (only the last two positions matter): α(4) = (.18+.24+.27)·.6 = .414; α(5) = (0+.18)·.3 = .054.
• Total: .414 + .054 = .468. Same answer, without listing paths.

Greedy decoding picks a, _, b → ab. Correct here, but greedy finds the single best path, while a transcript's probability is a sum over paths, so the two can disagree; prefix beam search fixes that.

Now change the target to aa. The only valid 3-frame path is a_a: there is no room for any other arrangement. A target with U labels and r adjacent repeats needs at least U + r frames. This is why heavy subsampling (8× in Fast Conformer) has to be checked against your token rate.

Analogy: CTC is like grading a student who must write one character per second on a timed tape, allowed to hold a letter or write a dash. You do not care when each letter appears, only that the tape, read with "squash repeats, drop dashes", spells the answer. The score is the total probability of every tape that would pass.

Numbers worth remembering

• Extended label sequence length: 2U + 1; forward pass cost O(T · U).
• Minimum frames for a target: U plus the number of adjacent repeated labels.
• Original experiments: TIMIT phoneme recognition with a BLSTM; CTC beat the framewise and HMM-hybrid baselines reported in the paper (check Table 1 for the exact label error rates, around 30%).
• Typical modern CTC output rate: 25 frames/s (4× subsampling of 10 ms features) or 12.5 frames/s (8×). CTC is the cheapest decoder to run on a CPU or a 4 GB GPU: one encoder pass and an argmax.

Where it shows up today

• Fine-tuning self-supervised encoders: wav2vec 2.0 (wav2vec 2.0), MMS (MMS) and IndicWav2Vec (IndicWav2Vec) all put a linear CTC head on top.
• NVIDIA's Parakeet-CTC models and the all-convolutional Jasper / QuartzNet / Citrinet line (Jasper → QuartzNet → Citrinet).
• As an auxiliary loss in hybrid models (Joint CTC-Attention) and in transducer training.
• Forced alignment and word timestamps: WhisperX (WhisperX) aligns Whisper text with a phoneme CTC model.
• The same "sum over monotonic alignments" idea reappears in TTS aligners (One TTS Alignment) and in Glow-TTS's monotonic alignment search (Glow-TTS).

What it got wrong, and what came next

• Conditional independence. Each frame ignores what was already emitted, so the model has no internal language model and makes spelling errors an LM would catch. RNN-T (RNN-T) adds a prediction network over previous tokens; attention models (Listen, Attend and Spell) condition on everything.
• At most one token per frame, and the output can never be longer than the input. Transducers allow several tokens per frame.
• Peaky, imprecise timing. Most frames are blank with one sharp spike per token, and spikes do not always sit where the sound is, which hurts timestamps.
• Hard to scale on its own early on. Deep Speech 2 (Deep Speech 2) showed that with enough data and engineering CTC is competitive; Joint CTC-Attention (Joint CTC-Attention) used CTC to stabilise attention models.

Check yourself

When you're back at a laptop

• Implement the forward algorithm in log space in PyTorch, for one utterance, using torch.logsumexp for the additions.
• Check it against torch.nn.functional.ctc_loss (reduction='sum', blank=0) on random log_softmax inputs with T=50, U=10, V=6.
• Reproduce the 3-frame ab example above and confirm 0.468.
• Write greedy decoding (argmax, merge repeats, drop blanks). Keep it: you will reuse it on wav2vec 2.0 logits in wav2vec 2.0.

def greedy(logits, blank=0):
    ids = logits.argmax(-1).tolist()
    out, prev = [], None
    for i in ids:
        if i != prev and i != blank:
            out.append(i)
        prev = i
    return out