Conformer

Why this paper exists

By 2020 there were two competing encoder styles:

• Transformers (self-attention) relate any frame to any other frame, so they capture global context well, but they are weak at fine local patterns. Attention has no built-in notion of "the next 3 frames" and must learn it from data.
• CNNs like QuartzNet and ContextNet (Jasper → QuartzNet → Citrinet, ContextNet) are excellent at local patterns, like the formant transitions that distinguish "b" from "d", but only see a window. ContextNet's squeeze-and-excitation is a crude global summary.

Each was losing something. The question was whether one block could have both, and in what arrangement.

The core idea

Speech needs both scales: local (tens of milliseconds, phones) and global (seconds, words and sentence context). Put self-attention for global context and a depthwise convolution for local patterns in the same block, in sequence, and sandwich them between two feed-forward layers that each contribute half a residual step (the "macaron" structure, named after the cookie with filling between two halves).

How it works

The whole model

waveform -> 80-dim log-mel, 10 ms frames
 -> conv subsampling (2 x Conv2d, stride 2): 4x
    = 40 ms per frame, 25 frames/s
 -> linear + dropout
 -> N x Conformer block
 -> decoder: single-layer LSTM transducer (RNN-T)

The paper uses an RNN-T decoder (RNN-T), but the encoder is decoder-agnostic: later systems put CTC, TDT or attention decoders on it.

One Conformer block

x (T x d)
|------------------.
FFN module         |
(x 0.5) ---------> +   x~ = x + 0.5*FFN(x)
|------------------.
MHSA (rel. pos.)   |
---------------->  +   x' = x~ + MHSA(x~)
|------------------.
Conv module        |
---------------->  +   x'' = x' + Conv(x')
|------------------.
FFN module         |
(x 0.5) ---------> +
|
LayerNorm              y = LN(x'' + 0.5*FFN(x''))
|
y (T x d)

Each module has its own pre-LayerNorm inside the residual branch.

FFN module: LayerNorm -> Linear(d, 4d) -> Swish -> Dropout -> Linear(4d, d) -> Dropout. Swish is x * sigmoid(x).

MHSA module: LayerNorm -> multi-head self-attention with relative positional encoding -> Dropout. Per head, attention is softmax(Q K^T / sqrt(d_k)) V, where Q, K, V are linear projections of the input and d_k is the per-head dimension. Relative positional encoding (from Transformer-XL) scores attention by the distance between frames rather than their absolute index, so it generalises better to utterance lengths unseen in training.

Convolution module:

LayerNorm
-> pointwise conv (d -> 2d)
-> GLU (2d -> d)
-> depthwise conv (kernel 32, over time)
-> BatchNorm -> Swish
-> pointwise conv (d -> d)
-> Dropout

GLU (gated linear unit) splits its input into halves a, b and returns a * sigmoid(b): a learned gate. The depthwise convolution is the QuartzNet trick (Jasper → QuartzNet → Citrinet): each channel filtered over time separately, so a 32-frame kernel (1.28 s at 40 ms) costs only 32 * d parameters.

Model sizes

Counting parameters by hand

Ignore biases and norms (small). For width d:

• each FFN: d*4d + 4d*d = 8d^2; two FFNs = 16d^2
• MHSA: Q, K, V, output = 4d^2, plus a positional projection d^2 = 5d^2
• conv module: 2d^2 (first pointwise) + d^2 (last pointwise) + 32d (depthwise) ≈ 3d^2
• total per block ≈ 24d^2

For Conformer-S, d = 144: 24 * 20,736 ≈ 0.50M per block, times 16 ≈ 8.0M. Add the subsampling convs and projection (a few hundred thousand) and the LSTM decoder and joint network (around 1-2M), and you land near the reported 10.3M. The same formula gives ~25M encoder parameters for M and ~107M for L, again consistent with the table. Two-thirds of the parameters are in the feed-forward layers; the convolution module is the cheapest part, yet the ablations show it matters most.

Build the intuition

Analogy: attention is a meeting where every frame can ask every other frame a question; convolution is each frame chatting with its immediate neighbours. Speech recognition needs the meeting (which word fits the sentence) and the chat (is this burst a "p" or a "t").

What each part does, by ablation: the paper removes or swaps parts one at a time (check its ablation table for exact WERs). Removing the convolution module hurts the most. Replacing the macaron pair with a single full FFN hurts. Putting the convolution after attention works better than before it. Relative positional encoding and Swish both help a little. Kernel sizes from 3 to 65 were tried; around 17-32 worked best.

Why half-step FFNs? The macaron design comes from viewing a residual network as solving an ODE step by step: splitting the FFN into two half steps around the attention resembles a more accurate numerical integration scheme. Practically: two half-weighted FFNs worked better than one full one at the same parameter count.

Frame-rate arithmetic: a 10 s utterance is 1,000 input frames, 250 after 4x subsampling. Attention compares all pairs: 250² = 62,500 scores per head per layer. A 5-minute meeting is 7,500 frames: 56M scores per head per layer, which is why long audio pushed NVIDIA to Fast Conformer (Fast Conformer).

Numbers worth remembering

• Conformer-L on LibriSpeech test-clean / test-other: ~2.1% / ~4.3% WER without a language model, ~1.9% / ~3.9% with one.
• Conformer-S (~10M params): ~2.7% / ~6.3% without LM, already competitive with much larger earlier models.
• 4x subsampling: 40 ms per frame, 25 frames per second.
• Per-block parameters ≈ 24d^2; depthwise kernel size 32.
• Trained on LibriSpeech 960 h with SpecAugment (SpecAugment).

Where it shows up today

• Fast Conformer (Fast Conformer) is the encoder of NVIDIA Parakeet (CTC, RNN-T, TDT TDT) and Canary (Canary).
• Google USM (Google USM) is a ~2B-parameter Conformer; BEST-RQ (data2vec / w2v-BERT / BEST-RQ) and w2v-BERT pretrain Conformers.
• The cache-aware streaming models (Cache-aware streaming Conformer) are Conformers trained with limited context.
• ESPnet's default ASR recipes (hybrid CTC/attention Joint CTC-Attention) use Conformer or E-Branchformer encoders (Squeezeformer / E-Branchformer / Zipformer); AI4Bharat's IndicConformer models are named after it.
• On the TTS side, ESPnet offers Conformer encoders for FastSpeech 2-style models (FastSpeech 2).

What it got wrong, and what came next

• Quadratic attention at a high frame rate. 25 frames per second times O(T²) attention is costly for long audio. Fast Conformer (Fast Conformer) moved to 8x subsampling and limited-context attention.
• Same resolution in every layer. Deep layers may not need 40 ms detail. Squeezeformer and Zipformer (Squeezeformer / E-Branchformer / Zipformer) vary the frame rate inside the encoder.
• Redundant pieces. Squeezeformer found the macaron structure and many LayerNorms could be simplified; E-Branchformer (Squeezeformer / E-Branchformer / Zipformer) runs attention and convolution in parallel instead of sequence.
• Full-context attention is not streamable. Streaming needs chunked, cached attention and causal convolutions (Cache-aware streaming Conformer, Emformer).
• BatchNorm in the conv module behaves differently with small or streaming batches; some later variants swap it for LayerNorm.

Check yourself

When you're back at a laptop

• Implement a Conformer block in PyTorch from the diagram above (standard nn.MultiheadAttention is fine at first; add relative positions later).
• Stack 16 blocks at d = 144, 4 heads, kernel 32 (Conformer-S; the guide suggests 12 blocks, which gives ~6M encoder parameters) and check sum(p.numel() for p in model.parameters()) against the 24d^2 estimate.
• Run a forward pass on a batch of 10 s of random log-mel features on your RTX 3050 and watch memory; then double the length and see the quadratic growth in attention.
• Ablate: remove the conv module and compare parameter count and (if you train on a small dataset) loss curves.