Squeezeformer / E-Branchformer / Zipformer
Squeezeformer, E-Branchformer and Zipformer: three improvements on the Conformer
In one sentence
Three encoder papers that each rework the Conformer: Squeezeformer squeezes time in the middle of the stack and simplifies the block, E-Branchformer runs attention and a convolutional MLP in parallel and merges them carefully, and Zipformer runs different stacks at different frame rates with a new block and optimizer, powering next-gen Kaldi (k2, icefall, sherpa-onnx).
Why this paper exists
The Conformer (Conformer) won, but it left obvious questions:
- Every layer runs at the same 40 ms frame rate. Deep layers model words and context, which change slowly; spending full resolution there wastes compute.
- The block arrangement was found by hand. Macaron FFNs, attention then convolution, pre-LayerNorm everywhere. Is that order best, and is every piece needed?
- Training recipes are fragile. Large Conformers need careful warm-up and tuning.
Each paper attacks a different one of these.
The core idea
- Squeezeformer: temporal U-Net. Downsample time partway through the encoder, run the middle layers at a lower frame rate, then upsample back with a skip connection. Also clean up the block.
- E-Branchformer: global (attention) and local (convolutional gated MLP) processing in parallel branches, merged by a learned module that itself uses a convolution, plus macaron FFNs.
- Zipformer: a U-Net-like encoder whose stacks run at 50, 25, 12.5, 6.25, 12.5 and 25 Hz, a redesigned block that reuses attention weights, new normalisation and activations, and the ScaledAdam optimizer.
How it works
Squeezeformer (UC Berkeley, 2022)
input -> subsampling (4x, 40 ms)
-> blocks at 40 ms
-> downsample (2x, 80 ms)
-> middle blocks at 80 ms
-> upsample (back to 40 ms) + skip from before
-> final blocks -> CTC- Block reordered from macaron (FFN, attention, conv, FFN) to attention, FFN, conv, FFN: each mixing module is followed by its own FFN, like a standard Transformer.
- Normalisation simplified: the many pre-LayerNorms are replaced by a scaled post-LayerNorm arrangement.
- Activations unified to Swish, and the subsampling uses depthwise separable convolutions.
- Trained with CTC. At equal FLOPs it reports lower WER than Conformer-CTC on LibriSpeech (check its tables).
E-Branchformer (ASAPP and CMU, 2022)
Builds on Branchformer (Peng et al., 2022), which splits each block into two parallel branches:
- Global branch: multi-head self-attention.
- Local branch: cgMLP, a convolutional gating MLP. Project up, split into two halves, run one half through a depthwise convolution over time, multiply it with the other half (a gate), project down.
E-Branchformer's "enhanced" merge: concatenate the two branch outputs, apply a depthwise convolution to the concatenation (so the merge itself sees local context), add it back, and project to the model width. It also wraps the block in macaron-style FFNs.
x -> 0.5 FFN -> [ MHSA ]
[ cgMLP ] -> concat
-> + depthwise conv -> linear
-> 0.5 FFN -> LN -> yIt is a standard strong encoder in ESPnet, typically with hybrid CTC/attention decoding (Joint CTC-Attention).
Zipformer (Xiaomi, 2023)
- Multi-rate stacks: a conv front end takes 100 Hz features to 50 Hz; then six stacks run at 50, 25, 12.5, 6.25, 12.5 and 25 Hz, with outputs combined so the final output is 25 Hz (40 ms). The middle stacks, where context modelling happens, see very few frames.
- New block: roughly twice as deep as a Conformer block. Attention weights are computed once and reused by two self-attention modules and a "non-linear attention" module. Bypass modules learn per-channel how much of the input to pass straight through.
- BiasNorm replaces LayerNorm (keeps some length information that LayerNorm removes), and SwooshR / SwooshL activations replace Swish.
- ScaledAdam: Adam with updates scaled by each parameter tensor's current magnitude (RMS), plus an explicitly learned scale per tensor. This keeps relative update sizes constant and made training faster and more stable. It is paired with a learning-rate schedule called Eden.
- Usually trained as a transducer (pruned RNN-T in k2, RNN-T), with streaming variants.
Build the intuition
Analogy: a Conformer is a factory where every station works at the same speed. Squeezeformer and Zipformer slow the middle stations down because they are doing big-picture work that does not change every 40 ms, and speed them back up at the end where fine timing matters again.
Frame counts for 10 s of audio:
| encoder | frames seen by middle layers |
|---|---|
| Conformer (40 ms everywhere) | 250 |
| Squeezeformer middle (80 ms) | 125 |
| Zipformer slowest stack (6.25 Hz) | ~63 |
Attention in the slowest Zipformer stack compares ~63² ≈ 4k pairs instead of 250² = 62.5k, about 16x fewer. Parallel branches (E-Branchformer) save no compute; they buy accuracy by letting the model decide per layer how much to rely on global versus local processing.
Numbers worth remembering
- Zipformer stack rates: 50, 25, 12.5, 6.25, 12.5, 25 Hz; output at 25 Hz.
- Squeezeformer middle resolution: 80 ms (2x below the Conformer's 40 ms).
- E-Branchformer reports ~1.8% / ~3.7% WER on LibriSpeech test-clean / test-other without external training data (check the paper for the exact setup, including LM use).
- Zipformer reports better WER than Conformer with faster inference and less memory on LibriSpeech, Aishell-1 and WenetSpeech (check its tables for exact figures).
Where it shows up today
- Zipformer is the default model in k2 / icefall and in sherpa-onnx, which runs streaming transducer ASR on CPUs, phones and Raspberry Pi. A practical option for CPU-only Indian voice agents.
- E-Branchformer is in ESPnet recipes and in OWSM's later Whisper-style models (OWSM).
- The variable-frame-rate idea connects to Fast Conformer's 8x subsampling (Fast Conformer) and to low-rate codecs like Mimi at 12.5 Hz (Moshi).
- Squeezeformer is in NeMo as an encoder option, alongside Fast Conformer.
What it got wrong, and what came next
- Complexity. Zipformer's many new parts (BiasNorm, Swoosh, bypass, ScaledAdam) make it harder to reimplement outside icefall; most of industry stayed with Fast Conformer (Fast Conformer) for simplicity and tooling.
- Gains are recipe-sensitive. Improvements of a few tenths of WER on LibriSpeech do not always carry to noisy, accented or multilingual data; check on your own data (Svarah Svarah, Vistaar Vistaar).
- Data and scale dominate. Whisper (Whisper) and Canary (Canary) show training data matters more than encoder tweaks.
Check yourself
Q1What does Squeezeformer's temporal U-Net do, and why?
It halves the frame rate in the middle of the encoder, from 40 ms to 80 ms, runs those blocks on fewer frames, then upsamples back to 40 ms with a skip connection from the earlier layers. Middle layers model slowly changing context, so a lower rate saves compute, while the skip keeps fine detail for the output.
Q2How does E-Branchformer differ from Conformer in combining attention and convolution?
Conformer runs attention and then convolution in sequence. E-Branchformer runs attention and a convolutional gated MLP in parallel branches and merges them by concatenation plus a depthwise convolution and projection. The merge lets each layer weigh global and local information.
Q3Name the frame rates of Zipformer's stacks and explain the benefit.
50, 25, 12.5, 6.25, 12.5 and 25 Hz. The middle stacks see far fewer frames, so attention there is much cheaper, and those layers model longer-range structure anyway. Outer stacks keep higher resolution for precise alignment.
Q4What is ScaledAdam?
An Adam variant that scales each parameter tensor's update by that tensor's current RMS magnitude and learns an explicit scale for each tensor. Updates are then relative to parameter size, which made Zipformer training faster and more stable than plain Adam.
Q5If you needed streaming ASR on a CPU-only device, which of these would you pick and why?
Zipformer through sherpa-onnx: it has streaming transducer models, ONNX export and runtimes for CPU and mobile. The multi-rate design also keeps compute low. Squeezeformer and E-Branchformer are mostly used offline in research toolkits.
When you're back at a laptop
- Install sherpa-onnx, download a pretrained streaming Zipformer transducer, and transcribe from your microphone on CPU. Note latency and CPU use.
- Write a table: for each of the three models, list what it changes relative to the Conformer block, frame rate and training recipe.
- Optionally implement a cgMLP layer in PyTorch (up-projection, split, depthwise conv gate, down-projection) and drop it into your Conformer block from Conformer in place of the conv module.