Reading Room

ASR · Phase 2: Efficient transducers

Jasper → QuartzNet → Citrinet

Jasper, QuartzNet and Citrinet: NVIDIA's all-convolutional CTC models

Li et al. (Jasper); Kriman et al. (QuartzNet); Majumdar et al. (Citrinet) · NVIDIA · 2019

Skim14 minDay 1 of 10#7 of 86

In one sentence

Three NVIDIA papers that showed a stack of plain 1D convolutions trained with CTC can do competitive speech recognition: Jasper made it deep, QuartzNet made it small with depthwise separable convolutions, and Citrinet added squeeze-and-excitation, subword tokens and 8x downsampling.

Why this paper exists

In 2019 end-to-end ASR mostly meant recurrent networks: Deep Speech 2 (Deep Speech 2) used convolution plus RNN layers, and Listen, Attend and Spell (Listen, Attend and Spell) used BiLSTMs. RNNs process frames one after another, so they use GPUs poorly and train slowly. NVIDIA wanted a model built only from operations GPUs are good at (convolutions, batch norm, ReLU) so it would train fast and deploy easily in their inference stack.

Jasper proved this works but needed ~333M parameters. That is too big for edge devices and expensive to train. QuartzNet asked: can we keep the accuracy with a fraction of the parameters? Citrinet then asked: can a pure-CTC convolutional model close the gap to the transducer and attention models that were winning benchmarks?

The core idea

A 1D convolution over time with a wide kernel sees a window of audio frames, and stacking many of them with residual connections gives a large receptive field (how many input frames influence one output). You do not need recurrence to see enough context for most words.

The key trick, from QuartzNet on, is the time-channel separable convolution: split a normal convolution into a depthwise part (each channel filtered over time on its own) and a pointwise part (a 1x1 convolution that mixes channels at each frame). This costs far fewer parameters for nearly the same modelling power.

How it works

Jasper (2019)

  • Input: log-mel features at 10 ms per frame. The first convolution has stride 2, so the network runs at 20 ms per frame.
  • Naming Jasper BxR: B blocks, each with R sub-blocks. A sub-block is 1D conv → batch norm → ReLU → dropout. Each block has a residual connection from its input to its last sub-block.
  • Jasper 10x5 has 54 conv layers in total (the 50 inside blocks plus a prologue and epilogue convolutions) and ~333M parameters.
  • Output: a softmax over characters plus blank at every frame, trained with CTC (CTC).
  • A "dense residual" variant feeds each block's output to all later blocks. The paper also introduced the NovoGrad optimizer (an Adam variant with per-layer second moments).

QuartzNet (2019)

Same BxR layout, but every conv is time-channel separable:

x (T x C)
 -> depthwise conv: kernel K over time, per channel
 -> pointwise conv: 1x1, C -> C (mixes channels)
 -> batch norm -> ReLU

Parameter count for one layer with C channels and kernel K:

  • Normal 1D conv: K * C * C
  • Separable: K * C + C * C

For C = 256, K = 33: normal is 33·256·256 ≈ 2.16M; separable is 8.4k + 65.5k ≈ 74k, about 29x fewer. Because depthwise kernels are so cheap, QuartzNet can afford very wide kernels (tens of frames), which gives it a big receptive field. QuartzNet 15x5 has ~19M parameters, versus Jasper 10x5's ~333M, with similar LibriSpeech accuracy.

Citrinet (2021)

  • Keeps QuartzNet's separable 1D conv blocks.
  • Adds squeeze-and-excitation (SE) to each block, borrowed from ContextNet (ContextNet): average the block's features over the whole utterance, pass them through a small MLP, and use the result to rescale each channel. This lets a local convolution know something about the global context (speaker, channel, noise).
  • Switches the output from characters to subword tokens (SentencePiece word pieces). Fewer output tokens per second means the model can run at a lower frame rate.
  • Downsamples time by 8x in stages (stride-2 convolutions), so the encoder runs at 80 ms per frame. Fast Conformer (Fast Conformer) later uses the same 8x.
  • Still trained with CTC only, no autoregressive decoder.

Build the intuition

Analogy: a normal convolution is a chef who, for every dish, tastes every ingredient against every other ingredient over a window of time. A separable convolution first lets each ingredient simmer alone over time (depthwise), then mixes all ingredients once at the end (pointwise). Most of the flavour, a tiny fraction of the work.

Receptive field, by hand: one conv with kernel 33 at 20 ms per frame sees 33 × 20 ms = 660 ms. Stack two and you see ~1.3 s. After a stride-2 layer, each step is 40 ms, so the same kernel covers twice as much audio. That is why a deep stack of convolutions sees whole words and short phrases without any recurrence or attention. What it does not see is the full utterance; SE is Citrinet's cheap fix for that.

Why subwords allow 8x downsampling: CTC needs at least one output frame per token (more if a token repeats, because of the blank). At 80 ms per frame you get 12.5 frames per second. Fast speech is ~15 characters per second, which would not fit, but it is only a few word pieces per second, which fits easily.

Numbers worth remembering

  • Jasper 10x5: 54 conv layers, ~333M parameters, ~3% WER on LibriSpeech test-clean with a Transformer-XL language model (check the paper's results table for the exact value).
  • QuartzNet 15x5: ~19M parameters, accuracy close to Jasper at ~1/17 of the size.
  • Separable conv saves roughly a factor of K (kernel width) in parameters for wide kernels: ~29x in the C = 256, K = 33 example above.
  • Frame rates: Jasper and QuartzNet run at 20 ms per frame (2x stride); Citrinet at 80 ms (8x).
  • Citrinet's largest model, Citrinet-1024, is on the order of ~140M parameters; check the paper for exact sizes and WERs.

Where it shows up today

  • The depthwise-plus-pointwise pattern is inside the Conformer convolution module (Conformer) and Fast Conformer's subsampling layers (Fast Conformer).
  • Squeeze-and-excitation came from ContextNet (ContextNet), and 8x downsampling moved from Citrinet into Fast Conformer, the encoder of Parakeet and Canary (Canary).
  • QuartzNet is still a good small model for CPU inference or keyword-style tasks, and it is one of the classic NeMo tutorial models.
  • Pure-CTC heads like Citrinet's live on in Parakeet-CTC and in the CTC fine-tuning of wav2vec 2.0 (wav2vec 2.0) and MMS (MMS).

What it got wrong, and what came next

  • No global context. Convolutions only see a window. SE gives a crude global summary, but self-attention does this far better. Conformer (Conformer) combined attention and convolution and beat these models.
  • CTC's independence assumption. Each frame's output is predicted independently, so these models lean on external language models. Transducers (RNN-T, TDT) and attention decoders (Whisper, Canary) remove that.
  • Character outputs in Jasper/QuartzNet forced high frame rates. Citrinet's subwords fixed that, and every later NVIDIA model uses subwords.

Check yourself

Q1What is a time-channel separable convolution, and why does it save parameters?

It splits a 1D convolution into a depthwise convolution (each channel filtered over time separately, K·C parameters) and a pointwise 1x1 convolution that mixes channels (C·C parameters). A full convolution needs K·C·C parameters. For wide kernels this is roughly K times cheaper, which is how QuartzNet got near Jasper's accuracy at ~19M instead of ~333M parameters.

Q2Why did NVIDIA want an all-convolutional model instead of an RNN?

RNNs step through frames sequentially, which limits GPU parallelism and slows training and inference. Convolutions process all frames in parallel and map well onto GPU kernels and inference engines. The trade-off is a limited receptive field.

Q3What does squeeze-and-excitation add to Citrinet?

It averages the block's features over the whole utterance, passes the average through a small bottleneck MLP with a sigmoid, and multiplies each channel by the result. This gives each convolution a summary of the global context, such as speaker or recording conditions, at almost no cost.

Q4Why could Citrinet use 8x downsampling when Jasper used 2x?

CTC needs at least one frame per output token. Jasper predicted characters, which arrive fast, so it needed fine frames. Citrinet predicts subword tokens, which arrive much more slowly, so 12.5 frames per second is enough, and fewer frames make the whole network cheaper.

Q5What would happen if you removed the residual connections from Jasper?

A network with 50+ conv layers would be very hard to train: gradients would vanish or explode through so many layers. Residual connections give gradients a short path back to the early layers, which is what made a 54-layer convolutional acoustic model trainable.

When you're back at a laptop

  • In NeMo, load the pretrained QuartzNet 15x5 English model (stt_en_quartznet15x5) and transcribe a recording of your own voice on CPU.
  • Print the model and count parameters per layer type. Confirm the depthwise layers are tiny next to the pointwise ones.
  • Write one separable 1D conv block in PyTorch (nn.Conv1d(C, C, K, groups=C) then nn.Conv1d(C, C, 1)) and compare its parameter count with nn.Conv1d(C, C, K).
Jasper arXiv ↗QuartzNet arXiv ↗Citrinet arXiv ↗