ASR · Phase 5: Scale and multilinguality
OWSM
Reproducing Whisper-Style Training Using an Open-Source Toolkit and Publicly Available Data
In one sentence
OWSM (Open Whisper-style Speech Model) retrains a Whisper-like multilingual, multitask encoder-decoder using only public data and the ESPnet toolkit, and releases data scripts, configs, logs and checkpoints so anyone can reproduce or modify it.
Why this paper exists
Whisper Whisper released weights but not its 680k hours of data or training code. So nobody could check for test-set leakage, study how data choices cause behaviours like hallucination, or retrain with a different architecture. You could fine-tune Whisper, but not do science on how it was made.
The core idea
Copy Whisper's recipe (one encoder-decoder, task and language chosen by special tokens, 30 s windows, timestamp prediction), build every ingredient from open sources, and measure how close you get with much less data.
How it works
- Data: many public ASR and speech-translation corpora (LibriSpeech, Common Voice, VoxPopuli, MuST-C, GigaSpeech and others) reformatted into Whisper-style long-form examples with timestamps.
- Prompt format: language token, task token (transcribe, or translate to a target language), timestamp mode, as in Whisper.
- Model: Transformer encoder-decoder trained with joint CTC + attention loss Joint CTC-Attention; the CTC head helps alignment and stability. v3.1 switched the encoder to E-Branchformer Squeezeformer / E-Branchformer / Zipformer.
- Release: recipes, logs and checkpoints are public.
Build the intuition
Analogy: Whisper is a dish you can taste but not watch being cooked. OWSM is the same dish from an open kitchen with a printed recipe: maybe less refined, but you can change one ingredient and see what happens.
Adding an Indian-language corpus to Whisper means fine-tuning and risking forgetting. With OWSM you can put it into the original mix and retrain, which is how large-scale data changes are actually tested.
Numbers worth remembering
- Whisper: 680k hours. OWSM v3: ~180k hours of public data (check the paper's data table per version).
- Versions v1, v2, v3 with growing data; v3 is ~1B parameters (check the model card).
- Competitive with Whisper on some English benchmarks, weaker on many other languages, mainly due to less data.
Where it shows up today
- Successors: OWSM v3.1 (E-Branchformer) and OWSM-CTC (non-autoregressive).
- A baseline in Canary Canary, alongside SeamlessM4T SeamlessM4T.
- Lead author Yifan Peng also co-authored the open Granary dataset Granary.
What it got wrong, and what came next
- Lower accuracy than Whisper, especially for low-resource languages; Canary Canary showed curation and balancing close the gap.
- Whisper's limits: 30 s windows, autoregressive, not streaming (Cache-aware streaming Conformer, Kyutai DSM).
- Public data is English/European-heavy; Indian coverage is thin (IndicWav2Vec, Vistaar).
Check yourself
Q1What can you do with OWSM that you cannot with Whisper?
Retrain from scratch with changed data, architecture or tokenizer, and audit the training data for test-set contamination. Whisper's data and training code are closed.
Q2Why add a CTC loss to an attention encoder-decoder?
CTC forces the encoder's frames to align monotonically with the text, which speeds convergence and makes the decoder less likely to skip or repeat. It can also rescore hypotheses at decode time.
Q3Why is OWSM weaker on many languages?
Public data is several times smaller than Whisper's and concentrated in a few languages, so low-resource languages get very few hours.
When you're back at a laptop
- Load an OWSM v3.1 checkpoint from ESPnet's Hugging Face page; compare with Whisper on the same clip.
- Read an ESPnet
egs2/owsm_*recipe: find data prep, the config YAML and the special tokens. - List which public corpora include Indian languages and their hours.