ASR · Phase 5: Scale and multilinguality
Granary
Granary: Speech Recognition and Translation Dataset in 25 European Languages
In one sentence
Granary is a large open ASR and speech-translation dataset for 25 European languages, built by pseudo-labelling public audio (segmentation, two-pass inference, hallucination filtering, punctuation restoration) and generating translations with an LLM; models trained on it match those trained on curated data using about half as much data.
Why this paper exists
Canary Canary showed data quality beats raw scale, but good labelled speech is scarce. Plenty of public audio exists with no or poor transcripts, and speech-translation data (audio in one language, text in another) is scarcer still. The authors present Granary as the first open effort at this scale for both transcription and translation.
The core idea
Treat dataset building as an engineering pipeline: let existing ASR models label the audio, then spend most of the effort finding and removing bad labels. Translate the cleaned transcripts with a text LLM, and filter those too.
How it works
The stages named in the abstract:
- Segmentation of long recordings into training-sized utterances.
- Two-pass inference: ASR is run twice to improve the pseudo-labels (pseudo-label = a model-made transcript). Check the paper for what changes between passes.
- Hallucination filtering: drop segments whose label is likely invented text, such as repetitions or text that does not fit the audio.
- Punctuation restoration for consistent, readable targets.
- Translation pairs: translate transcripts with EuroLLM (a European-language LLM), then filter, giving speech-to-text translation data.
The authors say the pipeline processes large amounts of data within hours.
Build the intuition
Analogy: a grain mill. Raw harvest (unlabelled audio) goes in, is sorted and sifted (segment, label, filter, punctuate), and only clean grain comes out. The mill matters more than the field.
Why can less data do as well? Each bad label pushes the model the wrong way, and hallucinated labels teach it to hallucinate. Removing noisy examples leaves fewer hours but a cleaner signal, the same reason Distil-Whisper Distil-Whisper drops clips above a WER threshold. Without the hallucination filter, the labeller's failures on silence and music would become targets, recreating Whisper's failure mode Whisper.
Numbers worth remembering
- 25 European languages, ASR and speech translation.
- On-par results with ~50% less data than previously curated datasets (from the abstract).
- Accepted at Interspeech 2025.
- Total and per-language hours: check the paper's tables (not in the abstract, so not stated here).
Where it shows up today
- NVIDIA's multilingual Parakeet and Canary releases from mid-2025 use Granary data; check each model card for which ones.
- Same pipeline family as WhisperX WhisperX and Distil-Whisper Distil-Whisper.
- The template for Indian languages: public Indic audio, a strong Indic labeller (IndicWav2Vec, Vistaar), an Indic LLM for translation.
What it got wrong, and what came next
- European only; MMS MMS and USM Google USM cover breadth with different trade-offs.
- Labeller bias: accents the labeller handles badly get mislabelled or filtered out, so they end up under-represented (compare Svarah Svarah).
- LLM translation errors become targets, as with Canary's synthetic data Canary.
Check yourself
Q1What are the main stages of the Granary pipeline?
Segmentation, two-pass ASR inference for pseudo-labels, hallucination filtering and punctuation restoration. Translations are then generated with EuroLLM and filtered.
Q2How can half the data match a curated dataset?
Label quality matters more than hours. Filtering removes hallucinated and mismatched labels that would teach wrong mappings, so the remaining data gives a cleaner signal.
Q3Why is hallucination filtering critical in pseudo-labelling?
Otherwise the labeller's hallucinations on silence, music or noise become targets, and the new model learns to hallucinate in the same places, compounding errors across model generations.
When you're back at a laptop
- Browse the Granary dataset card on Hugging Face: sources, languages, hours.
- Mini-version: segment 1 hour of public Hindi audio with a VAD, label with Whisper and an Indic model, and flag segments where they disagree or the text repeats.
- Write down the exact filter rules from the paper's pipeline section.