ASR · Phase 8: Indian languages and accents
IndicWav2Vec
Towards Building ASR Systems for the Next Billion Users
In one sentence
AI4Bharat collected ~17,000 hours of unlabelled audio in 40 Indian languages, pretrained wav2vec 2.0 models on it (IndicWav2Vec), and fine-tuned them with CTC to get state-of-the-art ASR in 9 Indian languages.
Why this paper exists
Big pretrained speech models mostly helped a few resource-rich languages. Indian languages had little labelled speech, scattered datasets, and models pretrained mostly on English (or on generic multilingual data with little Indic content). Yet unlabelled Indian-language audio is plentiful: news, education videos, podcasts.
The core idea
Use self-supervised pretraining (wav2vec 2.0) on lots of unlabelled Indian audio, across many languages at once, so related languages share acoustic knowledge. Then a small amount of labelled data per language is enough to fine-tune a good recognizer.
How it works
- Data: ~17k hours of raw speech for 40 Indian languages, from public sources (mainly online videos and broadcasts) across domains like education, news, technology and finance.
- Pretraining: wav2vec 2.0 (a CNN feature encoder plus Transformer, trained to pick the true quantized speech unit for masked frames among distractors, a contrastive loss). The paper trains several sizes; check it for the exact base/large setup and whether models start from an existing English checkpoint.
- Fine-tuning: add a linear layer and train with CTC (CTC) on labelled data for each of 9 languages.
- Decoding: beam search with a KenLM n-gram language model, which gives a clear WER gain over greedy decoding in these lower-resource settings.
- Analysis: codebook entries for similar-sounding phonemes are shared across languages, layer representations cluster by language family (Indo-Aryan vs Dravidian), and attention heads often look at small local windows.
Build the intuition
Analogy: a child who grows up hearing Hindi, Marathi and Gujarati around them learns the shared sounds before learning to read any of them. Teaching them to read one language afterwards is fast.
What would happen if you pretrained only on English? The model's units would not cover retroflex consonants or aspirated stops well, and fine-tuning would have to learn them from scarce labels.
Numbers worth remembering
- ~17,000 hours of unlabelled audio, 40 languages.
- ASR fine-tuned and evaluated in 9 languages; state of the art on 3 public benchmarks (check the paper for which, likely including MUCS 2021 and the MSR Indian-language challenge sets), including very low-resource Sinhala and Nepali.
- Models, code and data released openly.
Where it shows up today
IndicWav2Vec is the base of much later Indian ASR: IndicSUPERB baselines (IndicSUPERB) and Vistaar comparisons (Vistaar). AI4Bharat's later IndicConformer models moved to Conformer encoders (Conformer). Meta's MMS (MMS) applied the same "wav2vec 2.0 + CTC per language" recipe to 1,000+ languages. The TTS counterpart from the same group is Indic TTS (AI4Bharat).
What it got wrong, and what came next
- Offline and non-streaming; CTC without a language model is weak on spelling and code-mixed words.
- Fine-tuned on read or narrow-domain speech, so it struggles on noisy, conversational phone audio. Kathbath (IndicSUPERB) and Vistaar (Vistaar) built broader benchmarks; IndicVoices (see IndicVoices-R) later collected much more natural speech.
- Whisper-style supervised scaling with Indic data (IndicWhisper, Vistaar) later beat it on many benchmarks.
Check yourself
Q1Why pretrain one multilingual model rather than one per language?
Indian languages share many sounds, and some have very little audio. A joint model lets low-resource languages borrow acoustic knowledge from high-resource ones, which the paper's codebook analysis supports.
Q2What does the n-gram LM add to a CTC model?
CTC predicts each frame independently given the audio, so it has no built-in knowledge of which words follow each other. Beam search with an n-gram LM rescores candidates with word statistics, fixing many spelling and word-choice errors.
Q3What is the finding about layer representations?
Representations from the pretrained layers group languages by family, so related languages end up close. It suggests the model learned linguistically meaningful structure from audio alone.
When you're back at a laptop
- Load an AI4Bharat IndicWav2Vec Hindi checkpoint from Hugging Face, transcribe a Hindi clip greedily, then with a KenLM via
pyctcdecode, and compare. - Evaluate WER on a few Kathbath Hindi test utterances.