ASR · Phase 8: Indian languages and accents
Svarah
Svarah: Evaluating English ASR Systems on Indian Accents
In one sentence
Svarah is a ~9.6-hour benchmark of Indian-accented English from 117 speakers across many regions of India, and it shows that both open and commercial English ASR systems leave clear room for improvement on Indian accents.
Why this paper exists
India has one of the largest English-speaking populations in the world, yet standard English benchmarks such as LibriSpeech (audiobooks) and Switchboard (US telephone calls) contain very few Indian speakers. A model reporting 2% WER on LibriSpeech can be much worse on a caller from Chennai or Guwahati, and nobody could measure it on a common test set.
The core idea
Build a small but diverse, carefully transcribed test set of Indian English, spread across regions and native languages, including both read and spontaneous speech, and evaluate popular systems on it.
How it works
- Speakers: 117 speakers from many states and native-language backgrounds (check the paper for the region and demographic breakdown).
- Content: read speech plus conversational answers on everyday topics such as history, culture and tourism, so it covers both prepared and spontaneous style.
- Size: ~9.6 hours of transcribed audio. It is an evaluation set, not training data.
- Evaluation: six open-source and two commercial ASR systems, scored with WER, broken down by region and speaker attributes.
Build the intuition
Analogy: testing a car only on smooth highways and then selling it in a city full of potholes. LibriSpeech is the highway; Svarah is a drive through Indian streets.
What would happen if you fine-tuned on Svarah itself? You would overfit a 9.6-hour test set and lose your only honest measurement. Keep it for evaluation; get training data elsewhere.
Why would accent hurt an English model? Indian English differs in vowel quality, rhythm (more syllable-timed), retroflex consonants and stress patterns, plus Indian names, places and code-mixed words. A model that rarely heard these maps them to the wrong words.
Numbers worth remembering
- ~9.6 hours, 117 speakers, many regions of India.
- 6 open-source + 2 commercial systems evaluated.
- Read and spontaneous speech both included.
- For per-system WERs, check the results table; the headline is that every system has clear room to improve.
Where it shows up today
Svarah is the go-to test for Indian-accented English. Use it next to Vistaar (Vistaar, Indian languages) and Kathbath (IndicSUPERB) when choosing ASR for an Indian voice agent: Whisper (Whisper), Parakeet / Fast Conformer models (Fast Conformer), Kyutai STT (Kyutai DSM) and commercial APIs can all be compared on it.
What it got wrong, and what came next
- Small: 9.6 hours is enough to rank systems but too small for fine-grained per-region conclusions.
- Mostly clean recordings: real calls add telephone bandwidth, noise and heavy code-mixing (Hinglish), which Svarah does not focus on.
- It measures the problem but does not fix it; fixing needs Indian-English training data and adaptation.
Check yourself
Q1Why are LibriSpeech results a poor guide for an Indian customer-support bot?
LibriSpeech is read audiobooks by mostly non-Indian speakers. Accent, vocabulary (Indian names, places) and speaking style differ, so WER there understates errors on Indian callers.
Q2Why include spontaneous speech, not just read sentences?
Spontaneous speech has hesitations, restarts and more natural accent and rhythm. It is closer to real voice-agent traffic, and models typically do worse on it than on read text.
Q3Why should Svarah not be used for training?
It is the shared evaluation set. Training on it would make your numbers incomparable with others and overstate your accuracy on unseen Indian speakers.
When you're back at a laptop
- Download Svarah from AI4Bharat (Hugging Face) and run
openai/whisper-smalland a Parakeet English model on it. - Compute WER overall and per region, with Whisper's English text normalizer applied to both reference and hypothesis.