TTS · Phase 9: Data, evaluation and Indian languages
Rasa
Rasa: Building Expressive Speech Synthesis Systems for Indian Languages in Low-resource Settings
In one sentence
Rasa is the first multilingual expressive TTS dataset for Indian languages (Assamese, Bengali, Tamil; neutral speech plus the six Ekman emotions), and its ablations give a budget recipe: lots of cheap neutral data plus a small amount of emotional data gets you most of the expressiveness.
Why this paper exists
Indian-language TTS (Indic TTS (AI4Bharat)) was trained on neutral, read studio speech. A neutral voice is fine for reading news, but a voice agent apologizing for a failed payment, or a storytelling app, needs emotion. Expressive data is expensive: you need voice actors who can produce convincing anger or sadness on demand, and for most Indian languages no such dataset existed. The practical question was not "can we make emotional TTS" but "how little expressive data can we get away with, and how should we spend a small recording budget?"
The core idea
Record neutral and emotional speech from the same speakers, then vary the amount of each and measure. If neutral data (easy to record) carries most of the language and voice learning, emotional data only needs to teach the style shift, so a small amount may be enough.
How it works
- Languages: Assamese, Bengali and Tamil (two Indo-Aryan, one Dravidian).
- Per language: about 10 hours of neutral speech and 1-3 hours for each of the six Ekman emotions: anger, disgust, fear, happiness, sadness, surprise.
- Model: an emotion-conditioned TTS model trained on the mix (check the paper for the architecture; AI4Bharat's earlier recipe was FastPitch + HiFi-GAN Indic TTS (AI4Bharat)).
- Ablations: vary hours of neutral data and minutes of expressive data; compare syllabically balanced vs unbalanced text; compare training each emotion separately vs pooling emotions.
- Evaluation: MUSHRA listening tests (MUltiple Stimuli with Hidden Reference and Anchor: listeners rate several systems for the same sentence on a 0-100 scale side by side, with a hidden real recording and a deliberately bad anchor mixed in to check the raters).
- Syllabically balanced text means prompts chosen so that the syllables of the language appear with good coverage, so a small dataset still contains most sound combinations.
Build the intuition
Analogy: teaching an actor a new language is the expensive part; once they speak it fluently, a short workshop teaches them to play it angry or sad. Neutral hours are the language lessons, emotional minutes are the workshop.
What would happen if you had 3 hours total and spent all of it on emotional speech? Each emotion would get about 30 minutes, the model would see few examples of most syllables, and it would mispronounce words while sounding emotional. The paper's finding is that the neutral base is the better first investment.
Numbers worth remembering
- 3 languages: Assamese, Bengali, Tamil.
- ~10 hours neutral + 1-3 hours per emotion, 6 emotions, per language.
- ~1 hour neutral + ~30 minutes expressive already gives a system rated "Fair" on MUSHRA.
- Raising neutral data to 10 hours with minimal expressive data significantly improves expressiveness.
- Hardest emotions to generate: fear and surprise.
Where it shows up today
- Expressive Indian data like Rasa feeds description-controlled models such as Indic Parler-TTS Parler-TTS (check the model card for its training sets).
- It complements large neutral multi-speaker data from IndicVoices-R IndicVoices-R; the "cheap base data + small targeted data" pattern is the same one used when fine-tuning large models such as F5-TTS F5-TTS for Indian languages.
- Style control in general traces back to GST Global Style Tokens and StyleTTS StyleTTS → StyleTTS 2.
What it got wrong, and what came next
- Only three languages and a few speakers; it does not cover Hindi-heavy or code-mixed use.
- Acted emotions are exaggerated compared to real conversation; a support agent needs mild empathy, not stage anger. In-the-wild data (LibriTTS → Emilia) and conversation-aware models (Sesame CSM Sesame CSM) aim at natural, context-fitting prosody.
- Six discrete emotions is a coarse control; text descriptions Parler-TTS and continuous style vectors StyleTTS → StyleTTS 2 give finer control.
Check yourself
Q1What is the main practical recipe from Rasa?
Prioritize neutral data, which is easy to get, and add a small amount of expressive data. About 1 hour of neutral plus 30 minutes of expressive data already gives a "Fair" system, and growing the neutral data to 10 hours improves expressiveness even with little emotional data.
Q2Why does pooling emotions help?
Each emotion alone has little data, so a model trained per emotion sees few examples. Pooling lets the model learn shared things (pronunciation, the speaker's voice, general prosodic variation) from all emotional data while the emotion label handles the differences.
Q3Why use MUSHRA instead of MOS here?
MUSHRA shows several systems for the same sentence side by side with a hidden reference and anchor, which makes small differences between ablation variants easier to detect and lets you screen out unreliable raters. MOS rates clips independently, which is coarser for comparing many close variants.
Q4Why might fear and surprise be hardest?
They are less frequent and more variable in how people express them, and their acoustic cues (breathiness, sudden pitch jumps, tense voice quality) are subtle and short. Acted recordings of them are also less consistent across sentences, so the model gets a noisier target.
When you're back at a laptop
- Download Rasa (AI4Bharat Hugging Face) and listen to the same sentence in neutral and each emotion; plot pitch contours with
librosa.pyinand compare the ranges. - Compute speaking rate and mean energy per emotion; see which emotions differ most from neutral.
- If you have time, fine-tune a small TTS model on Tamil neutral data, then add 30 minutes of one emotion and listen to the difference.