TTS · Phase 9: Data, evaluation and Indian languages
UTMOS
UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022
In one sentence
UTMOS predicts the mean opinion score a human panel would give a speech clip, using a fine-tuned self-supervised speech model ensembled with simple regressors; it ranked top on several metrics in the VoiceMOS Challenge 2022 and became the default automatic naturalness score in TTS papers. This page also covers the rest of the standard TTS evaluation toolkit.
Why this paper exists
ASR has one right answer, so WER (word error rate) measures it. TTS has many valid outputs for one text, so there is no reference to compare against. The gold standard is a listening test: many people rate clips, which takes days and money, and scores from different tests are not comparable. Researchers needed a cheap, repeatable number for "how natural does this sound" that tracks human ratings. The VoiceMOS Challenge 2022 posed exactly this task: given TTS and voice-conversion clips with human MOS labels, predict the MOS of unseen clips (a main in-domain track and an out-of-domain track with little labelled data).
The core idea
Self-supervised speech models (wav2vec 2.0 wav2vec 2.0, HuBERT HuBERT, WavLM WavLM) already encode what natural speech sounds like, because they were trained to model huge amounts of it. Fine-tune one to regress a MOS, add tricks that handle noisy human labels, and ensemble it with simple regressors for robustness.
How it works
UTMOS itself
- Strong learner: an SSL model (wav2vec 2.0-style) fine-tuned end to end. Its frame features go through a BLSTM and a linear head; frame scores are averaged into one utterance score. Tricks:
- Listener-dependent modelling: each rater has biases (some score everything a 3), so the model also takes a listener ID embedding during training and averages over listeners (or uses a "mean listener") at test time.
- Contrastive loss: besides plain regression, penalize getting the difference between two clips' scores wrong, roughly
L = max(0, |(y_i - y_j) - (ŷ_i - ŷ_j)| - α), whereyare true MOS,ŷpredictions,αa margin. Rankings matter more than absolute values. - Phoneme information from an ASR model and data augmentation (speed and pitch changes). Check the paper for the full list.
- Weak learners: classic regressors (for example ridge regression, SVR, gradient-boosted trees) trained on frozen SSL features.
- Stacking: a meta-regressor combines strong and weak learners. Most people today use only the strong learner checkpoint ("UTMOS22 strong"), which outputs a score on the 1-5 scale.
The TTS evaluation toolkit
| What | Metric | How |
|---|---|---|
| Naturalness | MOS | listeners rate 1-5; report mean and 95% confidence interval |
| Naturalness (A vs B) | CMOS | listeners hear both, rate -3 to +3; better at small differences |
| Voice match | SMOS | 1-5 rating of how similar the voice is to the reference |
| Naturalness (auto) | UTMOS | this paper; also DNSMOS for noise quality |
| Intelligibility | WER / CER | transcribe output with ASR, compare to input text |
| Speaker similarity | SIM | cosine of speaker embeddings, prompt vs output |
| Speed | RTF | synthesis time / audio duration |
| Latency | TTFA | time from text in to first audio out |
| Preference | arena | pairwise votes aggregated with Elo or Bradley-Terry |
Details that matter:
- WER via ASR: the usual judges are Whisper-large-v3 for English and multilingual, a HuBERT-large CTC model fine-tuned on LibriSpeech (VALL-E VALL-E, F5-TTS F5-TTS on LibriSpeech test-clean), and Paraformer for Chinese (Seed-TTS-eval Seed-TTS). Normalize text (case, punctuation, numbers) before scoring, or the number measures formatting. WER below about 2% on English is now normal; Kyutai quotes under 1%.
- SIM: usually a WavLM-large speaker-verification model (ECAPA-TDNN on WavLM features).
SIM = (e_p · e_o) / (|e_p| |e_o|), wheree_pande_oare the prompt and output embeddings. Papers report SIM-o (against the original prompt audio) or SIM-r (against the prompt after codec/vocoder resynthesis); SIM-r is easier, so check which one. - RTF:
RTF = generation time / audio length. RTF 0.1 means 10 s of audio in 1 s. State the hardware and batch size. - TTFA: what a voice agent feels. A model with great RTF can still have poor TTFA if it must see the whole sentence first (see Delayed Streams Modeling (TTS)).
- Arenas: TTS Arena and Artificial Analysis show two clips for the same text and ask which is better. They measure overall preference, not why.
Build the intuition
Analogy: UTMOS is a wine critic trained by tasting thousands of rated bottles. Reliable on wines like the ones it trained on, unreliable on a style it never tasted.
What would happen if you optimized a TTS model to maximize UTMOS directly? It would learn whatever UTMOS rewards, such as clean, studio-like, slightly flat audio, possibly at the cost of expressiveness, and could exploit its blind spots. This is why UTMOS is a proxy you report next to human MOS, not instead of it. The same holds for WER: a monotone robot voice can score 0% WER.
Numbers worth remembering
- MOS scale: 1 (bad) to 5 (excellent); recorded human speech typically scores ~4.3-4.6 in such tests.
- CMOS scale: -3 to +3; a difference of ~0.1-0.2 is often within noise, so look at the confidence interval.
- VoiceMOS 2022: UTMOS scored highest on several metrics in both the main and out-of-domain tracks (check the paper's tables).
- RTF < 1 is faster than real time; a voice agent wants TTFA of roughly 100-300 ms.
Where it shows up today
- UTMOS is reported in most recent TTS papers, alongside WER and SIM: F5-TTS F5-TTS, CosyVoice 2 CosyVoice → CosyVoice 2, Llasa / Spark-TTS Llasa + Spark-TTS, and Kyutai's work CALM (Pocket TTS), Delayed Streams Modeling (TTS).
- Seed-TTS-eval Seed-TTS standardized the WER + SIM pair on fixed prompt sets.
- Data pipelines filter training clips with automatic MOS predictors (Emilia LibriTTS → Emilia uses DNSMOS-style quality scores; IndicVoices-R IndicVoices-R needs similar filtering).
- The ASR side is the same machinery as ASR evaluation: Whisper Whisper as the judge, and its weaknesses on Indian languages Vistaar also bias TTS WER.
What it got wrong, and what came next
- UTMOS was trained on English and Japanese-heavy challenge data; on Indian languages, code-mixed speech or expressive speech its scores are much less trustworthy. Always check against a small human test.
- It scores clips in isolation, so it says nothing about whether the prosody fits the conversation (Sesame's point, Sesame CSM).
- UTMOSv2 (2024) and other newer predictors improved robustness; arena preferences and large human preference datasets (for example customer-support preference data) are the current direction for product decisions.
Check yourself
Q1Why can't you evaluate TTS just with WER?
WER only measures whether the words are recoverable. A flat, robotic or wrong-voice output can have perfect WER. You also need naturalness (MOS or UTMOS), speaker similarity for cloning, and speed and latency for products.
Q2What is the difference between MOS and CMOS, and when would you use CMOS?
MOS asks listeners for an absolute 1-5 rating of each clip. CMOS plays two systems side by side and asks for a relative -3 to +3 judgement. When two strong systems are close, absolute ratings bunch near the top and lose resolution, so CMOS detects small differences better.
Q3Why does UTMOS model individual listeners?
Raters have consistent personal biases: some are harsh, some generous. Without modelling them, the network treats that bias as label noise. A listener embedding lets the model explain rater bias separately, and at test time you predict for an average listener.
Q4What is the difference between SIM-o and SIM-r?
SIM-o compares the output to the original prompt recording. SIM-r compares it to the prompt after it was passed through the system's codec or vocoder, which removes some differences and inflates the score. Comparing a SIM-r number with a SIM-o number is misleading.
Q5A model has RTF 0.05 on your GPU but a voice agent built on it feels slow. What metric did you miss?
Time to first audio. RTF measures throughput for the whole utterance; if the model waits for the full sentence from the LLM, or generates everything before returning, the user waits for the LLM sentence plus the synthesis. A streaming model with worse RTF can feel much faster.
When you're back at a laptop
pip installthe SpeechMOS package and load UTMOS withtorch.hub.load("tarepan/SpeechMOS:v1.2.0", "utmos22_strong", trust_repo=True). Score a clean recording, the same clip with added noise, and a TTS clip.- Write a 30-line eval script: synthesize 50 sentences, transcribe with Whisper-large-v3, normalize text, compute WER with
jiwer, SIM with a WavLM speaker-verification model, UTMOS, and RTF and TTFA withtime.perf_counter(). - Run it on Hindi or Hinglish sentences and note where UTMOS and WER stop agreeing with your own ears.