TTS · Phase 4: End-to-end and expressive TTS
YourTTS
YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for Everyone
In one sentence
YourTTS extends VITS with speaker-verification embeddings and language embeddings (plus a speaker-consistency loss), giving zero-shot multi-speaker, multilingual TTS and voice conversion from a short reference clip, before codec language models took over cloning.
Why this paper exists
VITS (VITS) gives excellent quality but knows only the speakers it was trained on, through a speaker ID table. SV2TTS (SV2TTS) had shown zero-shot cloning from a speaker-verification embedding, but on a weaker two-stage model. Most multi-speaker data is English; the authors wanted cloning that also works in languages with few speakers (they used Portuguese with a single speaker), and a model that could adapt to a new voice with very little data.
The core idea
Keep VITS, but condition it on an external speaker embedding computed from any reference clip by a pretrained speaker-verification model, and on a language embedding. Since the embedding space covers thousands of speakers, a new voice is just a new point in it.
How it works
- Input: raw characters, not phonemes, so languages without a good G2P (grapheme-to-phoneme converter) still work. A trainable language embedding is concatenated to each character embedding.
- Speaker embedding: from a pretrained speaker-verification network (an H/ASP model trained on VoxCeleb), fed to the flow, the decoder, the posterior encoder and the duration predictor.
- Speaker Consistency Loss (SCL): run the speaker encoder on the generated audio and on the ground truth, and add a loss on the negative cosine similarity of the two embeddings. This pushes the output to sound like the target speaker, not just be plausible speech.
- Data: English (VCTK, LibriTTS), Portuguese (one speaker) and French (M-AILABS).
- Voice conversion: encode the source audio with the posterior encoder (with the source speaker's embedding), pass it through the flow, and decode with the target speaker's embedding.
- Few-shot fine-tuning: with about a minute of a new speaker's audio, fine-tuning improves similarity noticeably.
Build the intuition
Analogy: VITS with speaker IDs is a choir where each singer has a numbered seat. YourTTS gives each singer a description card ("deep, nasal, fast") written by a voice expert. A new singer just needs a card, no seat.
What would happen in a language with one training speaker? Without other speakers in that language, a speaker-ID model cannot separate "Portuguese" from "this one person". The language embedding and the shared speaker space let the model learn speaker identity from English data and language from Portuguese, then combine them.
Numbers worth remembering
- Three languages in training: English, Portuguese, French.
- Reported state-of-the-art zero-shot results on VCTK at the time (check the paper's SECS and MOS tables).
- Fine-tuning works with around 1 minute of target speech (check the paper's setup).
Where it shows up today
- Shipped in Coqui TTS and widely used for cloning experiments until XTTS (XTTS), which kept the "speaker latent from a reference clip" idea on a GPT-style model.
- The speaker-consistency loss idea (use a speaker verifier as a training signal) reappears in many cloning systems, and speaker similarity (SIM) is now a standard metric (UTMOS).
- A pattern for Indian languages: share speakers across languages, add a language embedding, and work from characters when G2P is weak.
What it got wrong, and what came next
- Speaker similarity for unseen voices was modest by today's standards; a single fixed-size embedding cannot capture accent, emotion and recording conditions. Codec LMs like VALL-E (VALL-E) and flow-matching infilling (Voicebox, F5-TTS) condition on the prompt audio itself and clone far better.
- Character inputs in a multilingual model can mispronounce; later systems use larger data or text LLM backbones (CosyVoice → CosyVoice 2).
Check yourself
Q1How does YourTTS represent a new, unseen speaker?
With an embedding from a pretrained speaker-verification model, computed from a reference clip. The TTS model is conditioned on this vector instead of a learned speaker ID, so any clip gives a usable speaker representation.
Q2What does the speaker consistency loss do?
It runs the speaker encoder on the generated and real audio and penalizes low cosine similarity between their embeddings. That directly trains the model to reproduce the target voice's identity.
Q3Why use characters instead of phonemes?
Phonemes need a G2P system per language, which may not exist or be poor for low-resource languages. Characters work everywhere, at the cost of the model learning pronunciation itself.
When you're back at a laptop
pip install TTS(Coqui) and runtts_models/multilingual/multi-dataset/your_ttswith a 5-10 s clip of your voice in English; try the French and Portuguese outputs in your voice.- Compute speaker similarity between your clip and the output with a WavLM-based verifier and compare with XTTS later.