SV2TTS
Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis
In one sentence
SV2TTS trains a speaker-verification network on thousands of noisy, untranscribed speakers, then conditions a Tacotron 2 synthesizer on its embedding, so a few seconds of a new person's speech are enough to synthesize in an approximation of their voice with no retraining.
Why this paper exists
Multi-speaker TTS before 2018 learned one embedding per training speaker (a lookup table). A new voice meant collecting transcribed audio and fine-tuning. TTS datasets are small, clean and transcribed, so they cover maybe a few hundred to a thousand speakers: far too few to learn a smooth "space of voices". Speaker verification, on the other hand, can use tens of thousands of speakers from cheap, noisy, untranscribed audio, because it only needs to know who is speaking. The idea: borrow the voice space from verification.
The core idea
Decouple "what does this voice sound like" from "how do I speak text". Train three networks independently:
- a speaker encoder that maps any clip to a fixed vector, trained for verification;
- a synthesizer (Tacotron 2, Tacotron 2) that maps text + that vector to a mel spectrogram;
- a vocoder (WaveNet, WaveNet) that maps mel to waveform, speaker-agnostic.
At inference, compute the vector from a few seconds of a never-seen speaker. If the embedding space is good, the synthesizer generalizes to that point.
How it works
ref clip (5 s) -> speaker encoder (3-layer LSTM, GE2E)
-> 256-d d-vector (L2-normalized)
|
text -> Tacotron 2 encoder -> concat d-vector to every state
-> attention decoder -> mel
-> WaveNet -> waveform- Speaker encoder: an LSTM stack (3 layers of 768 units with 256-d projections) over 40-channel log-mel frames, trained with the GE2E (generalized end-to-end) verification loss, which pulls embeddings of the same speaker toward their centroid and pushes them away from other speakers' centroids. The utterance embedding is the average of window embeddings over ~800 ms windows, L2-normalized. Trained on a large internal voice-search corpus (~18K speakers; check the paper).
- Synthesizer: Tacotron 2; the speaker embedding is concatenated with the encoder output at every input position. It is trained with the speaker encoder frozen, on a TTS corpus (VCTK or LibriSpeech), using each training utterance's own embedding.
- Vocoder: a sample-by-sample WaveNet trained on synthesizer outputs; it needs no speaker input because the mel already carries the voice.
- Evaluation: naturalness MOS and speaker similarity MOS (listeners rate whether two clips are the same speaker), plus verification equal error rate on synthesized speech.
Build the intuition
What would happen if you trained the speaker encoder jointly with the synthesizer on VCTK alone? With ~100 speakers, the encoder would memorize them; a new speaker would land somewhere arbitrary in the space and the synthesizer would produce one of the training voices or mush. Training on thousands of speakers fills the space densely, so a new voice lands between familiar ones and the synthesizer can interpolate. The paper's key ablation shows exactly this: similarity on unseen speakers improves as the encoder's training set grows.
A fun consequence: sample a random point on the unit sphere and you get a plausible voice that belongs to no one, a "fictitious speaker".
Analogy: the speaker encoder is a face-recognition model, the synthesizer is a portrait painter. The recogniser was trained on millions of faces so its "face code" generalizes; the painter only needs to learn to paint from that code.
For you: this is why "upload 10 seconds of your voice" features exist, and also why naive SV2TTS cloning on Indian-accented speakers can drift toward American-sounding voices: the synthesizer only saw the accents in its TTS corpus.
Numbers worth remembering
- Reference audio needed: a few seconds (they use ~5 s clips at inference).
- Speaker encoder trained on ~18K speakers of untranscribed data; synthesizer on VCTK (109 speakers) or LibriSpeech (~1.2K speakers).
- Naturalness of unseen speakers stays close to seen ones, but speaker similarity for unseen speakers is modest (similarity MOS around 3 on a 5-point scale; check the paper's tables).
Where it shows up today
- "Speaker embedding from a separate pretrained model" is still common: YourTTS (YourTTS) uses verification embeddings in VITS; XTTS (XTTS) and CosyVoice (CosyVoice → CosyVoice 2) feed a speaker embedding alongside the audio prompt.
- Speaker similarity (SIM) in modern papers is computed with a verification model (WavLM-based, see WavLM), the same tool used here as an encoder.
- Corentin Jemine's open "Real-Time Voice Cloning" repo implemented SV2TTS and popularized voice cloning around 2019.
What it got wrong, and what came next
- Limited similarity: a single 256-d vector cannot capture accent, speaking style or fine timbre. Audio-prompt models like VALL-E (VALL-E), Voicebox (Voicebox) and F5-TTS (F5-TTS) condition on the prompt's frames directly and get much higher similarity.
- Synthesizer data bottleneck: cloning quality is capped by the TTS corpus's speaker diversity; codec LMs trained on 50K+ hours (VALL-E) fixed this with scale.
- Inherits Tacotron 2's alignment problems and slow vocoder (Tacotron 2).
- Misuse risk: easy voice cloning raises consent and fraud concerns that the paper already discusses.
Check yourself
Q1Why train the speaker encoder separately instead of jointly with the synthesizer?
Verification can use tens of thousands of untranscribed, noisy speakers, while TTS corpora have far fewer clean, transcribed ones. A separately trained encoder learns a dense, general space of voices that a jointly trained one on a small TTS set would not.
Q2Why does the vocoder not need the speaker embedding?
The synthesizer's mel spectrogram already encodes the speaker's timbre and pitch, and the vocoder's job is only to turn that mel into a waveform. A vocoder trained on many speakers generalizes to new mels.
Q3What happens if you feed a random unit vector as the speaker embedding?
The synthesizer produces a plausible, natural-sounding voice that does not match any real speaker, because the embedding space is dense and smooth. The paper uses this to generate fictitious speakers.
Q4Why are unseen speakers less similar than seen ones?
The synthesizer only learned to map the embeddings of its training speakers to voices; for a new point it interpolates, and the single vector loses details such as accent and style. The result is a voice in the right neighbourhood rather than an exact match.
When you're back at a laptop
- Use
resemblyzer(a GE2E encoder) to embed 10 clips each from 3 speakers, plot with t-SNE, and compute cosine similarities within and across speakers. - Clone your own voice with an open SV2TTS-style or YourTTS model and with F5-TTS; compare similarity with the WavLM verification model.