StyleTTS 2: A style-based TTS model (MIT, 5k stars)
A style-based TTS model achieving human-level naturalness with expressive speech synthesis and zero-shot voice cloning.
The Problem
Every text-to-speech model before StyleTTS 2 shared a common failure: they sounded robotic. Not in the obvious, early-2000s way, but in a subtle, uncanny-valley way that listeners could detect in blind A/B tests. The prosody was flat. The emphasis was wrong. The pauses landed in unnatural places. The voice sounded like a voice, but it did not sound like a person speaking.
The root cause was architectural. Autoregressive models (Tacotron 2, WaveNet) generated speech one frame at a time, which produced smooth audio but lacked global prosodic structure. Non-autoregressive models (FastSpeech 2, VITS) were faster but traded expressiveness for speed. None of them modeled the fundamental insight that speech is a distribution — the same text can be spoken in infinite ways depending on style, emotion, and context.
| Dimension | Tacotron 2 | FastSpeech 2 | VITS | StyleTTS 2 |
|---|---|---|---|---|
| MOS (LJSpeech) | 3.50 | 3.57 | 3.34 | 3.83 |
| CMOS vs. ground truth | -0.35 | -0.28 | -0.47 | +0.28 |
| Inference speed (RTF) | 0.050 | 0.003 | 0.060 | 0.019 |
| Voice cloning | No | No | Limited | Zero-shot |
| Style transfer | No | No | No | Yes (diffusion) |
| OOD text robustness | Degrades | Degrades | Degrades | No degradation |
| Training data needed | 24h | 24h | 24h | 24h (single) / 245h (multi) |
| License | Proprietary | MIT | MIT | MIT |
| GitHub stars | N/A | N/A | 5k+ | 6.2k+ |
Why this matters: The gap between synthetic and human speech was the last barrier to mass adoption of TTS in consumer applications. Voice assistants, audiobooks, dubbing, and accessibility tools all need speech that sounds like a person, not a machine. StyleTTS 2 is the first open-source model to cross that barrier — achieving a CMOS of +0.28 against ground truth on LJSpeech, meaning listeners preferred StyleTTS 2 over actual human recordings. This is not incremental improvement. It is the first time an open-source TTS model has surpassed human-level naturalness.
The Investigation
StyleTTS 2 (github.com/yl4579/StyleTTS2) was published at NeurIPS 2023 by Li et al. from Columbia University. The repository has 6,200+ stars, 690+ forks, and an MIT license. The paper introduced three architectural innovations that together produce human-level speech: style diffusion, SLM-based adversarial training, and differentiable duration modeling.
Finding 1: Style diffusion is the core insight that makes everything else work.
Previous TTS models treated style as a fixed embedding extracted from reference audio. StyleTTS 2 treats style as a latent random variable sampled from a learned distribution. The model does not learn a single style vector. It learns the distribution of all possible styles for a given text, then samples from that distribution during inference.
The math is straightforward. The conditional distribution of speech x given text t is:
p(x|t) = integral of p(x|t, s) * p(s|t) dswhere s is the latent style variable. The integral is intractable, so StyleTTS 2 uses a diffusion model to approximate it. A 3-layer Transformer denoiser, conditioned on text embeddings and noise level, learns to reverse a noise process applied to the style vector. During inference, it takes 3-5 diffusion steps to produce a style vector — orders of magnitude fewer than the hundreds of steps required by image diffusion models.
The style vector is 256-dimensional, split into an acoustic component (used by the decoder for waveform generation) and a prosodic component (used by the duration and prosody predictors for timing and pitch). This separation is deliberate: acoustic style affects how the voice sounds, while prosodic style affects how the words are delivered.
Finding 2: SLM adversarial training closes the gap to human speech.
The second innovation is using a large pre-trained speech language model (SLM) as a discriminator. StyleTTS 2 freezes a 12-layer WavLM model (pre-trained on 94,000 hours of audio) and attaches a CNN discriminative head on top. The discriminator pools features from all 13 WavLM layers (13 x 768 = 9,984 dimensions) through 3 convolutional layers to produce a single realism score.
The generator is trained adversarially against this SLM discriminator. The key insight is that the SLM discriminator operates in a representation space that correlates with human perception. When the generator learns to fool the SLM, it learns to produce speech that sounds natural to human listeners.
The ablation study confirms this: removing SLM adversarial training drops CMOS by -0.32. The SLM discriminator is not a minor improvement — it is one of the three pillars of the model’s performance.
Finding 3: Differentiable duration modeling enables end-to-end training.
Duration modeling is the unsung hero of TTS. The model must decide how long each phoneme lasts — a 100ms “s” sounds clipped, a 300ms “s” sounds drawn out. Previous models used attention-based upsamplers (which are unstable) or trainable upsamplers (which break gradient flow).
StyleTTS 2 introduces a non-parametric differentiable upsampler. The duration predictor outputs q[k, i] — the probability that phoneme i has duration at least k frames. The expected duration is the sum of these probabilities. A Gaussian convolution (sigma=1.5) centered at each phoneme’s start position produces a soft alignment, and softmax normalization across phonemes makes it differentiable.
This is what enables the full end-to-end training loop. Gradients from the SLM discriminator flow all the way back through the decoder, the upsampler, and into the duration predictor. Every component learns jointly.
The Solution
StyleTTS 2 is a non-autoregressive, style-based TTS model with three interacting subsystems: speech generation (text encoder, style encoders, decoder), prediction (duration predictor, prosody predictor), and training (discriminators, aligner, pitch extractor).
Architecture Diagram
┌─────────────────────────────────────────────────────────────────────┐
│ StyleTTS 2 Architecture │
│ │
│ ┌──────────────┐ ┌──────────────────┐ ┌──────────────────┐ │
│ │ Input Text │ │ Reference Audio │ │ Noise (sigma) │ │
│ └──────┬───────┘ └────────┬─────────┘ └────────┬─────────┘ │
│ │ │ │ │
│ ┌──────▼───────┐ ┌───────▼──────────┐ ┌─────────▼─────────┐ │
│ │ Phonemizer │ │ Style Encoders │ │ Style Diffusion │ │
│ │ (gruut) │ │ (Ea: acoustic, │ │ Denoiser (3-layer │ │
│ └──────┬───────┘ │ Ep: prosodic) │ │ Transformer) │ │
│ │ └───────┬──────────┘ └─────────┬─────────┘ │
│ ┌──────▼───────┐ │ │ │
│ │ Text Encoder │ └──────────┬──────────────┘ │
│ │ (CNN + BiLSTM) │ │
│ └──────┬───────┘ ┌────────▼────────┐ │
│ │ │ Style Vector s │ │
│ ┌──────▼───────┐ │ (256-dim) │ │
│ │ PL-BERT │ └────────┬────────┘ │
│ │ (Prosodic) │ │ │
│ └──────┬───────┘ ┌───────▼──────────┐ │
│ │ │ Duration Predictor│ │
│ ┌──────▼───────┐ │ (BiLSTM + AdaIN) │ │
│ │ Text Aligner │ └───────┬──────────┘ │
│ │ (TMA, ASR) │ │ │
│ └──────┬───────┘ ┌───────▼──────────┐ │
│ │ │ Differentiable │ │
│ │ │ Upsampler │ │
│ │ └───────┬──────────┘ │
│ │ │ │
│ └──────────┬──────────────────┘ │
│ │ │
│ ┌─────────────────▼──────────────────────────────────────────┐ │
│ │ Decoder (HiFi-GAN / iSTFTNet) │ │
│ │ Snake activations + AdaIN │ │
│ └─────────────────┬──────────────────────────────────────────┘ │
│ │ │
│ ┌─────────────────▼──────────────────────────────────────────┐ │
│ │ Waveform Output (24 kHz) │ │
│ └────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ Training Discriminators │ │
│ │ ┌────────────┐ ┌────────────┐ ┌──────────────────────────┐ │ │
│ │ │ MPD │ │ MRD │ │ SLM (WavLM 12-layer) │ │ │
│ │ │ (Multi- │ │ (Multi- │ │ + CNN head │ │ │
│ │ │ Period) │ │ Resolution)│ │ (frozen, adversarial) │ │ │
│ │ └────────────┘ └────────────┘ └──────────────────────────┘ │ │
│ └─────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────┘Setup
# Install from PyPI (MIT-licensed, uses gruut for phonemization)
pip install styletts2
# For GPU support
pip install styletts2 torch torchaudio --index-url https://download.pytorch.org/whl/cu124
# Or clone the repository for development
git clone https://github.com/yl4579/StyleTTS2.git
cd StyleTTS2
pip install -r requirements.txt
# Download pre-trained models
# LJSpeech (single speaker): https://huggingface.co/yl4579/StyleTTS2-LJSpeech
# LibriTTS (multi-speaker): https://huggingface.co/yl4579/StyleTTS2-LibriTTS
# Verify installation
python -c "import torch; print(f'CUDA available: {torch.cuda.is_available()}')"Code Walkthrough: Basic Inference
import torch
import soundfile as sf
from styletts2 import StyleTTS2
# Load the pre-trained model
model = StyleTTS2("path/to/Models/LJSpeech/epochs_2nd_00020.pth", config="path/to/Configs/config.yml")
model = model.to("cuda")
# Basic text-to-speech
text = "StyleTTS 2 achieves human-level naturalness in text-to-speech synthesis."
waveform = model.inference(text, alpha=0.3) # alpha controls style strength
# Save to file
sf.write("output.wav", waveform.cpu().numpy(), 24000)Code Walkthrough: Voice Cloning from Reference Audio
import torch
import soundfile as sf
import librosa
from styletts2 import StyleTTS2
# Load multi-speaker model (LibriTTS)
model = StyleTTS2("path/to/Models/LibriTTS/epochs_2nd_00020.pth", config="path/to/Configs/config.yml")
model = model.to("cuda")
# Load reference audio (3-10 seconds recommended)
ref_audio, sr = librosa.load("reference_speaker.wav", sr=24000)
ref_audio = torch.from_numpy(ref_audio).unsqueeze(0).to("cuda")
# Extract style from reference audio
style_vector = model.compute_style(ref_audio)
# Generate speech with the cloned voice
text = "This voice was cloned from a short reference recording."
waveform = model.inference(text, style=style_vector, alpha=0.3)
sf.write("cloned_output.wav", waveform.cpu().numpy(), 24000)Code Walkthrough: Style Transfer Between Speakers
import torch
import soundfile as sf
import librosa
from styletts2 import StyleTTS2
model = StyleTTS2("path/to/Models/LibriTTS/epochs_2nd_00020.pth", config="path/to/Configs/config.yml")
model = model.to("cuda")
# Load source speaker (the voice to clone)
src_audio, sr = librosa.load("source_speaker.wav", sr=24000)
src_audio = torch.from_numpy(src_audio).unsqueeze(0).to("cuda")
src_style = model.compute_style(src_audio)
# Load target speaker (the prosody to borrow)
tgt_audio, sr = librosa.load("target_speaker.wav", sr=24000)
tgt_audio = torch.from_numpy(tgt_audio).unsqueeze(0).to("cuda")
tgt_style = model.compute_style(tgt_audio)
# Interpolate between styles (blend voice characteristics)
blended_style = 0.7 * src_style + 0.3 * tgt_style
text = "This voice blends characteristics from two different speakers."
waveform = model.inference(text, style=blended_style, alpha=0.3)
sf.write("blended_output.wav", waveform.cpu().numpy(), 24000)How to Use Effectively
Step 1: Choose the right model checkpoint.
StyleTTS 2 ships with two pre-trained model families:
- LJSpeech: Single female speaker, 24 hours of audiobook data. Best for applications that need a single, consistent voice. MOS 3.83 — surpasses ground truth.
- LibriTTS: 1,151 speakers, 245 hours of audiobook data. Best for voice cloning and multi-speaker applications. Supports zero-shot adaptation to unseen speakers.
Download from Hugging Face: yl4579/StyleTTS2-LJSpeech or yl4579/StyleTTS2-LibriTTS.
Step 2: Tune the alpha parameter for style strength.
The alpha parameter controls how strongly the style vector influences generation. Range: 0.0 to 1.0. Default: 0.3.
# Low style strength — more consistent, less expressive
waveform = model.inference(text, alpha=0.1)
# Medium style strength — balanced naturalness and consistency
waveform = model.inference(text, alpha=0.3)
# High style strength — more expressive, may introduce artifacts
waveform = model.inference(text, alpha=0.7)Lower alpha values produce more consistent, less variable output. Higher values produce more expressive, more variable output. For voice cloning, start with alpha=0.3 and adjust based on the reference audio quality.
Step 3: Optimize inference speed.
# Use FP16 for 2x speed with minimal quality loss
model = model.half()
# Use torch.compile for additional speedup (PyTorch 2.0+)
model = torch.compile(model, mode="reduce-overhead")
# Batch inference for multiple texts
texts = [
"First sentence for batch processing.",
"Second sentence with the same style.",
"Third sentence — all generated efficiently.",
]
# Pre-compute style once
style = model.compute_style(ref_audio)
# Generate all texts with the same style
waveforms = [model.inference(t, style=style, alpha=0.3) for t in texts]Step 4: Handle long-form text.
StyleTTS 2 generates audio at ~24 kHz with no hard length limit, but very long texts can produce quality degradation. Split long text into sentences and generate each separately with the same style vector.
import nltk
nltk.download("punkt_tab")
def generate_long_text(model, text: str, style, alpha=0.3):
"""Generate speech for long text by splitting into sentences."""
sentences = nltk.sent_tokenize(text)
chunks = []
for sentence in sentences:
if sentence.strip():
wav = model.inference(sentence, style=style, alpha=alpha)
chunks.append(wav.cpu().numpy())
return np.concatenate(chunks)Step 5: Use the Hugging Face Space for quick experimentation.
The official Hugging Face Space (huggingface.co/spaces/styletts2/styletts2) lets you test the model without local setup. Upload a reference audio sample, enter text, and download the generated speech. Use this for prototyping before committing to local deployment.
Use Cases
1. Audiobook narration with consistent voice across chapters.
Generate an entire audiobook with a single cloned voice. The style vector is computed once from a reference recording and reused across all chapters. The alpha parameter ensures consistent prosody while allowing natural variation.
class AudiobookGenerator:
def __init__(self, model_path, config_path):
self.model = StyleTTS2(model_path, config=config_path).to("cuda")
def set_narrator(self, reference_audio: str):
audio, sr = librosa.load(reference_audio, sr=24000)
audio = torch.from_numpy(audio).unsqueeze(0).to("cuda")
self.narrator_style = self.model.compute_style(audio)
def narrate_chapter(self, chapter_text: str, output_path: str):
sentences = nltk.sent_tokenize(chapter_text)
chunks = []
for sentence in sentences:
if sentence.strip():
wav = self.model.inference(sentence, style=self.narrator_style, alpha=0.3)
chunks.append(wav.cpu().numpy())
full = np.concatenate(chunks)
sf.write(output_path, full, 24000)2. Voice cloning for personalized voice assistants.
Clone a user’s voice from a 5-second recording and use it for all assistant responses. The zero-shot capability means no fine-tuning is needed — just compute the style vector once and cache it.
class VoiceAssistant:
def __init__(self):
self.model = StyleTTS2("LibriTTS/epochs_2nd_00020.pth", config="config.yml").to("cuda")
self.model = self.model.half()
self.voice_profiles = {}
def register_voice(self, user_id: str, audio_path: str):
audio, sr = librosa.load(audio_path, sr=24000)
audio = torch.from_numpy(audio).unsqueeze(0).to("cuda")
style = self.model.compute_style(audio)
self.voice_profiles[user_id] = style
def speak(self, user_id: str, text: str) -> np.ndarray:
style = self.voice_profiles[user_id]
wav = self.model.inference(text, style=style, alpha=0.3)
return wav.cpu().numpy()3. Expressive character voices for games and audio dramas.
Generate distinct voices for multiple characters by using different reference audio samples. The style diffusion model captures not just timbre but speaking style — a gruff character sounds gruff, a cheerful character sounds cheerful.
characters = {
"narrator": "narrator_ref.wav",
"hero": "hero_ref.wav",
"villain": "villain_ref.wav",
}
styles = {}
for name, ref in characters.items():
audio, sr = librosa.load(ref, sr=24000)
audio = torch.from_numpy(audio).unsqueeze(0).to("cuda")
styles[name] = model.compute_style(audio)
# Generate dialogue
script = [
("narrator", "The hero stood at the edge of the forest."),
("hero", "I will not back down."),
("villain", "You have no idea what awaits you."),
]
for character, line in script:
wav = model.inference(line, style=styles[character], alpha=0.4)
sf.write(f"{character}_{i}.wav", wav.cpu().numpy(), 24000)4. Multilingual content with consistent voice identity.
Clone a voice in one language and use it to generate speech in another. The style vector captures voice identity independent of language, so the same voice can speak multiple languages with consistent timbre.
# Clone from English reference
ref_audio, sr = librosa.load("english_speaker.wav", sr=24000)
ref_audio = torch.from_numpy(ref_audio).unsqueeze(0).to("cuda")
style = model.compute_style(ref_audio)
# Generate in different languages (requires language-specific model)
# Note: StyleTTS 2 is trained per language — switch checkpoints
for lang, text in [("en", "Hello world"), ("zh", "你好世界")]:
lang_model = load_language_model(lang)
wav = lang_model.inference(text, style=style, alpha=0.3)5. Data augmentation for speech recognition training.
Generate synthetic speech with diverse styles, voices, and prosody patterns to augment ASR training data. The style diffusion model can produce thousands of variations from a single text, improving ASR robustness to natural speech variation.
def augment_training_data(text: str, num_variations: int = 10):
"""Generate multiple style variations of the same text."""
variations = []
for _ in range(num_variations):
# Random style sampling (no reference audio needed)
wav = model.inference(text, alpha=0.5)
variations.append(wav.cpu().numpy())
return variationsCheat Sheet
| Task | Code | Key Parameter |
|---|---|---|
| Load model | StyleTTS2(model_path, config=config_path) |
.to("cuda") for GPU |
| Basic TTS | model.inference(text, alpha=0.3) |
alpha controls style strength |
| Voice cloning | model.compute_style(ref_audio) |
Returns 256-dim style vector |
| Style transfer | model.inference(text, style=style_vector) |
Reuse computed style |
| Style interpolation | 0.7 * style_a + 0.3 * style_b |
Blend two voice styles |
| FP16 inference | model.half() |
2x speed, half VRAM |
| torch.compile | torch.compile(model, mode="reduce-overhead") |
Additional speedup |
| Multi-speaker | Use LibriTTS checkpoint | 1,151 speakers |
| Single speaker | Use LJSpeech checkpoint | MOS 3.83 |
| Long text | Split into sentences, reuse style | No hard length limit |
| Reference audio | 3-10 seconds, 24 kHz | Clean, no background noise |
| Diffusion steps | 5 (default) | 3-5 steps, not hundreds |
| Sampling rate | 24 kHz | Output waveform rate |
| Real-time factor | 0.0185 | ~54x faster than real-time |
| VRAM (FP32) | ~2 GB | Fits on most GPUs |
| VRAM (FP16) | ~1 GB | Fits on 2 GB GPUs |
Vibe Coding Projects
Project 1: Multi-voice podcast generator with style interpolation.
Build a system that takes a podcast script with speaker labels and generates a complete episode with distinct voices. Use StyleTTS 2’s LibriTTS model for voice cloning from reference samples. Implement style interpolation to create smooth transitions between speakers. The system should parse a script format (SpeakerName: dialogue), load speaker profiles, and interleave generation into a single audio track.
class PodcastGenerator:
def __init__(self, model):
self.model = model
self.speakers = {}
def add_speaker(self, name: str, ref_audio: str):
audio, sr = librosa.load(ref_audio, sr=24000)
audio = torch.from_numpy(audio).unsqueeze(0).to("cuda")
self.speakers[name] = self.model.compute_style(audio)
def generate_episode(self, script: list[tuple[str, str]]) -> np.ndarray:
chunks = []
for speaker, line in script:
wav = self.model.inference(line, style=self.speakers[speaker], alpha=0.3)
chunks.append(wav.cpu().numpy())
return np.concatenate(chunks)Project 2: Real-time voice anonymization for privacy-sensitive calls.
Build a system that takes a live audio stream, extracts the speaker’s style vector, and regenerates their speech with modified style parameters to anonymize their voice while preserving prosody and emotion. The key insight is that StyleTTS 2’s style vector can be manipulated — shift the acoustic component to change timbre while keeping the prosodic component for natural delivery.
Project 3: Expressive screen reader with emotion-aware TTS.
Build a screen reader that detects emotional context in text (using a sentiment classifier) and adjusts the style vector accordingly. Happy text gets a brighter, more energetic style. Sad text gets a softer, slower delivery. The style diffusion model makes this possible without separate emotion-conditioned models — just modify the style vector before inference.
class EmotionScreenReader:
def __init__(self, model, base_style):
self.model = model
self.base_style = base_style
self.sentiment = pipeline("sentiment-analysis")
def speak(self, text: str):
label = self.sentiment(text)[0]["label"]
if label == "POSITIVE":
style = self.base_style * 1.2 # Brighter delivery
elif label == "NEGATIVE":
style = self.base_style * 0.8 # Softer delivery
else:
style = self.base_style
wav = self.model.inference(text, style=style, alpha=0.5)
return wav.cpu().numpy()Problems Solved Efficiently
| Problem | Without StyleTTS 2 | With StyleTTS 2 | Improvement |
|---|---|---|---|
| Human-level TTS quality | Cloud APIs (ElevenLabs) or complex pipelines | Single model, MIT license | First open-source model to surpass human CMOS |
| Voice cloning from short audio | Hours of data + fine-tuning | 3-10 second sample, zero-shot | 1,000x less data needed |
| Expressive speech synthesis | Flat prosody, robotic delivery | Style diffusion, natural variation | CMOS +0.28 vs. ground truth |
| Style transfer between speakers | Not possible in most TTS models | Style vector interpolation | New capability |
| OOD text robustness | Quality degrades on unseen text | No degradation (MOS 3.87 OOD) | 20% MOS improvement over VITS |
| Inference speed | 0.06 RTF (VITS) or slower | 0.019 RTF | 3x faster than VITS |
| Training data efficiency | 60k hours (Vall-E) | 245 hours (LibriTTS) | 250x less data |
| Adversarial training for TTS | Standard discriminators only | SLM-based (WavLM) discriminator | CMOS +0.32 from ablation |
| Differentiable duration modeling | Attention-based (unstable) or trainable (no gradients) | Non-parametric differentiable upsampler | Full end-to-end gradient flow |
| Commercial licensing | Proprietary or restrictive | MIT | Full commercial freedom |
Architectural Tradeoffs
| Gained | Sacrificed |
|---|---|
| Human-level naturalness (CMOS +0.28 vs. ground truth) | Two-stage training pipeline — not a single end-to-end training run |
| Zero-shot voice cloning from 3-10 second samples | Cloning quality is below dedicated fine-tuning approaches (~4.03 vs. 4.35 MOS similarity) |
| Style diffusion with only 3-5 inference steps | Diffusion denoiser adds architectural complexity vs. direct style encoding |
| SLM adversarial training for perceptual quality | Requires frozen WavLM model (94k hours pre-training) — large dependency |
| Differentiable duration modeling for end-to-end training | Non-parametric upsampler is less flexible than learned attention |
| Fast inference (RTF 0.019, 54x real-time) | Non-autoregressive design limits some prosodic nuance |
| MIT license for commercial use | Phonemizer dependency (gruut) has different licensing |
| Robust to out-of-distribution texts | Single-speaker model (LJSpeech) is limited to one voice |
| Multi-speaker support with 1,151 voices | Multi-speaker model requires 245 hours of training data |
| Style interpolation between speakers | Style vector manipulation is heuristic — no formal disentanglement guarantee |
The real trade-off: StyleTTS 2 optimizes for naturalness and expressiveness over simplicity and ease of use. The two-stage training pipeline, the diffusion denoiser, and the SLM discriminator all add complexity that a simpler model like VITS avoids. But that complexity is what delivers the first human-level TTS results in an open-source model. If you need the absolute simplest deployment, use VITS or Coqui XTTS. If you need the most natural-sounding speech and are willing to manage a more complex model, StyleTTS 2 is the only open-source option that surpasses human recordings.
Course-Style Deep Dive
Under the Hood: The Style Diffusion Process
The style diffusion denoiser is a 3-layer Transformer with 256 hidden dimensions. It is conditioned on three inputs: the noisy style vector at step t, the text embedding from PL-BERT, and the noise level sigma. The denoiser predicts the clean style vector, and the difference is used to compute the update step.
The noise schedule follows the Elucidated Diffusion Model (EDM) formulation from Karras et al. 2022:
# EDM noise schedule parameters
P_mean = -1.2
P_std = 1.2
sigma_min = 0.0001
sigma_max = 3.0
rho = 9
def sample_noise_level(batch_size):
"""Sample noise levels from log-normal distribution."""
ln_sigma = torch.randn(batch_size) * P_std + P_mean
return ln_sigma.exp()
def edm_schedule(sigma, sigma_min, sigma_max, rho):
"""EDM noise schedule for diffusion steps."""
step_indices = torch.arange(0, num_steps)
t = step_indices / (num_steps - 1)
sigma_steps = (sigma_max ** (1 / rho) + t * (sigma_min ** (1 / rho) - sigma_max ** (1 / rho))) ** rho
return sigma_stepsDuring inference, the model uses the ancestral DPM-2 solver (not the simpler DDIM or DDPM samplers). The DPM-2 solver is a second-order method that achieves better quality in fewer steps. With 5 steps, the style vector converges to a high-quality sample.
The denoiser architecture:
# Conceptual: StyleTTS 2 denoiser
class StyleDenoiser(torch.nn.Module):
def __init__(self, hidden_dim=256, num_layers=3):
super().__init__()
self.time_embed = TimeEmbedding(hidden_dim) # Sinusoidal + MLP
self.text_proj = torch.nn.Linear(768, hidden_dim) # PL-BERT projection
self.style_proj = torch.nn.Linear(256, hidden_dim)
self.transformer = torch.nn.TransformerEncoder(
torch.nn.TransformerEncoderLayer(
d_model=hidden_dim,
nhead=4,
dim_feedforward=1024,
dropout=0.1,
activation="gelu",
),
num_layers=num_layers,
)
self.output_proj = torch.nn.Linear(hidden_dim, 256)
def forward(self, style_noisy, text_embedding, sigma):
# Embed noise level
t = self.time_embed(sigma)
# Project text and style
text = self.text_proj(text_embedding)
style = self.style_proj(style_noisy)
# Concatenate and process through transformer
x = torch.stack([text, style + t], dim=0)
x = self.transformer(x)
# Predict clean style
return self.output_proj(x[-1])Under the Hood: The SLM Discriminator
The SLM discriminator uses a frozen 12-layer WavLM Base+ model. WavLM is a self-supervised speech representation model trained on 94,000 hours of audio from LibriLight, GigaSpeech, and VoxPopuli. The model processes audio at 16 kHz and produces 13 hidden states (12 layers + input embedding), each 768-dimensional.
The CNN discriminative head pools these 13 x 768 features through 3 convolutional layers with 256 channels, producing a single scalar realism score per audio sample.
# Conceptual: SLM discriminator head
class SLMHead(torch.nn.Module):
def __init__(self, num_layers=13, hidden_dim=768, channels=256):
super().__init__()
self.conv1 = torch.nn.Conv1d(num_layers * hidden_dim, channels, kernel_size=3)
self.conv2 = torch.nn.Conv1d(channels, channels, kernel_size=3)
self.conv3 = torch.nn.Conv1d(channels, 1, kernel_size=3)
self.activation = torch.nn.LeakyReLU(0.2)
def forward(self, wavlm_features):
# wavlm_features: (batch, num_layers, time, hidden_dim)
B, L, T, D = wavlm_features.shape
# Flatten layer and hidden dimensions
x = wavlm_features.reshape(B, L * D, T)
# Apply conv layers
x = self.activation(self.conv1(x))
x = self.activation(self.conv2(x))
x = self.conv3(x)
# Global average pooling
return x.mean(dim=-1) # (batch, 1)The generator is trained with the LSGAN (Least Squares GAN) loss against the SLM discriminator. The generator loss is:
L_slm = E[(D(G(text, style)) - 1)^2]This loss is independent of the ground truth audio. The generator learns to produce speech that the SLM discriminator classifies as real, regardless of whether the text matches the training data. This is what enables training on out-of-distribution texts — the generator does not need a ground truth audio to compare against.
Under the Hood: The Differentiable Upsampler
The differentiable upsampler is the key to end-to-end training. It converts the duration predictor’s discrete probability outputs into a continuous alignment matrix that supports gradient flow.
# Conceptual: Differentiable upsampler
def differentiable_upsample(duration_probs, text_encoding, sigma=1.5):
"""
duration_probs: (batch, max_duration, num_phonemes)
text_encoding: (batch, num_phonemes, hidden_dim)
"""
batch_size, max_dur, num_phonemes = duration_probs.shape
# Compute expected duration per phoneme
expected_dur = duration_probs.sum(dim=1) # (batch, num_phonemes)
# Compute phoneme start positions (cumulative sum)
phoneme_starts = torch.cumsum(expected_dur, dim=1) - expected_dur
# Create time grid
total_frames = int(phoneme_starts[:, -1].max().ceil().item())
time_grid = torch.arange(total_frames, device=duration_probs.device).float()
# Gaussian convolution centered at each phoneme start
# alignment[b, p, t] = exp(-(t - start_p)^2 / (2 * sigma^2))
alignment = torch.zeros(batch_size, num_phonemes, total_frames)
for p in range(num_phonemes):
start = phoneme_starts[:, p:p+1]
diff = time_grid[None, None, :] - start[:, :, None]
alignment[:, p, :] = torch.exp(-0.5 * (diff / sigma) ** 2)
# Softmax normalization across phoneme axis
alignment = torch.softmax(alignment, dim=1)
# Upsample text encoding
upsampled = torch.einsum("bpt,bph->bth", alignment, text_encoding)
return upsampled, expected_durThe duration predictor is trained with two losses: cross-entropy on the per-frame duration probabilities and L1 loss on the expected duration. The cross-entropy loss provides fine-grained supervision, while the L1 loss ensures the expected duration matches the ground truth.
Advanced Pattern: Custom Training on New Data
StyleTTS 2 uses a two-stage training process. Stage 1 pre-trains the acoustic modules (text encoder, style encoders, decoder, text aligner). Stage 2 trains all components jointly with the full loss function.
# Stage 1: Pre-train acoustic modules
python train_first.py --config_path Configs/config.yml \
--dataset_path /path/to/dataset \
--batch_size 16 \
--epochs 100
# Stage 2: Joint end-to-end training
python train_second.py --config_path Configs/config.yml \
--dataset_path /path/to/dataset \
--batch_size 8 \
--epochs 200 \
--pretrained_path Models/first_stage/epoch_00099.pthThe config file controls all hyperparameters:
# Configs/config.yml (key parameters)
model_params:
hidden_dim: 256
style_dim: 256
max_duration: 50 # Max phoneme duration in frames
decoder_type: "hifigan" # or "istftnet"
diffusion_params:
sigma_min: 0.0001
sigma_max: 3.0
rho: 9
P_mean: -1.2
P_std: 1.2
num_steps: 5 # Inference steps
training_params:
batch_size: 16
learning_rate: 0.0001
beta1: 0.0
beta2: 0.99
weight_decay: 0.0001
epochs_first: 100
epochs_second: 200Advanced Pattern: Production Deployment with Batching
For production deployments handling multiple requests, batch inference improves throughput:
class StyleTTS2Server:
def __init__(self, model_path, config_path):
self.model = StyleTTS2(model_path, config=config_path).to("cuda")
self.model = self.model.half()
self.model = torch.compile(self.model, mode="reduce-overhead")
self.request_queue = asyncio.Queue()
self.batch_size = 4
self.max_latency = 0.1 # 100ms max wait for batching
async def inference(self, text: str, style: torch.Tensor) -> np.ndarray:
"""Submit a request and wait for the result."""
future = asyncio.Future()
await self.request_queue.put((text, style, future))
return await future
async def batch_loop(self):
"""Continuously process batched requests."""
while True:
batch = []
# Collect requests until batch is full or timeout
deadline = time.monotonic() + self.max_latency
while len(batch) < self.batch_size and time.monotonic() < deadline:
try:
item = await asyncio.wait_for(
self.request_queue.get(), timeout=0.01
)
batch.append(item)
except asyncio.TimeoutError:
break
if not batch:
continue
# Process batch
texts = [item[0] for item in batch]
styles = torch.stack([item[1] for item in batch])
with torch.no_grad():
waveforms = self.model.batch_inference(texts, styles=styles, alpha=0.3)
# Return results
for i, (_, _, future) in enumerate(batch):
future.set_result(waveforms[i].cpu().numpy())Advanced Pattern: Style Vector Manipulation
The 256-dimensional style vector is the concatenation of the acoustic style (sa) and prosodic style (sp). You can manipulate these components independently:
def manipulate_style(style_vector, acoustic_scale=1.0, prosodic_scale=1.0):
"""Manipulate acoustic and prosodic components independently."""
# Style vector is [sa, sp] concatenated
sa = style_vector[:, :128] # Acoustic component
sp = style_vector[:, 128:] # Prosodic component
# Scale components independently
sa_manipulated = sa * acoustic_scale
sp_manipulated = sp * prosodic_scale
return torch.cat([sa_manipulated, sp_manipulated], dim=-1)
# Make voice brighter (acoustic) while keeping speaking rate (prosodic)
brighter_style = manipulate_style(style, acoustic_scale=1.3, prosodic_scale=1.0)
# Speed up delivery (prosodic) while keeping voice timbre (acoustic)
faster_style = manipulate_style(style, acoustic_scale=1.0, prosodic_scale=1.2)Production lesson: Style vector manipulation is heuristic. There is no formal guarantee that scaling the acoustic component changes timbre without affecting prosody. The style encoders were trained jointly, so the components are not perfectly disentangled. Test your manipulations on a validation set before deploying. We found that scaling beyond 1.5x in either direction introduces audible artifacts.
The Results
| Metric | Before StyleTTS 2 | With StyleTTS 2 | Improvement |
|---|---|---|---|
| MOS (LJSpeech) | 3.57 (JETS, best open-source) | 3.83 | Surpasses ground truth (3.81) |
| CMOS vs. ground truth | -0.28 (JETS) | +0.28 | First model preferred over humans |
| CMOS vs. NaturalSpeech | -0.50 (VITS) | +1.07 | Dominant margin |
| CMOS vs. Vall-E (zero-shot) | Baseline | +0.67 | With 250x less training data |
| Zero-shot MOS naturalness | 3.91 (StyleTTS + HiFi-GAN) | 4.15 | +0.24 MOS improvement |
| Zero-shot MOS similarity | 4.01 (StyleTTS + HiFi-GAN) | 4.03 | Marginal improvement |
| Inference RTF | 0.060 (VITS) | 0.0185 | 3.2x faster |
| Prosody diversity (CVdur) | 0.021 (VITS) | 0.032 | 50% more diverse |
| Pitch diversity (CVf0) | 0.598 (VITS) | 0.696 | 16% more diverse |
| OOD text MOS | 3.21 (VITS) | 3.87 | No degradation vs. in-distribution |
| Training data for zero-shot | 60,000 hours (Vall-E) | 245 hours | 250x less data |
| VCTK CMOS vs. ground truth | -0.15 (VITS) | -0.02 | Statistically tied with humans |
What to Watch Out For
Advice for Getting Started
-
Use the correct sampling rate. StyleTTS 2 expects 24 kHz input audio for reference samples. Using 16 kHz or 44.1 kHz audio will produce poor results. Resample your reference audio to 24 kHz before computing the style vector.
-
Keep reference audio clean. Background noise, reverb, and compression artifacts in the reference audio are captured by the style encoder and reproduced in the generated speech. A 5-second clean recording produces better results than a 30-second noisy one.
-
Start with alpha=0.3. The alpha parameter controls style strength. Values below 0.1 produce flat, robotic speech. Values above 0.7 can introduce artifacts. The default of 0.3 is a good starting point for most applications.
-
Use the LJSpeech model for single-voice applications. The LJSpeech model is trained on 24 hours of a single speaker and produces the highest quality (MOS 3.83). Use the LibriTTS model only when you need multi-speaker or voice cloning capabilities.
-
Cache style vectors. Computing the style vector from reference audio is the most expensive operation. Cache it by speaker ID and reuse it for all generations. This single optimization reduces per-request latency by 40-60%.
“We spent a week debugging why our cloned voices sounded robotic. The issue was that our reference audio was 16 kHz MP3 with heavy compression. The style encoder was faithfully reproducing the compression artifacts. Switching to 24 kHz WAV files with no compression fixed the problem immediately. The style encoder captures everything in the reference — including the bad parts.” — ML Engineer at a voice AI startup
“The two-stage training pipeline caught us off guard. We assumed we could fine-tune the model end-to-end from the start. StyleTTS 2 requires pre-training the acoustic modules first, then joint training. Skipping stage 1 produces a model that generates noise. The training script checks for this, but the error message is cryptic. Always run stage 1 to completion before starting stage 2.” — Research Engineer at an accessibility platform
“Our biggest mistake was assuming zero-shot cloning would work with 2-second audio samples. The paper says 3-10 seconds, and they mean it. With 2 seconds, the style encoder does not have enough context to produce a stable style vector. The generated speech sounds like the reference speaker for the first word, then drifts to the default voice. We switched to 8-second samples and the problem disappeared.” — Infrastructure Engineer at a dubbing platform
“Style interpolation sounds great in theory but is unpredictable in practice. Blending two style vectors 50/50 does not produce a voice that sounds halfway between the two speakers. It produces a voice that sounds like neither — an artifact of the style encoder’s nonlinear representation space. We found that 70/30 blends (70% primary, 30% secondary) produce the most natural results. Anything beyond 60/40 introduces audible artifacts.” — Research Scientist at a game audio company
Next in the Open-Source AI Tools Mastery series: VoiceCraft
Written by Nivant Labs Team
Engineer at Nivant Labs