·15 min read

Fish Speech: A state-of-the-art text-to-speech model (Apache 2.0, 16k stars)

Supporting 8 languages with voice cloning, emotion control, and 4x faster than real-time inference — a state-of-the-art TTS model with permissive licensing.

#fish-speech#tts#voice-cloning#multilingual#ai-voice

The Problem

Every text-to-speech system on the market makes the same implicit trade-off: naturalness or control. You can have a voice that sounds human but cannot express emotion, or you can have fine-grained prosody control with audio that sounds like a GPS navigator from 2012. The two rarely coexist.

The commercial leaders (ElevenLabs, Murf, WellSaid) deliver impressive naturalness but lock you into their pricing, their latency, and their data-privacy model. Every voice generation sends your text and reference audio to a third-party server. For enterprise use cases — healthcare, finance, defense — that is a non-starter.

Open-source alternatives have historically been worse on both axes. Coqui TTS, Tacotron, and FastSpeech produce robotic output that requires significant fine-tuning to sound passable. Voice cloning requires hours of reference audio and per-speaker model training. Multilingual support means training separate models per language.

Dimension Commercial TTS (ElevenLabs) Legacy Open-Source TTS (Tacotron, FastSpeech) Fish Speech v1.5
Naturalness (MOS) 4.2-4.5 3.0-3.5 4.05
Voice cloning Instant (10s sample) Hours of training per voice Zero-shot (10-30s sample)
Languages 29 1-2 per model 8+ in single model
Emotion control Preset sliders None 15,000+ natural language tags
Inference speed Cloud-dependent Real-time on GPU 4-15x real-time on consumer GPU
Data privacy None (cloud-only) Full (self-hosted) Full (self-hosted)
Cost $5-500+/month Free (electricity only) Free (electricity only)
License Proprietary Varies (BSD, MIT) Apache 2.0 (code)

Why this matters: The TTS market is bifurcated between “sounds great but costs too much and owns your data” and “free but sounds robotic.” Fish Speech is the first open-source model that closes the quality gap with commercial offerings while keeping your data on your hardware. This is not a compromise — it is a genuine alternative for anyone who needs production-quality TTS without vendor lock-in.

The Investigation

The Fish Audio team (Beijing-based, spun out of academic research) spent two years investigating why open-source TTS could not match commercial quality. The answer is not model architecture — it is data scale and training methodology.

Finding 1: Tokenization is the bottleneck, not the model.

Traditional TTS systems use mel-spectrograms as the intermediate representation. The model predicts mel frames from text, then a vocoder converts mel frames to audio. This two-stage pipeline introduces information loss at every boundary: the text-to-mel model cannot express acoustic detail that the mel representation cannot capture, and the vocoder cannot reconstruct detail that was lost in quantization.

Fish Speech’s investigation found that direct audio tokenization with residual vector quantization (RVQ) preserves significantly more acoustic information than mel-spectrograms. The key insight: by using 10 parallel codebooks (1 semantic + 9 residual), the model can represent fine-grained acoustic detail — pitch contour, timbre, breathiness, emotional prosody — that mel-spectrograms discard as “noise.”

What this means: The traditional TTS pipeline throws away information at the first step. Fish Speech’s VQGAN codec preserves it, and the dual-AR transformer learns to reconstruct it. The result is audio that retains the natural variability of human speech — including the imperfections that make it sound real.

Finding 2: Autoregressive generation beats non-autoregressive for expressiveness.

Non-autoregressive TTS models (FastSpeech, VITS) generate all frames in parallel. They are fast — single-shot, no sequential decoding — but they cannot model long-range dependencies in prosody. A sentence’s emotional arc, the buildup to a punchline, the trailing off at the end of a thought — these require sequential context that parallel models cannot capture.

Fish Speech’s dual-autoregressive architecture generates tokens sequentially, one frame at a time. The slow transformer predicts the semantic codebook (linguistic content, global prosody), and the fast transformer predicts the 9 residual codebooks (acoustic detail, local variation). This hierarchical sequential generation produces significantly more natural prosody than any non-autoregressive approach.

Approach MOS RTF Prosody Quality
Non-autoregressive (VITS) 3.2-3.5 0.01-0.05 Flat, monotone
Single-AR (Tacotron 2) 3.5-3.8 0.1-0.3 Better, but robotic
Dual-AR (Fish Speech) 4.05 0.07-0.25 Near-human

What this means: Speed is not the only metric. Non-autoregressive models are fast because they cheat — they generate all frames independently and hope the post-processing network smooths out the discontinuities. Autoregressive generation is slower but produces coherent prosody because every frame is conditioned on every previous frame. Fish Speech’s dual-AR design gets the best of both: the slow transformer handles global structure, the fast transformer fills in local detail.

Finding 3: DPO alignment beats supervised fine-tuning for naturalness.

Most TTS models are trained with a simple next-token prediction loss. This produces technically correct audio — the right phonemes at the right time — but it does not optimize for what humans perceive as “natural.” A model trained on next-token prediction will happily produce audio that is technically accurate but sounds flat, rushed, or awkward.

Fish Speech’s three-stage training pipeline addresses this:

  1. Pre-training (1M+ hours, 8xH100, ~1 week): Standard next-token prediction on large-scale multilingual data. The model learns basic phoneme-to-acoustic mapping.
  2. Supervised Fine-Tuning (high-quality subset): The model learns to match high-quality recording standards — clean audio, consistent pacing, proper articulation.
  3. Direct Preference Optimization (human-labeled pairs): The model learns to prefer the audio that humans prefer. This is the critical stage: DPO shifts the model’s output distribution from “technically correct” to “sounds good to humans.”

The DPO stage alone accounts for a 0.3-0.5 MOS improvement over SFT-only models. This is the difference between “passable” and “pleasant to listen to.”

What this means: If you are building a TTS system and skipping preference optimization, you are leaving a 0.5 MOS improvement on the table. DPO is not optional — it is the difference between a demo and a product.

The Solution

Fish Speech v1.5 is a ~50,000-line Python/PyTorch application (Apache 2.0 code license, 30,000+ GitHub stars, 1M+ Docker pulls) that runs entirely on your hardware. It uses a dual-autoregressive transformer architecture with VQGAN audio tokenization to convert text to natural-sounding speech in 8+ languages.

┌──────────────────────────────────────────────────────────────────────────┐
│                          Fish Speech Architecture                          │
│                                                                           │
│  ┌──────────────┐    ┌──────────────────────┐    ┌──────────────────┐    │
│  │  Input Text   │    │   Slow Transformer   │    │  Fast Transformer │    │
│  │  (tokenized)  │───▶│   (text2semantic)    │───▶│  (semantic2acous)│    │
│  │               │    │                      │    │                  │    │
│  │  • UTF-8 BPE  │    │  • 7B params         │    │  • 400M params   │    │
│  │  • 8+ langs   │    │  • Predicts codebook 0│    │  • Predicts cb 1-9│    │
│  │  • No G2P     │    │  • Global prosody     │    │  • Acoustic detail│    │
│  └──────┬───────┘    └──────────┬───────────┘    └────────┬─────────┘    │
│         │                       │                          │              │
│         │              ┌────────┴──────────┐              │              │
│         │              │  VQGAN Audio Codec │              │              │
│         │              │  (10 codebooks)    │              │              │
│         │              │                    │              │              │
│         │              │  • Codebook 0:     │              │              │
│         │              │    semantic tokens │              │              │
│         │              │  • Codebooks 1-9:  │              │              │
│         │              │    residual detail │              │              │
│         │              └────────┬───────────┘              │              │
│         │                       │                          │              │
│  ┌──────┴───────────────────────┴──────────────────────────┴──────────┐  │
│  │                    FF-GAN Vocoder (GFSQ)                            │  │
│  │  ┌──────────────┐  ┌──────────────┐  ┌──────────────────────────┐  │  │
│  │  │ ParallelBlock│  │ Depthwise    │  │ Grouped Finite Scalar    │  │  │
│  │  │ (MRF replace)│  │ Separable    │  │ Vector Quantization      │  │  │
│  │  │              │  │ Convolutions │  │ (FSQ + GVQ fusion)       │  │  │
│  │  └──────────────┘  └──────────────┘  └──────────────────────────┘  │  │
│  └────────────────────────────────────────────────────────────────────┘  │
│                                                                           │
│  ┌────────────────────────────────────────────────────────────────────┐  │
│  │                    Training Pipeline                                 │  │
│  │  Pre-train (1M+ hrs) → SFT (high-quality) → DPO (human pref.)      │  │
│  └────────────────────────────────────────────────────────────────────┘  │
└──────────────────────────────────────────────────────────────────────────┘

Here is what each component does:

  • Input Text Tokenizer: UTF-8 byte-pair encoding that handles 8+ languages without grapheme-to-phoneme conversion. The model learns phoneme-to-acoustic mapping directly from data, eliminating the traditional G2P pipeline that introduces language-specific errors.
  • Slow Transformer (text2semantic): A 7B-parameter LLaMA-style transformer that predicts the primary semantic codebook (codebook 0) from text embeddings. This captures global linguistic structure — word emphasis, sentence intonation, emotional arc. It runs at ~21 Hz frame rate, one token per ~48ms of audio.
  • Fast Transformer (semantic2acoustic): A 400M-parameter transformer that predicts the 9 residual codebooks (codebooks 1-9) conditioned on the slow transformer’s output. This captures local acoustic detail — pitch micro-variation, breathiness, formant transitions. It runs at the same 21 Hz frame rate but processes all 9 codebooks per step.
  • VQGAN Audio Codec: A modified DAC (Descript Audio Codec) with residual vector quantization. Converts raw audio waveforms (16-48 kHz) into 10 parallel discrete token streams. The first codebook carries semantic content; the remaining 9 carry residual acoustic detail. Achieves ~100% codebook utilization through GFSQ.
  • FF-GAN Vocoder: A Grouped Finite Scalar Vector Quantization (GFSQ) vocoder that reconstructs audio from the 10 codebook streams. Replaces traditional Multi-Receptive Field (MRF) modules with ParallelBlock architecture using depthwise separable convolutions and dilated convolutions for superior compression and reconstruction quality.
  • Training Pipeline: Three-stage training — pre-training on 1M+ hours of multilingual data (8xH100, ~1 week), supervised fine-tuning on high-quality subsets, and Direct Preference Optimization using human-labeled preference pairs. The DPO stage is responsible for the final 0.3-0.5 MOS improvement.

Setup

# Prerequisites
apt install portaudio19-dev libsox-dev ffmpeg

# Clone and install
git clone https://github.com/fishaudio/fish-speech.git
cd fish-speech
conda create -n fish-speech python=3.12
conda activate fish-speech
pip install -e .[cu129]  # or [cu126], [cu128], [cpu]

# Download model weights
huggingface-cli download fishaudio/fish-speech-1.5 \
  --local-dir checkpoints/fish-speech-1.5

# Launch WebUI
python -m tools.webui \
  --llama-checkpoint-path "checkpoints/fish-speech-1.5" \
  --decoder-checkpoint-path "checkpoints/fish-speech-1.5/firefly-gan-vq-fsq-8x1024-21hz-generator.pth" \
  --decoder-config-name firefly_gan_vq \
  --compile  # ~10x speedup on CUDA

Production-Grade API Server

# Start the API server
python -m tools.api_server \
  --listen 0.0.0.0:8080 \
  --llama-checkpoint-path "checkpoints/fish-speech-1.5" \
  --decoder-checkpoint-path "checkpoints/fish-speech-1.5/firefly-gan-vq-fsq-8x1024-21hz-generator.pth" \
  --decoder-config-name firefly_gan_vq \
  --compile \
  --half  # FP16 for VRAM efficiency

Code Walkthrough: The Inference Pipeline

The heart of Fish Speech is the three-step inference pipeline in tools/llama/generate.py and tools/vqgan/inference.py. Here is the simplified flow:

# Step 1: Encode reference audio into voice prompt tokens
# tools/vqgan/inference.py
import torch
from fish_speech.models.vqgan.modules.firefly import FireflyArchitecture

def encode_audio(audio_path: str, checkpoint_path: str) -> np.ndarray:
    """Encode reference audio into VQGAN token prompt."""
    model = FireflyArchitecture.from_pretrained(checkpoint_path)
    model.eval().cuda()

    # Load and preprocess audio (16kHz mono)
    audio, sr = torchaudio.load(audio_path)
    audio = torchaudio.functional.resample(audio, sr, 16000)
    audio = audio.unsqueeze(0).cuda()  # [1, 1, T]

    with torch.no_grad():
        # VQGAN encodes waveform into 10 codebook streams
        # Shape: [1, 10, T'] where T' = T / 320 (21 Hz frame rate)
        encoded = model.encode(audio)
        # Take codebook 0 as semantic prompt
        semantic_tokens = encoded[0, 0, :]  # [T']

    return semantic_tokens.cpu().numpy()

# Step 2: Generate semantic tokens from text
# tools/llama/generate.py
from fish_speech.models.text2semantic.llama import DualARTransformer

def text_to_semantic(
    text: str,
    prompt_tokens: np.ndarray,
    prompt_text: str,
    checkpoint_path: str,
    num_samples: int = 2,
) -> list[np.ndarray]:
    """Generate semantic tokens from text, conditioned on voice prompt."""
    model = DualARTransformer.from_pretrained(checkpoint_path)
    model.eval().cuda()

    # Tokenize input text
    input_ids = tokenizer.encode(text, return_tensors="pt").cuda()

    # Prepare voice prompt
    prompt_ids = torch.from_numpy(prompt_tokens).long().cuda().unsqueeze(0)

    with torch.no_grad():
        # Slow transformer generates codebook 0 tokens autoregressively
        # Shape: [num_samples, T_out]
        output = model.generate(
            input_ids=input_ids,
            prompt_tokens=prompt_ids,
            max_new_tokens=1024,
            temperature=0.7,
            top_p=0.9,
            repetition_penalty=1.1,
            num_samples=num_samples,
        )

    return [output[i].cpu().numpy() for i in range(num_samples)]

# Step 3: Decode semantic tokens to audio waveform
def semantic_to_audio(
    semantic_tokens: np.ndarray,
    decoder_checkpoint: str,
) -> np.ndarray:
    """Decode semantic tokens back to audio waveform."""
    decoder = FireflyArchitecture.from_pretrained(decoder_checkpoint)
    decoder.eval().cuda()

    tokens = torch.from_numpy(semantic_tokens).long().cuda()
    # Expand to 10 codebooks: codebook 0 is semantic, 1-9 are predicted
    # by the fast transformer internally
    tokens = tokens.unsqueeze(0).unsqueeze(0)  # [1, 1, T]

    with torch.no_grad():
        # FF-GAN vocoder reconstructs waveform
        # Shape: [1, 1, T * 320] (upsamples 320x)
        audio = decoder.decode(tokens)

    return audio.squeeze().cpu().numpy()

The dual-AR generation is the most architecturally interesting piece:

# Simplified from fish_speech/models/text2semantic/llama.py
class DualARTransformer(nn.Module):
    def __init__(self, config):
        super().__init__()
        # Slow transformer: predicts codebook 0 (semantic)
        self.slow_transformer = LLaMATransformer(
            vocab_size=config.semantic_vocab_size,  # ~1024 tokens
            hidden_size=config.hidden_size,          # 4096
            num_layers=config.num_layers,            # 32
            num_attention_heads=config.num_heads,     # 32
        )
        # Fast transformer: predicts codebooks 1-9 (residual)
        self.fast_transformer = LLaMATransformer(
            vocab_size=config.acoustic_vocab_size,   # ~1024 per codebook
            hidden_size=config.fast_hidden_size,      # 1024
            num_layers=config.fast_num_layers,        # 8
            num_attention_heads=config.fast_num_heads,# 8
        )
        # VQ codebook embeddings
        self.codebook_embeddings = nn.Embedding(
            config.semantic_vocab_size, config.hidden_size
        )

    def generate(self, input_ids, prompt_tokens, max_new_tokens, **kwargs):
        """Generate audio tokens from text."""
        batch_size = input_ids.shape[0]

        # Phase 1: Slow transformer generates codebook 0
        # Start with prompt tokens as prefix
        generated = prompt_tokens  # [B, T_prompt]

        for _ in range(max_new_tokens):
            # Encode text + generated tokens
            hidden = self.slow_transformer(
                input_ids=input_ids,
                prefix_tokens=generated,
            )
            # Predict next semantic token
            logits = self.lm_head(hidden[:, -1, :])
            next_token = sample(logits, **kwargs)
            generated = torch.cat([generated, next_token], dim=1)

        # Phase 2: Fast transformer generates codebooks 1-9
        # Conditioned on codebook 0 tokens from phase 1
        acoustic_tokens = self.fast_transformer.generate(
            semantic_tokens=generated,
            num_codebooks=9,
            **kwargs,
        )

        # Stack: [B, 10, T] — codebook 0 + 9 residual codebooks
        return torch.stack([generated, *acoustic_tokens], dim=1)

How to Use Effectively

Step 1: Set up the environment

# Create isolated environment
conda create -n fish-speech python=3.12
conda activate fish-speech
pip install -e .[cu129]

# Download models
huggingface-cli download fishaudio/fish-speech-1.5 \
  --local-dir checkpoints/fish-speech-1.5

# Verify GPU detection
python -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))"

Step 2: Prepare reference audio for voice cloning

Voice cloning quality depends almost entirely on the reference audio. A 10-second clip of clean, consistent speech produces better results than a 2-minute clip with background noise.

# Convert reference audio to 16kHz mono WAV
ffmpeg -i input.mp3 -ar 16000 -ac 1 reference.wav

# Encode reference audio into voice prompt
python tools/vqgan/inference.py \
  -i "reference.wav" \
  --checkpoint-path "checkpoints/fish-speech-1.5/firefly-gan-vq-fsq-8x1024-21hz-generator.pth"
# Produces fake.npy

Production pitfall: The reference audio’s transcript must match exactly. If the transcript says “I love coding” but the audio says “I love coding in Python,” the model will hallucinate the mismatch. Always provide the exact transcript of the reference audio.

Step 3: Generate speech from text

# Single generation
python tools/llama/generate.py \
  --text "The quick brown fox jumps over the lazy dog." \
  --prompt-text "Your reference text that matches the audio exactly." \
  --prompt-tokens "fake.npy" \
  --checkpoint-path "checkpoints/fish-speech-1.5" \
  --num-samples 2 \
  --compile

# Decode to audio
python tools/vqgan/inference.py \
  -i "codes_0.npy" \
  --checkpoint-path "checkpoints/fish-speech-1.5/firefly-gan-vq-fsq-8x1024-21hz-generator.pth"
# Produces fake.wav

Step 4: Use the API server for production

# Start server
python -m tools.api_server \
  --listen 0.0.0.0:8080 \
  --llama-checkpoint-path "checkpoints/fish-speech-1.5" \
  --decoder-checkpoint-path "checkpoints/fish-speech-1.5/firefly-gan-vq-fsq-8x1024-21hz-generator.pth" \
  --decoder-config-name firefly_gan_vq \
  --compile

# Client usage
python -m tools.api_client \
  --text "Text to synthesize" \
  --reference_audio "reference.wav" \
  --reference_text "Content of reference audio" \
  --streaming True

Step 5: Use the Fish Audio Python SDK for cloud API

pip install fish-audio-sdk
export FISH_API_KEY=your_api_key_here
from fishaudio import FishAudio
from fishaudio.utils import save

client = FishAudio()

# Basic TTS with a specific voice
audio = client.tts.convert(
    text="Hello from Fish Speech.",
    reference_id="802e3bc2b27e49c2995d23ef70e6ac89"
)
save(audio, "output.mp3")

# Instant voice cloning with reference audio
with open("sample.wav", "rb") as f:
    audio = client.tts.convert(
        text="Spoken in the cloned voice.",
        references=[{
            "audio": f.read(),
            "text": "Transcript of the sample."
        }],
    )
save(audio, "cloned_output.mp3")

# Streaming for long-form content
audio_stream = client.tts.stream(
    text="A very long passage of text that will be streamed chunk by chunk..."
)
with open("long_output.mp3", "wb") as f:
    for chunk in audio_stream:
        f.write(chunk)

Use Cases

1. Conversational AI Voice Pipeline

When you would use this: You are building a voice assistant, customer service bot, or interactive NPC that needs to respond in natural-sounding speech with low latency.

Why Fish Speech fits: The streaming API achieves ~150ms time-to-first-audio with chunked generation. Combined with Whisper for STT and any LLM for response generation, you get a complete voice pipeline that runs entirely on your hardware. The emotion tags let the TTS match the conversational context — a customer service bot can sound empathetic when handling a complaint and confident when confirming a resolution.

2. Audiobook and Long-Form Narration

When you would use this: You need to convert a 50,000-word manuscript into natural-sounding audio with consistent voice quality across hours of content.

Why Fish Speech fits: The voice cloning preserves timbre and speaking style across arbitrarily long text. The streaming API handles memory-efficient generation of long content. The emotion control lets you inject appropriate affect per chapter or scene. A 10-hour audiobook costs approximately $0 in API fees on self-hosted hardware — versus $500-2,000 on commercial TTS services.

3. Multilingual Dubbing and Localization

When you would use this: You have video content in one language and need to dub it into 5+ languages while preserving the original speaker’s voice characteristics.

Why Fish Speech fits: A single model handles 8+ languages. Voice cloning works cross-lingually — clone a voice from English audio and generate speech in Chinese, Japanese, or German with the same timbre. The emotion tags transfer across languages, so an excited delivery in English maps to an excited delivery in French. No per-language model training required.

4. Accessibility and Assistive Technology

When you would use this: You are building a screen reader, communication aid for non-verbal individuals, or reading assistant for people with visual impairments.

Why Fish Speech fits: Self-hosted deployment means no data leaves the user’s device — critical for medical and accessibility applications. The naturalness (MOS 4.05) makes extended listening comfortable, unlike robotic TTS that causes listener fatigue. Voice cloning lets a non-verbal individual communicate in their own voice, not a generic synthetic one.

5. Content Creation and Voiceovers

When you would use this: You produce YouTube videos, podcasts, or social media content and need voiceovers without hiring voice actors.

Why Fish Speech fits: Zero-shot voice cloning from 10 seconds of reference audio. Emotion tags for expressive delivery. Multiple languages for international audiences. The API server integrates directly into video editing pipelines. A content creator can generate 50 voiceover variants in 10 minutes and pick the best one — something that would take days with human voice actors.

Cheat Sheet

Aspect Detail
Repository github.com/fishaudio/fish-speech
License Apache 2.0 (code), CC-BY-NC-SA-4.0 (model weights)
Language Python/PyTorch (~50,000 lines)
GPU Requirements 12 GB minimum, 24 GB recommended
Setup Time 10 minutes (clone + install + download weights)
Key Features Voice cloning (10-30s sample), emotion tags (15,000+), 8+ languages, streaming, 4-15x real-time inference
Common Gotchas Reference audio transcript mismatch; missing --compile flag (10x slower); noisy reference audio; wrong CUDA version in pip extras
Best Hardware RTX 4090 (home), A100 80GB (production), H200 (extreme)
Cost (Self-Hosted) $0 (electricity only)
Cost (Cloud API) ~$0.003/second of audio
Missing Features No built-in voice activity detection; no streaming STT; no multi-speaker diarization

Vibe Coding Projects

Project 1: Personal Voice Assistant with Custom Wake Word

What it does: A voice-activated assistant that listens for a wake word, transcribes speech with Whisper, processes it through an LLM, and responds with Fish Speech TTS in a cloned voice. Runs entirely on a single GPU.

What you will learn: How to build a complete STT-LLM-TTS pipeline. How to integrate Fish Speech’s streaming API for low-latency response. How to clone a voice from a short reference sample and use it consistently. How to handle audio buffering, silence detection, and concurrent streaming.

Effort: 4-6 hours. ~$0 in API costs (self-hosted).

Project 2: Multilingual Audiobook Generator

What it does: A tool that takes a text file (English, Chinese, Japanese, or German), generates natural-sounding narration with chapter-level emotion control, and outputs chapterized MP3 files. Supports voice cloning so all chapters use the same narrator voice.

What you will learn: How to use Fish Speech’s batch generation for long-form content. How to apply emotion tags per chapter for narrative effect. How to handle text segmentation, token limits, and audio concatenation. How to use the streaming API for memory-efficient generation of long content.

Effort: 3-5 hours. ~$0 in API costs (self-hosted).

Project 3: Real-Time Dubbing System for Video Calls

What it does: A real-time speech-to-speech translation system that listens to one speaker, transcribes with Whisper, translates through an LLM, and generates speech in the target language using Fish Speech with the original speaker’s voice characteristics. Sub-2-second end-to-end latency.

What you will learn: How to optimize Fish Speech for low-latency inference (chunk_length, compile, half-precision). How to build a real-time audio pipeline with WebSocket streaming. How to handle cross-lingual voice cloning. How to manage concurrent audio streams and buffer underruns.

Effort: 8-12 hours. ~$0 in API costs (self-hosted).

Problems Solved Efficiently

Problem Type Why Fish Speech Fits When to Look Elsewhere
Voice cloning from short samples Zero-shot from 10-30s, no fine-tuning Use ElevenLabs for instant cloud cloning
Multilingual TTS in one model 8+ languages, no G2P pipeline Use Google Cloud TTS for 100+ languages
Emotion-controlled speech 15,000+ natural language tags Use Amazon Polly for preset emotions only
Self-hosted production TTS Apache 2.0, full data privacy Use ElevenLabs for managed cloud service
Real-time conversational TTS ~150ms TTFA, streaming API Use Azure Speech for enterprise SLAs
Long-form audiobook generation Consistent voice, chapter-level control Use Play.ht for managed audiobook production
Cross-lingual voice cloning Clone once, generate in any supported language Use Respeecher for celebrity voice licensing
Low-cost batch TTS $0/GPU-hour on self-hosted hardware Use any cloud API for one-off generation

Architectural Tradeoffs

What we gained:

  • Data privacy. Every voice generation stays on your hardware. No text, no reference audio, no generated speech ever leaves your network. For healthcare, finance, legal, and defense use cases, this is the difference between deployable and forbidden.
  • Voice cloning without training. Zero-shot cloning from 10-30 seconds of reference audio. No per-speaker fine-tuning, no GPU-hours spent on training, no model storage for each voice. Clone a voice in the time it takes to encode one audio file.
  • Emotion control at sub-word granularity. 15,000+ natural language tags that can be applied inline: [whisper], [excited], [angry], [laughing], [sad]. The model interprets these as prosody instructions, not just labels. You can switch emotion mid-sentence.
  • Multilingual without phonemes. The model learns phoneme-to-acoustic mapping directly from data. No grapheme-to-phoneme conversion, no language-specific preprocessing, no per-language model variants. Add a language by adding training data, not by building a new pipeline.
  • Competitive quality at zero marginal cost. MOS 4.05 matches commercial offerings that charge $5-500/month. The quality gap between open-source and commercial TTS has effectively closed.
  • Standard LLM infrastructure. The dual-AR transformer is structurally isomorphic to standard LLMs. It runs on the same infrastructure (SGLang, vLLM) with the same acceleration techniques (KV-cache, torch compile, continuous batching).

What we sacrificed:

  • No cloud SLA. Self-hosted means self-managed. If your GPU goes down, your TTS goes down. No 99.9% uptime guarantee, no auto-scaling, no regional redundancy. For mission-critical applications, you need a fallback or a managed service.
  • No voice library. Commercial TTS services offer curated voice libraries with hundreds of professionally recorded voices. Fish Speech gives you the cloning tool — you provide the reference audio. If you do not have a good reference sample, you cannot generate a good voice.
  • No streaming STT. Fish Speech is TTS-only. Building a complete voice pipeline requires integrating Whisper (or another STT model) separately. There is no built-in voice activity detection, no diarization, no end-to-end voice pipeline.
  • GPU dependency. Fish Speech requires a GPU for practical inference. CPU inference is possible but painfully slow (~45 seconds per generation). This limits deployment to machines with NVIDIA GPUs or cloud GPU instances.
  • No real-time factor guarantee. The 4-15x real-time inference depends on GPU, model size, batch size, and compilation status. Without --compile, performance drops 10x. Without a modern GPU, real-time generation is not guaranteed.
  • Model weights are not Apache 2.0. The code is Apache 2.0, but the model weights are CC-BY-NC-SA-4.0. Commercial use requires a separate license from Fish Audio. This is a common source of confusion for teams evaluating the project.

The real lesson: Fish Speech is not a replacement for commercial TTS — it is a complement that covers the gaps commercial services leave open. Use Fish Speech when you need data privacy, voice cloning at scale, or zero marginal cost. Use ElevenLabs or Azure Speech when you need managed infrastructure, curated voices, or enterprise SLAs. The teams that get the most out of TTS run both — and switch based on the use case.

Course-Style Deep Dive

How the VQGAN Audio Codec Works Under the Hood

The VQGAN codec is the foundation of Fish Speech’s quality. Here is how it works, step by step:

  1. Encoder Convolutional Stack. Raw audio (16 kHz, mono) passes through a stack of convolutional layers with progressive downsampling. Each layer reduces the temporal resolution by a factor of 2, for a total downsampling factor of 320. A 1-second audio clip (16,000 samples) becomes a 50-frame latent sequence.

  2. Residual Vector Quantization (RVQ). Each latent frame is quantized through 10 stacked codebooks. Codebook 0 captures the coarse semantic structure (which phoneme, approximate pitch). Codebooks 1-9 capture progressively finer acoustic detail (exact pitch contour, timbral texture, breathiness). Each codebook has 1,024 entries, for a total of 10,240 learnable centroids.

  3. Grouped Finite Scalar Quantization (GFSQ). Traditional RVQ suffers from codebook collapse — some codebook entries are never used, wasting capacity. GFSQ addresses this by grouping dimensions and applying finite scalar quantization within each group. The result is near-100% codebook utilization, meaning every centroid contributes to reconstruction quality.

  4. Decoder Convolutional Stack. The quantized tokens pass through a symmetric decoder stack that upsamples back to the original waveform resolution. Skip connections from the encoder improve reconstruction fidelity.

  5. Discriminator Training. The codec is trained adversarially with a multi-scale discriminator that distinguishes real audio from reconstructed audio. This forces the codec to preserve perceptually important detail that L1 or L2 loss would discard.

Advanced Pattern 1: Emotion-Controlled Narration

# Inject emotion tags directly into the text
text = (
    "[excited] We just shipped the new feature! [/excited] "
    "[whisper] But the tests are still failing... [/whisper] "
    "[angry] Who merged without CI? [/angry]"
)

# Generate with emotion control
python tools/llama/generate.py \
  --text "$text" \
  --prompt-text "Reference transcript." \
  --prompt-tokens "fake.npy" \
  --checkpoint-path "checkpoints/fish-speech-1.5" \
  --compile

The emotion tags are not fixed presets — they are learned embeddings that modulate the transformer’s hidden states. [excited] increases pitch range and speaking rate. [whisper] reduces volume and adds breathiness. [angry] increases vocal intensity and lowers pitch. The model learns these mappings from the DPO training data, not from explicit rules.

Advanced Pattern 2: Multi-Speaker Dialogue Generation

# Multi-speaker dialogue with speaker tokens
text = (
    "<|speaker:0|> Hello, how can I help you today? "
    "<|speaker:1|> I need to reset my password. "
    "<|speaker:0|> Sure, I can help with that. "
    "<|speaker:1|> Thanks!"
)

# Encode separate reference audio for each speaker
python tools/vqgan/inference.py -i "agent.wav" --checkpoint-path "checkpoints/fish-speech-1.5/..."
# Produces agent.npy

python tools/vqgan/inference.py -i "customer.wav" --checkpoint-path "checkpoints/fish-speech-1.5/..."
# Produces customer.npy

# Generate with multi-speaker conditioning
python tools/llama/generate.py \
  --text "$text" \
  --prompt-text "Agent: reference text. Customer: reference text." \
  --prompt-tokens "agent.npy" "customer.npy" \
  --checkpoint-path "checkpoints/fish-speech-1.5" \
  --compile

The model learns to associate <|speaker:0|> with the first prompt token sequence and <|speaker:1|> with the second. This enables natural dialogue with distinct voices per speaker, including turn-taking prosody and cross-speaker emotional dynamics.

Advanced Pattern 3: Production Deployment with Docker and Kubernetes

# docker-compose.yml
version: "3.8"
services:
  fish-speech:
    image: fishaudio/fish-speech:latest
    entrypoint: ["/app/.venv/bin/python", "-m", "tools.api_server"]
    command: [
      "--listen", "0.0.0.0:8080",
      "--llama-checkpoint-path", "/models/fish-speech-1.5",
      "--decoder-checkpoint-path", "/models/fish-speech-1.5/firefly-gan-vq-fsq-8x1024-21hz-generator.pth",
      "--decoder-config-name", "firefly_gan_vq",
      "--compile",
      "--half"
    ]
    ports:
      - "8080:8080"
    volumes:
      - ./models:/models
      - huggingface_cache:/root/.cache/huggingface
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 120s  # Model loading takes 2+ minutes

volumes:
  huggingface_cache:
# kubernetes-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: fish-speech
spec:
  replicas: 1
  selector:
    matchLabels:
      app: fish-speech
  template:
    metadata:
      labels:
        app: fish-speech
    spec:
      containers:
      - name: fish-speech
        image: fishaudio/fish-speech:latest
        command: ["/app/.venv/bin/python", "-m", "tools.api_server"]
        args: [
          "--listen", "0.0.0.0:8080",
          "--llama-checkpoint-path", "/models/fish-speech-1.5",
          "--decoder-checkpoint-path", "/models/fish-speech-1.5/firefly-gan-vq-fsq-8x1024-21hz-generator.pth",
          "--decoder-config-name", "firefly_gan_vq",
          "--compile",
          "--half"
        ]
        ports:
        - containerPort: 8080
        resources:
          limits:
            nvidia.com/gpu: "1"
            memory: "24Gi"
            cpu: "8"
        volumeMounts:
        - name: models
          mountPath: /models
        - name: cache
          mountPath: /root/.cache/huggingface
        startupProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 120
          periodSeconds: 10
          failureThreshold: 6
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          periodSeconds: 30
      volumes:
      - name: models
        persistentVolumeClaim:
          claimName: fish-speech-models
      - name: cache
        emptyDir: {}

Production Considerations

GPU memory management. Fish Speech v1.5 requires approximately 5 GB of VRAM for the model weights plus 2-4 GB for activations and KV-cache. With --half (FP16), the model fits comfortably on a 12 GB GPU. Without half-precision, you need 16 GB minimum. For batch inference, add 1-2 GB per additional sequence.

Model caching. The Hugging Face model cache (~3.5 GB for v1.5) should be mounted on a persistent volume. Without caching, every container restart triggers a 3.5 GB download that takes 5-15 minutes depending on bandwidth.

Compilation overhead. The --compile flag triggers CUDA graph fusion on the first forward pass. This takes 2-5 minutes but delivers a 10x speedup on subsequent generations. For production deployments, pre-compile by running a dummy generation during the startup probe period.

Streaming vs. batch. The streaming API (chunked generation) starts producing audio in ~150ms but has higher per-token overhead. The batch API (full generation) is more efficient for long content but has higher first-packet latency. Choose based on your use case: streaming for conversational AI, batch for audiobook generation.

Error handling. The API server returns HTTP 503 if the model is still loading, HTTP 429 if the request queue is full, and HTTP 400 for malformed requests. Implement client-side retry with exponential backoff for production integrations.

The Results

Metric Before Fish Speech After Fish Speech Improvement
Voice cloning setup time 2-4 hours (fine-tuning) 10 seconds (zero-shot) 720-1440x faster
Voice cloning reference length 30-60 minutes 10-30 seconds 60-360x less data
Languages per model 1-2 8+ 4-8x more languages
MOS (naturalness) 3.0-3.5 (open-source) 4.05 +0.5-1.0 MOS
WER (voice cloning) 9.22% (ground truth) 6.89% -2.33 pts (better than real)
Inference speed (RTX 4090) 1x real-time 4-15x real-time 4-15x faster
Time-to-first-audio 500-1000ms ~150ms 3-7x faster
Cost per 10 hours of audio $500-2,000 (cloud) $0 (self-hosted) Infinite ROI
Emotion control Preset sliders (3-5) 15,000+ tags 3,000-5,000x more options
Data privacy None (cloud) Full (self-hosted) Binary improvement

What this means for you: Fish Speech closes the quality gap between open-source and commercial TTS while adding capabilities (emotion control, zero-shot cloning, multilingual) that commercial offerings cannot match. The 4-15x real-time inference on consumer GPUs means you do not need a data center to run it. The Apache 2.0 code license means you can integrate it into any project without legal overhead. The key is matching the deployment model to your use case: self-hosted for privacy and cost, cloud API for convenience and scale.

What to Watch Out For

  1. Reference audio quality is everything. A 10-second clip of clean, consistent speech produces better voice cloning than a 2-minute clip with background noise, reverb, or volume variation. Use a high-quality microphone, record in a quiet room, and normalize the audio to -3 dB peak. The model cannot fix bad source audio.

  2. The transcript must match the audio exactly. If the reference audio says “I love coding in Python” and the transcript says “I love coding,” the model will try to reconcile the mismatch and produce artifacts. Always provide the verbatim transcript. For long reference clips, use Whisper to generate the transcript, then manually verify every word.

  3. Emotion tags are not magic. [excited] works well for moderate excitement. Extreme emotions ([screaming], [sobbing]) can produce artifacts because the training data contains fewer examples. Test emotion tags on short phrases before committing to a full script.

  4. The --compile flag is not optional. Without it, Fish Speech runs at 30 tokens/second on an RTX 4090 — barely real-time. With --compile, it runs at 300+ tokens/second — 10x real-time. The 2-5 minute compilation overhead on the first run is worth the wait. For production, pre-compile during the startup probe period.

  5. Model weights are CC-BY-NC-SA-4.0, not Apache 2.0. The code is Apache 2.0, which means you can fork, modify, and redistribute the code freely. The model weights are CC-BY-NC-SA-4.0, which prohibits commercial use without a separate license. If you are building a commercial product, contact Fish Audio for a commercial license or use the cloud API.

  6. CPU inference is not practical. Fish Speech requires a GPU for usable inference. On CPU, a single generation takes ~45 seconds. If you do not have a GPU, use the Fish Audio cloud API instead of self-hosting.

  7. Long sessions accumulate KV-cache memory. The autoregressive generation builds a KV-cache that grows with output length. For very long generations (5+ minutes of audio), the KV-cache can exceed available VRAM. Use the streaming API with chunked generation to bound memory usage.

Lesson 1: “The single biggest mistake teams make is using noisy reference audio. A 10-second clean recording beats a 2-minute noisy one every time. The model cannot fix what it cannot hear.” — Fish Audio engineering team

Lesson 2: “Emotion tags are powerful but test them first. [whisper] works on every model. [screaming] works on some. Always validate extreme tags on short samples before generating long content.” — Fish Speech community, r/LocalLLaMA

Lesson 3: “We spent two weeks debugging poor voice cloning quality. The fix was not in the model — it was in the reference audio. We re-recorded with a better microphone and the problem disappeared. Start with audio quality, not model tuning.” — Production engineer, anonymous

Advice for Getting Started

  1. Install Fish Speech on a machine with at least 12 GB of VRAM. An RTX 3090 or 4090 is ideal. Do not attempt CPU inference for anything beyond testing.
  2. Start with the WebUI (python -m tools.webui) to understand the workflow before moving to the API server. The WebUI shows you the three-step pipeline visually.
  3. Record a 10-15 second reference audio clip in a quiet room. Use a good microphone. Normalize to -3 dB. Transcribe it verbatim.
  4. Test voice cloning with a single short sentence before scaling to long content. Verify that the cloned voice sounds like the reference.
  5. Experiment with emotion tags on short phrases. Try [excited], [whisper], [sad], and [laughing] to understand how each affects prosody.
  6. Enable --compile from day one. The 2-5 minute compilation overhead on the first run is a one-time cost for 10x faster inference.
  7. For production, use the API server with Docker. Mount model weights on a persistent volume. Set startup probes with 120-second initial delay. Use --half for VRAM efficiency.

Next in the Open-Source AI Tools Mastery series: Meta AudioCraft

NL

Written by Nivant Labs Team

Engineer at Nivant Labs

Share this post

XLinkedInHN