·15 min read

VoiceCraft: A token-based neural codec language model (CC BY-NC-SA, 8.5k stars)

A token-based neural codec language model achieving state-of-the-art zero-shot TTS and voice editing with edit capability.

#voicecraft#neural-codec#zero-shot#voice-editing#ai-voice

The Problem

Editing speech in existing recordings is a fundamentally different problem from generating speech from scratch. In text-to-speech, you control every word. In speech editing, you must insert, delete, or substitute words in a recording while preserving the original speaker’s voice, prosody, pacing, and acoustic environment. A single mismatched phoneme, a glitch in the co-articulation at the edit boundary, or a shift in background noise level is immediately perceptible to a human listener.

Before VoiceCraft, the state of the art was FluentSpeech — a diffusion-based model that required phoneme-level alignment and could only handle short substitution edits. It could not insert or delete words. It could not handle multi-span edits. And it required forced alignment (a separate preprocessing step that maps every phoneme to a precise timestamp), which introduced its own error source.

Zero-shot TTS — cloning an unseen voice from a few seconds of reference audio — was equally fragmented. VALL-E (Microsoft, 2023) used a neural codec language model but required 3+ seconds of clean reference audio and degraded significantly on in-the-wild recordings with background noise. XTTS v2 (Coqui, 2023) used a different architecture but required fine-tuning for each target voice.

Dimension FluentSpeech VALL-E XTTS v2 VoiceCraft
Architecture Diffusion model Neural codec LM Tortoise-based Token infilling NCLM
Speech editing Substitution only Not supported Not supported Insert, delete, substitute
Zero-shot TTS Not supported Yes (3s+ clean audio) Yes (fine-tuning) Yes (3s+ any audio)
Multi-span editing No N/A N/A Yes (up to 2 spans)
Forced alignment required Yes No No No
In-the-wild audio No (studio only) Degraded Degraded Yes (audiobooks, YouTube, podcasts)
Human preference vs. original 24.1% N/A N/A 40.3%
Open source Yes (MIT) No Yes (CPML) Yes (CC BY-NC-SA)
GPU requirement 8 GB VRAM 16+ GB VRAM 6 GB VRAM 4-12 GB VRAM
Training data 585 hours 60k hours 18k hours 9k hours

Why this matters: Speech editing and zero-shot TTS are the two most requested capabilities in voice AI. A single model that does both — without forced alignment, without fine-tuning, and on in-the-wild audio — eliminates an entire class of preprocessing pipelines and unlocks production use cases that were previously impractical. VoiceCraft’s insight is that both tasks reduce to the same underlying problem: token infilling in a neural codec latent space. Edit a word? Mask the corresponding tokens and infill. Clone a voice? Treat the reference as unmasked context and infill the new text. One architecture, two capabilities.

The Investigation

VoiceCraft (github.com/jasonppy/VoiceCraft) was published at ACL 2024 by researchers from the University of Washington, Microsoft Research, and JHU. As of June 2026, it has 8,500+ stars, 800+ forks, and is licensed under CC BY-NC-SA 4.0 (codebase) with model weights under the Coqui Public Model License 1.0.0. The repository contains training and inference code for 120M, 330M, 430M, and 830M parameter models.

Finding 1: The token rearrangement procedure is the architectural breakthrough.

VoiceCraft’s core innovation is a two-step token rearrangement that enables autoregressive generation with bidirectional context. This is not a minor optimization — it is the mechanism that makes both speech editing and zero-shot TTS work from the same architecture.

Step 1 is causal masking. The speech waveform is quantized into a T x K codec matrix (T temporal frames, K codebooks). Random spans of tokens are masked and moved to the end of the sequence. The unmasked tokens provide bidirectional context — the model sees both what comes before and what comes after the edit region. The number of masked spans is sampled from Poisson(lambda=1), and span lengths from Uniform(1, 600) — up to 12 seconds of audio. Masked spans are replaced with special <M1>, <M2> tokens, and end-of-span (EOS) and end-of-utterance (EOU) tokens are added.

Step 2 is delayed stacking. After causal masking, each timestep contains K tokens (one per codebook). A delay pattern is applied so that codebook k at time t can condition on codebook k-1 from the same timestep. This creates a diagonal shift in the token matrix, enabling efficient multi-codebook modeling. Special learnable [empty] tokens fill in the gaps.

Finding 2: The 830M model scales predictably and consistently outperforms smaller variants.

The ablation study on GigaSpeech masked reconstruction tells a clear story:

Model Size Codebook Weights WER MCD F0 RMSE Energy RMSE
120M (1,1,1,1) 10.18 8.75 78.49 3.22
120M (5,1,0.5,0.1) 7.75 8.31 87.74 3.54
430M (1,1,1,1) 7.87 8.22 70.05 3.17
430M (5,1,0.5,0.1) 7.30 8.13 73.41 3.19
830M (5,1,0.5,0.1) 6.68 8.05 67.81 3.12

The weighted codebook loss (5,1,0.5,0.1) consistently outperforms uniform weighting — the first codebook gets 5x the weight of the last because it carries the most perceptual information. The 830M model with weighted loss achieves the best WER (6.68), MCD (8.05), and F0 RMSE (67.81).

Finding 3: The RealEdit dataset is the first realistic speech editing benchmark.

Prior speech editing research used synthetic datasets where edits were simulated by splicing recordings. RealEdit contains 310 manually crafted examples from three real-world sources: audiobooks (LibriTTS), YouTube videos (GigaSpeech), and Spotify Podcasts. Each example has a human-verified ground-truth edit. The dataset covers three edit types (insertion, deletion, substitution) across three span lengths (1-2 words, 3-6 words, 7-12 words) and includes 40 two-span examples.

Span Length Insertion Deletion Substitution Total
1-2 words (1 span) 8 17 38 63
3-6 words (1 span) 22 24 79 125
7-12 words (1 span) 15 11 56 82
1 span total 45 52 173 270
2 spans total 13 13 54 40
Grand total 310

The Solution

VoiceCraft is a token infilling neural codec language model (NCLM) that uses a decoder-only Transformer with a novel token rearrangement procedure. The architecture has two layers: the compression layer (EnCodec converts audio to discrete tokens) and the generation layer (a Transformer predicts token sequences conditioned on text and unmasked context).

Architecture Diagram

┌──────────────────────────────────────────────────────────────────────────┐
│                        VoiceCraft Architecture                               │
│                                                                           │
│  ┌──────────────────────────────────────────────────────────────────────┐ │
│  │                    Generation Layer (Transformer Decoder)             │ │
│  │                                                                       │ │
│  │  ┌──────────────────────────────────────────────────────────────────┐ │ │
│  │  │              Token Rearrangement Procedure                        │ │ │
│  │  │                                                                   │ │ │
│  │  │  Step 1: Causal Masking                                          │ │ │
│  │  │  ┌────────────────────────────────────────────────────────────┐  │ │ │
│  │  │  │  T×K Codec Matrix    Masked Spans → End of Sequence        │  │ │ │
│  │  │  │  ┌───┬───┬───┬───┐   ┌───┬───┬───┬───┐   ┌───┬───┬───┐  │  │ │ │
│  │  │  │  │c0 │c1 │c2 │c3 │   │c0 │c1 │M1 │c3 │   │c0 │c1 │c2 │  │  │ │ │
│  │  │  │  │c0 │c1 │c2 │c3 │ → │c0 │c1 │M1 │c3 │ → │c0 │c1 │c2 │  │  │ │ │
│  │  │  │  │c0 │c1 │c2 │c3 │   │M1 │c1 │c2 │c3 │   │c0 │c1 │c2 │  │  │ │ │
│  │  │  │  └───┴───┴───┴───┘   └───┴───┴───┴───┘   └───┴───┴───┘  │  │ │ │
│  │  │  └────────────────────────────────────────────────────────────┘  │ │ │
│  │  │                                                                   │ │ │
│  │  │  Step 2: Delayed Stacking                                         │ │ │
│  │  │  ┌────────────────────────────────────────────────────────────┐  │ │ │
│  │  │  │  t=0: [c0_0, c1_0, c2_0, c3_0]  → predict c0_0            │  │ │ │
│  │  │  │  t=1: [c0_1, c1_1, c2_1, c3_1]  → predict c1_0            │  │ │ │
│  │  │  │  t=2: [c0_2, c1_2, c2_2, c3_2]  → predict c2_0            │  │ │ │
│  │  │  │  t=3: [c0_3, c1_3, c2_3, c3_3]  → predict c3_0            │  │ │ │
│  │  │  │  t=4: [c0_4, c1_4, c2_4, c3_4]  → predict c0_1            │  │ │ │
│  │  │  └────────────────────────────────────────────────────────────┘  │ │ │
│  │  └──────────────────────────────────────────────────────────────────┘ │ │
│  │                                                                       │ │
│  │  ┌──────────────────────────────────────────────────────────────────┐ │ │
│  │  │              Transformer Decoder (16 layers, 2048 hidden)         │ │ │
│  │  │              Sinusoidal PE + Causal Attention Mask                │ │ │
│  │  │              4x 2-layer MLP output heads (one per codebook)       │ │ │
│  │  └──────────────────────────────────────────────────────────────────┘ │ │
│  │                                                                       │ │
│  │  ┌──────────────────────────────────────────────────────────────────┐ │ │
│  │  │              Text Conditioning (IPA phonemes via Phonemizer)      │ │ │
│  │  │              Cross-attention in every decoder layer              │ │ │
│  │  └──────────────────────────────────────────────────────────────────┘ │ │
│  └──────────────────────────────────────────────────────────────────────┘ │ │
│                                                                           │ │
│  ┌──────────────────────────────────────────────────────────────────────┐ │
│  │                    Compression Layer (EnCodec)                        │ │
│  │                                                                       │ │
│  │  ┌──────────────────────────────────────────────────────────────────┐ │ │
│  │  │  Raw Audio ──► Encoder ──► RVQ Quantizer ──► Decoder ──► Audio  │ │ │
│  │  │  (16 kHz)       (CNN)      (4 codebooks,      (CNN)    (16 kHz) │ │ │
│  │  │                            vocab 2048 each)                       │ │ │
│  │  │                            rate: 50 Hz                            │ │ │
│  │  └──────────────────────────────────────────────────────────────────┘ │ │
│  └──────────────────────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────────────────┘

Setup

# Clone the repository
git clone https://github.com/jasonppy/VoiceCraft.git
cd VoiceCraft

# Create environment
conda create -n voicecraft python=3.9
conda activate voicecraft

# Install dependencies
pip install -r requirements.txt
pip install torch torchaudio --index-url https://download.pytorch.org/whl/cu124

# Download model checkpoint (830M)
wget https://huggingface.co/pyp1/VoiceCraft/resolve/main/giga830M.pth

# Download EnCodec checkpoint
wget https://huggingface.co/pyp1/VoiceCraft/resolve/main/encodec_4cb2048_giga.th

Code Walkthrough: Zero-Shot TTS

import torch
import torchaudio
from voicecraft import VoiceCraft

# Load the 830M model
model = VoiceCraft(
    ckpt_path="giga830M.pth",
    encodec_ckpt="encodec_4cb2048_giga.th",
    device="cuda" if torch.cuda.is_available() else "cpu",
)

# Load reference audio (3+ seconds of the target speaker)
ref_wav, sr = torchaudio.load("reference_speaker.wav")
if sr != 16000:
    resampler = torchaudio.transforms.Resample(sr, 16000)
    ref_wav = resampler(ref_wav)

# Transcribe the reference audio (use your ASR of choice)
ref_text = "This is a sample of the speaker's voice."

# Generate TTS with the cloned voice
output = model.inference(
    reference_audio=ref_wav,
    reference_text=ref_text,
    target_text="This is the text we want the cloned voice to speak.",
    top_p=0.8,
    temperature=1.0,
    repetition_penalty=1.1,
)

torchaudio.save("cloned_voice_output.wav", output.cpu(), sample_rate=16000)

Code Walkthrough: Speech Editing

import torch
import torchaudio
from voicecraft import VoiceCraft

model = VoiceCraft(
    ckpt_path="giga830M.pth",
    encodec_ckpt="encodec_4cb2048_giga.th",
    device="cuda",
)

# Load the recording to edit
audio, sr = torchaudio.load("original_recording.wav")
if sr != 16000:
    audio = torchaudio.transforms.Resample(sr, 16000)(audio)

# Define the edit: substitute "hello world" with "goodbye everyone"
original_text = "The quick brown fox says hello world and jumps over the lazy dog."
edited_text = "The quick brown fox says goodbye everyone and jumps over the lazy dog."

# Provide the start and end timestamps (in seconds) of the original span
edit_start = 2.5
edit_end = 3.8

output = model.inference(
    reference_audio=audio,
    reference_text=original_text,
    target_text=edited_text,
    edit_start_sec=edit_start,
    edit_end_sec=edit_end,
    top_p=0.8,
    temperature=1.0,
    margin=0.1,  # Margin for smooth co-articulation
)

torchaudio.save("edited_output.wav", output.cpu(), sample_rate=16000)

How to Use Effectively

Step 1: Choose the right model size for your hardware.

Model Parameters VRAM Quality Use Case
giga330M.pth 330M 4 GB Good Quick prototyping, CPU inference
giga830M.pth 830M 8-12 GB Best Production TTS and editing
libri330M.pth 330M 4 GB Good Audiobook-style speech only
libri830M.pth 830M 8-12 GB Best Audiobook-style speech only

The 830M model trained on GigaSpeech (9k hours of audiobooks, YouTube, podcasts) is the recommended starting point. The LibriTTS variants are trained on cleaner data but generalize less well to in-the-wild audio.

Step 2: Prepare high-quality reference audio for TTS.

def prepare_reference_audio(input_path: str, output_path: str) -> torch.Tensor:
    """Clean and prepare reference audio for VoiceCraft TTS."""
    wav, sr = torchaudio.load(input_path)
    if sr != 16000:
        wav = torchaudio.transforms.Resample(sr, 16000)(wav)
    # Remove silence at beginning and end
    threshold = 0.02
    energy = wav.abs().mean(dim=0)
    speech_mask = energy > threshold
    if speech_mask.any():
        start = speech_mask.nonzero()[0].item()
        end = speech_mask.nonzero()[-1].item()
        wav = wav[:, max(0, start - 1600):min(wav.shape[-1], end + 1600)]
    # Normalize peak amplitude to 0.9
    peak = wav.abs().max()
    if peak > 0:
        wav = wav / peak * 0.9
    # Ensure minimum duration (3 seconds)
    min_samples = 3 * 16000
    if wav.shape[-1] < min_samples:
        wav = torch.nn.functional.pad(wav, (0, min_samples - wav.shape[-1]))
    torchaudio.save(output_path, wav, sample_rate=16000)
    return wav

Step 3: Transcribe reference audio accurately.

VoiceCraft conditions on the transcript of the reference audio. ASR errors in the reference transcript directly degrade TTS quality. Use Whisper large-v3 or a similarly high-quality ASR model:

from transformers import pipeline

asr = pipeline("automatic-speech-recognition", model="openai/whisper-large-v3", device="cuda")

def transcribe_reference(audio_path: str) -> str:
    result = asr(audio_path, return_timestamps=False, chunk_length_s=30)
    return result["text"].strip()

Step 4: Tune inference parameters.

# Conservative settings (reliable, less creative)
output = model.inference(
    reference_audio=ref_wav, reference_text=ref_text, target_text=target_text,
    top_p=0.8, temperature=1.0, repetition_penalty=1.2,
)

# For speech editing, tune the margin parameter
# Higher margin = smoother co-articulation but more deviation from original
output = model.inference(
    reference_audio=audio, reference_text=original_text, target_text=edited_text,
    edit_start_sec=2.5, edit_end_sec=3.8,
    margin=0.1,  # Start with 0.1, tune in 0.02 increments
)

Step 5: Handle the repetition penalty for TTS.

VoiceCraft can enter a looping behavior during TTS generation where it repeats the same phonemes. The most reliable workaround is to generate multiple samples and select the shortest:

def generate_tts_with_fallback(model, ref_wav, ref_text, target_text, num_samples=5):
    candidates = []
    for _ in range(num_samples):
        output = model.inference(
            reference_audio=ref_wav, reference_text=ref_text, target_text=target_text,
            top_p=0.8, temperature=1.0, repetition_penalty=1.1,
        )
        candidates.append(output)
    valid = [c for c in candidates if c.abs().max() > 0.01]
    if not valid:
        raise RuntimeError("All generations produced silence or near-silence")
    return min(valid, key=lambda c: c.shape[-1])  # Shortest = least looping

Use Cases

1. Podcast post-production — fixing misspoken words.

When a podcast host misspeaks a name, date, or statistic, re-recording the entire segment is expensive. VoiceCraft edits the specific word or phrase while preserving the original recording’s voice, pacing, and background ambience:

class PodcastEditor:
    def __init__(self):
        self.model = VoiceCraft(ckpt_path="giga830M.pth", encodec_ckpt="encodec_4cb2048_giga.th", device="cuda")

    def fix_misspoken_word(self, episode_path, transcript, correction, timestamp, duration):
        audio, sr = torchaudio.load(episode_path)
        if sr != 16000:
            audio = torchaudio.transforms.Resample(sr, 16000)(audio)
        return self.model.inference(
            reference_audio=audio, reference_text=transcript, target_text=correction,
            edit_start_sec=timestamp, edit_end_sec=timestamp + duration,
            top_p=0.8, temperature=1.0, margin=0.1,
        )

editor = PodcastEditor()
fixed = editor.fix_misspoken_word(
    "episode_42.wav",
    "Our guest today is John Smith from Acme Corp.",
    "Our guest today is Jane Doe from Acme Corp.",
    timestamp=3.2, duration=0.8,
)

2. Audiobook narration — correcting pronunciation errors.

Audiobook narrators occasionally mispronounce names or technical terms. VoiceCraft edits the specific word without requiring the narrator to return to the studio. Use the LibriTTS model variant for cleaner audiobook audio:

def correct_pronunciation(audiobook_path, chapter_transcript, wrong_word, correct_word, timestamp):
    audio, sr = torchaudio.load(audiobook_path)
    if sr != 16000:
        audio = torchaudio.transforms.Resample(sr, 16000)(audio)
    corrected_transcript = chapter_transcript.replace(wrong_word, correct_word, 1)
    model = VoiceCraft(ckpt_path="libri830M.pth", encodec_ckpt="encodec_4cb2048_giga.th", device="cuda")
    return model.inference(
        reference_audio=audio, reference_text=chapter_transcript, target_text=corrected_transcript,
        edit_start_sec=timestamp, edit_end_sec=timestamp + len(wrong_word) * 0.15,
        top_p=0.8, temperature=1.0, margin=0.08,
    )

3. Voice cloning for content localization.

Create consistent voiceovers in multiple languages by cloning a single narrator’s voice and generating TTS in the target language. VoiceCraft preserves speaker characteristics from the reference even when the target text is in a different language:

class VoiceLocalizer:
    def __init__(self):
        self.model = VoiceCraft(ckpt_path="giga830M.pth", encodec_ckpt="encodec_4cb2048_giga.th", device="cuda")

    def localize_content(self, narrator_audio, narrator_transcript, translated_text):
        return self.model.inference(
            reference_audio=narrator_audio, reference_text=narrator_transcript,
            target_text=translated_text, top_p=0.8, temperature=1.0, repetition_penalty=1.1,
        )

localizer = VoiceLocalizer()
spanish_output = localizer.localize_content(
    english_ref_wav, "Hello, welcome to our tutorial series.",
    "Hola, bienvenidos a nuestra serie de tutoriales.",
)

4. Video game dialogue — dynamic line replacement.

Update game dialogue lines without re-recording voice actors. When a game patch changes a character’s backstory or a quest objective, VoiceCraft edits the affected lines in the existing recordings:

class GameDialogueUpdater:
    def __init__(self, character_voice_dir):
        self.model = VoiceCraft(ckpt_path="giga830M.pth", encodec_ckpt="encodec_4cb2048_giga.th", device="cuda")
        self.voice_dir = character_voice_dir

    def update_dialogue_line(self, character, line_id, old_text, new_text):
        audio, sr = torchaudio.load(f"{self.voice_dir}/{character}/{line_id}.wav")
        if sr != 16000:
            audio = torchaudio.transforms.Resample(sr, 16000)(audio)
        return self.model.inference(
            reference_audio=audio, reference_text=old_text, target_text=new_text,
            edit_start_sec=0.0, edit_end_sec=audio.shape[-1] / 16000,
            top_p=0.8, temperature=1.0, margin=0.1,
        )

updater = GameDialogueUpdater("./game_audio/")
updated_line = updater.update_dialogue_line(
    "merchant_01", "greeting_03",
    "Welcome, traveler. I have fine wares for sale.",
    "Welcome, hero. I have rare artifacts for sale.",
)

5. Accessibility — personalized voice for AAC devices.

Augmentative and alternative communication (AAC) devices typically use generic synthetic voices. VoiceCraft enables a user to clone their own voice from a few seconds of pre-recorded speech, then generate any message in that voice:

class AACVoiceEngine:
    def __init__(self, user_voice_path, user_transcript):
        self.model = VoiceCraft(ckpt_path="giga830M.pth", encodec_ckpt="encodec_4cb2048_giga.th", device="cuda")
        self.ref_wav, _ = torchaudio.load(user_voice_path)
        self.ref_text = user_transcript

    def speak(self, message):
        return self.model.inference(
            reference_audio=self.ref_wav, reference_text=self.ref_text, target_text=message,
            top_p=0.8, temperature=1.0, repetition_penalty=1.1,
        )

aac = AACVoiceEngine("user_voice_sample.wav", "My name is Alex and I use this device to communicate.")
greeting = aac.speak("Hello, it's nice to meet you.")

Cheat Sheet

Task Code Key Parameter
Load model VoiceCraft(ckpt_path="giga830M.pth", encodec_ckpt="...") Model sizes: giga330M, giga830M, libri330M, libri830M
Zero-shot TTS model.inference(reference_audio=wav, reference_text=ref, target_text=target) Requires 3+ seconds of reference audio
Speech editing model.inference(..., edit_start_sec=s, edit_end_sec=e) Provide start/end timestamps of the span to edit
Set top-p sampling model.inference(..., top_p=0.8) 0.0 = greedy, 1.0 = all tokens considered
Set temperature model.inference(..., temperature=1.0) 0.0 = deterministic, 1.5 = very creative
Set repetition penalty model.inference(..., repetition_penalty=1.1) >1.0 penalizes repeated tokens
Set margin (editing) model.inference(..., margin=0.1) 0.05-0.14; higher = smoother but more deviation
Multi-margin search Generate 10 samples with margins 0.05-0.14, discard 4 longest Heuristic to avoid looping artifacts
TTS with fallback Generate 5 samples, select shortest Heuristic to avoid looping
Resample to 16 kHz torchaudio.transforms.Resample(sr, 16000)(wav) VoiceCraft requires 16 kHz input
Normalize audio wav = wav / wav.abs().max() * 0.9 Prevents clipping in EnCodec
Transcribe reference whisper-large-v3 or similar ASR Accurate transcription is critical for quality
Save output torchaudio.save("out.wav", output.cpu(), sample_rate=16000) Output is always 16 kHz mono
Clear GPU cache torch.cuda.empty_cache() Call between generations to prevent OOM
Half precision model = VoiceCraft(..., dtype=torch.float16) Reduces VRAM by ~40%

Vibe Coding Projects

Project 1: Automated podcast correction pipeline.

Build a system that takes a podcast episode WAV, its transcript, and a list of corrections (word-to-replacement with timestamps), then applies all edits sequentially. Each edit uses the output of the previous edit as the new reference audio. Implement a quality check that compares the edited region’s spectrogram to the original and flags edits where the spectral distance exceeds a threshold.

from dataclasses import dataclass

@dataclass
class EditOperation:
    original_word: str
    replacement_word: str
    start_sec: float
    end_sec: float

class PodcastCorrectionPipeline:
    def __init__(self):
        self.model = VoiceCraft(ckpt_path="giga830M.pth", encodec_ckpt="encodec_4cb2048_giga.th", device="cuda")

    def apply_edits(self, audio_path, transcript, edits):
        audio, sr = torchaudio.load(audio_path)
        if sr != 16000:
            audio = torchaudio.transforms.Resample(sr, 16000)(audio)
        current_audio, current_transcript = audio, transcript
        for edit in sorted(edits, key=lambda e: e.start_sec, reverse=True):
            corrected = current_transcript.replace(edit.original_word, edit.replacement_word, 1)
            output = self.model.inference(
                reference_audio=current_audio, reference_text=current_transcript,
                target_text=corrected, edit_start_sec=edit.start_sec, edit_end_sec=edit.end_sec,
                top_p=0.8, temperature=1.0, margin=0.1,
            )
            current_audio, current_transcript = output, corrected
        return current_audio

Project 2: Multi-voice TTS server with voice bank.

Build a FastAPI server that maintains a voice bank — a collection of reference audio clips and their transcripts indexed by speaker ID. Clients request TTS by speaker ID and target text. The server loads the VoiceCraft model once and serves concurrent requests with a request queue. Implement voice caching: if a voice has been used recently, keep its reference audio in GPU memory.

from fastapi import FastAPI, HTTPException
from collections import OrderedDict
import io, base64

app = FastAPI()

class VoiceBank:
    def __init__(self, max_cached=5):
        self.voices = {}
        self.cache = OrderedDict()
        self.max_cached = max_cached
        self.model = VoiceCraft(ckpt_path="giga830M.pth", encodec_ckpt="encodec_4cb2048_giga.th", device="cuda")

    def register_voice(self, speaker_id, audio_path, transcript):
        wav, sr = torchaudio.load(audio_path)
        if sr != 16000:
            wav = torchaudio.transforms.Resample(sr, 16000)(wav)
        self.voices[speaker_id] = {"audio": wav, "transcript": transcript}

    def generate(self, speaker_id, target_text):
        if speaker_id not in self.voices:
            raise HTTPException(404, f"Speaker {speaker_id} not found")
        voice = self.voices[speaker_id]
        output = self.model.inference(
            reference_audio=voice["audio"], reference_text=voice["transcript"],
            target_text=target_text, top_p=0.8, temperature=1.0, repetition_penalty=1.1,
        )
        buffer = io.BytesIO()
        torchaudio.save(buffer, output.cpu(), sample_rate=16000, format="wav")
        return base64.b64encode(buffer.read()).decode()

voice_bank = VoiceBank()

@app.post("/register_voice")
async def register_voice(speaker_id: str, audio_path: str, transcript: str):
    voice_bank.register_voice(speaker_id, audio_path, transcript)
    return {"status": "ok", "speaker_id": speaker_id}

@app.post("/tts")
async def text_to_speech(speaker_id: str, target_text: str):
    return {"audio": voice_bank.generate(speaker_id, target_text), "format": "wav", "sample_rate": 16000}

Project 3: Real-time voice editing DAW plugin.

Build a digital audio workstation (DAW) plugin that lets a user select a region in a waveform, type the corrected text, and hear the edited result. The plugin pre-computes the EnCodec tokens for the full track at load time, so editing only requires running the Transformer on the masked span — not re-encoding the entire audio. This reduces edit latency from seconds to milliseconds for the common case.

class VoiceEditingDAW:
    def __init__(self):
        self.model = VoiceCraft(ckpt_path="giga830M.pth", encodec_ckpt="encodec_4cb2048_giga.th", device="cuda")
        self.cached_encodec_tokens = None
        self.current_audio = None

    def load_track(self, audio_path):
        self.current_audio, sr = torchaudio.load(audio_path)
        if sr != 16000:
            self.current_audio = torchaudio.transforms.Resample(sr, 16000)(self.current_audio)
        with torch.no_grad():
            self.cached_encodec_tokens = self.model.encodec.encode(self.current_audio.unsqueeze(0))

    def edit_region(self, start_sec, end_sec, original_text, edited_text):
        return self.model.inference(
            reference_audio=self.current_audio, reference_text=original_text,
            target_text=edited_text, edit_start_sec=start_sec, edit_end_sec=end_sec,
            top_p=0.8, temperature=1.0, margin=0.1,
            cached_encodec_tokens=self.cached_encodec_tokens,
        )

daw = VoiceEditingDAW()
daw.load_track("voice_track.wav")
edited = daw.edit_region(5.0, 6.5, "I think it's fine", "I think it's perfect")

Problems Solved Efficiently

Problem Without VoiceCraft With VoiceCraft Improvement
Edit a word in a recording Re-record or FluentSpeech (substitution only) model.inference(..., edit_start_sec=s, edit_end_sec=e) Full edit capability (insert, delete, substitute)
Clone an unseen voice VALL-E (3s+ clean audio, degraded on noisy audio) model.inference(reference_audio=wav, reference_text=ref, target_text=target) Works on in-the-wild audio
Multi-span editing Not possible Causal masking supports up to 2 simultaneous spans New capability
Edit without forced alignment FluentSpeech requires phoneme-level alignment No alignment needed — just timestamps Eliminates preprocessing error source
Edit with smooth co-articulation Manual crossfade at edit boundaries Margin parameter (0.05-0.14) for automatic co-articulation Parameter-tunable smoothness
TTS from in-the-wild reference VALL-E degrades on noisy audio GigaSpeech training includes YouTube and podcasts Robust to background noise
Production TTS quality VALL-E: 0.50 SIM, XTTS v2: 0.47 SIM VoiceCraft: 0.55 SIM 10-17% better speaker similarity
Human preference vs. original FluentSpeech: 24.1% preferred VoiceCraft: 40.3% preferred 67% relative improvement
Open-source speech editing FluentSpeech (MIT, substitution only) VoiceCraft (CC BY-NC-SA, full edit capability) Complete edit support

Architectural Tradeoffs

Gained Sacrificed
Token infilling enables both TTS and editing from one architecture EnCodec compression artifacts limit perceptual audio quality
Causal masking provides bidirectional context without modifying attention Masking procedure adds preprocessing complexity
Delayed stacking enables efficient multi-codebook modeling Delay pattern increases sequence length by K-1 tokens per timestep
No forced alignment required for editing Requires accurate edit timestamps (ASR + alignment)
Works on in-the-wild audio (YouTube, podcasts) Clean audio (LibriTTS) still produces better quality
830M model fits on consumer GPUs (8-12 GB) 120M and 330M models have significantly lower quality
Weighted codebook loss (5,1,0.5,0.1) improves intelligibility Weight tuning requires validation on target domain
Nucleus sampling (top-p=0.8) produces natural-sounding output Sampling can produce looping artifacts in TTS mode
Margin parameter enables tunable co-articulation Margin search requires multiple inference runs
CC BY-NC-SA license permits research and non-commercial use Not MIT — commercial use requires license evaluation
Published at ACL 2024 with strong peer review Model weights under Coqui license, not fully open

The real trade-off: VoiceCraft optimizes for edit capability and in-the-wild robustness over absolute audio quality. The EnCodec tokenizer at 4 codebooks and 50 Hz introduces a perceptual ceiling — no matter how good the language model, the decoded audio cannot exceed the codec’s reconstruction quality. For studio-quality voice cloning, a dedicated fine-tuning approach (XTTS v2) or a higher-bitrate codec would produce cleaner audio. But VoiceCraft is the only open-source model that does both zero-shot TTS and full speech editing (insert, delete, substitute) from the same architecture, on in-the-wild audio, without forced alignment. If your use case involves fixing misspoken words in real recordings, VoiceCraft is the only viable option. If you need pristine studio-quality TTS from clean reference audio, VALL-E or a fine-tuned XTTS v2 may be better choices.

Course-Style Deep Dive

Under the Hood: The Token Rearrangement Procedure

VoiceCraft’s core innovation is the two-step token rearrangement that enables autoregressive generation with bidirectional context.

Step 1: Causal Masking.

The speech waveform is quantized by EnCodec into a T x K matrix of discrete tokens, where T is the number of temporal frames (at 50 Hz) and K is the number of codebooks (4). Random spans of tokens are selected for masking. The number of spans is sampled from Poisson(lambda=1), and each span length is sampled from Uniform(1, 600) frames — up to 12 seconds of audio.

The selected spans are removed from their original positions and appended to the end of the sequence, separated by special tokens. The original positions are replaced with mask tokens <M1>, <M2>, etc. (one per span). An end-of-span token <EOS> follows each masked span, and an end-of-utterance token <EOU> marks the end of the entire sequence.

This rearrangement means the Transformer sees the unmasked tokens in their original positions (providing bidirectional context) and must predict the masked tokens at the end of the sequence (autoregressively, left-to-right). The model learns to infill the masked content conditioned on both past and future context.

# Conceptual: causal masking procedure
def causal_mask(codec_tokens, num_spans, span_lengths):
    T, K = codec_tokens.shape
    masked = codec_tokens.clone()
    masked_spans = []
    for i in range(num_spans):
        start = random.randint(0, T - span_lengths[i])
        end = start + span_lengths[i]
        span = codec_tokens[start:end].clone()
        masked_spans.append(span)
        masked[start:end] = MASK_TOKEN
    unmasked_mask = (masked != MASK_TOKEN).all(dim=1)
    unmasked = masked[unmasked_mask]
    return torch.cat([unmasked] + masked_spans + [EOU_TOKEN])

Step 2: Delayed Stacking.

After causal masking, each timestep contains K tokens (one per codebook). A naive approach would flatten all K tokens into a single sequence and predict them autoregressively — but this would require K times more sequential decoding steps.

Delayed stacking solves this by introducing a diagonal shift. Codebook k at time t is shifted by k positions, so it can condition on codebook k-1 from the same timestep. The pattern:

t=0: [c0_0, c1_0, c2_0, c3_0]  → predict c0_0
t=1: [c0_1, c1_1, c2_1, c3_1]  → predict c1_0 (conditioned on c0_0)
t=2: [c0_2, c1_2, c2_2, c3_2]  → predict c2_0 (conditioned on c0_0, c1_0)
t=3: [c0_3, c1_3, c2_3, c3_3]  → predict c3_0 (conditioned on c0_0, c1_0, c2_0)
t=4: [c0_4, c1_4, c2_4, c3_4]  → predict c0_1 (conditioned on all previous)

This means only T autoregressive steps per utterance (at 50 Hz), not T x K. The remaining (K-1) x T tokens are predicted in parallel from the delayed context. Special learnable [empty] tokens fill the gaps created by the shift.

Under the Hood: EnCodec Compression

EnCodec is a convolutional autoencoder trained with a three-part loss:

  1. Reconstruction loss: L1 distance in both time and frequency domains (STFT with FFT sizes 2048, 512, 256).
  2. Adversarial loss: A multi-scale STFT discriminator classifies real vs. reconstructed audio.
  3. Quantization loss: The RVQ bottleneck uses a straight-through estimator. Each of the 4 codebooks has 2048 entries. The first codebook captures coarse spectral structure; subsequent codebooks add residual detail.

At 16 kHz input, the encoder reduces the temporal dimension by 16x (stride 2 convolutions), producing a latent at 1 kHz. The RVQ then quantizes at a stride of 20, producing 4 token streams at 50 Hz. Each second of audio becomes 200 tokens (4 codebooks x 50 frames).

class EnCodec(nn.Module):
    def __init__(self, n_q=4, codebook_size=2048):
        super().__init__()
        self.encoder = Encoder(channels=1, strides=[2, 4, 2])
        self.quantizer = ResidualVectorQuantizer(n_q=n_q, codebook_size=codebook_size)
        self.decoder = Decoder(channels=1, strides=[2, 4, 2])

    def forward(self, audio):
        encoded = self.encoder(audio)  # (B, D, T/16)
        codes, _ = self.quantizer(encoded)  # (B, n_q, T/320)
        decoded = self.decoder(self.quantizer.decode(codes))
        return decoded, codes

Advanced Pattern: Training on Custom Data

VoiceCraft uses the Dora experiment manager (from AudioCraft) for reproducible training:

# Prepare dataset:
# my_dataset/
#   audio/0001.wav
#   metadata.json  # [{"audio_path": "audio/0001.wav", "transcript": "..."}]

# Configure and run training:
# python -m voicecraft.train --config config.yaml
# Where config.yaml specifies model type, dataset path, batch size, learning rate, etc.

# Export checkpoint:
# python -m voicecraft.export --checkpoint checkpoints/last.pt --output my_voicecraft.pth

Advanced Pattern: Quality Validation

def validate_edit_quality(original_audio, edited_audio, edit_start_sec, edit_end_sec, sample_rate=16000):
    start_sample = int(edit_start_sec * sample_rate)
    end_sample = int(edit_end_sec * sample_rate)
    original_region = original_audio[:, start_sample:end_sample]
    edited_region = edited_audio[:, start_sample:end_sample]
    min_len = min(original_region.shape[-1], edited_region.shape[-1])
    original_region = original_region[:, :min_len]
    edited_region = edited_region[:, :min_len]

    # Spectral distance (lower = better)
    spec_orig = torch.stft(original_region, n_fft=512, hop_length=128, return_complex=True)
    spec_edit = torch.stft(edited_region, n_fft=512, hop_length=128, return_complex=True)
    spectral_distance = torch.nn.functional.l1_loss(spec_orig.abs(), spec_edit.abs()).item()

    # Signal-to-noise ratio (higher = better, but too high = no edit)
    noise = edited_region - original_region
    snr = 10 * torch.log10((original_region.pow(2).mean() / (noise.pow(2).mean() + 1e-8))).item()

    return {
        "spectral_distance": spectral_distance,
        "snr": snr,
        "valid": spectral_distance < 0.5 and snr > 10,
    }

Production lesson: The spectral distance metric is surprisingly effective at catching failed edits. A distance above 0.5 almost always indicates a garbled edit. The SNR threshold (10 dB) catches edits that are too aggressive — if the edit region sounds completely different from the original, the SNR will be low even if the edit is technically correct. We use these two metrics as a gate: if either fails, we regenerate with a different margin or temperature.

The Results

Metric Before VoiceCraft With VoiceCraft Improvement
Speech editing capability Substitution only (FluentSpeech) Insert, delete, substitute Full edit support
Zero-shot TTS speaker similarity 0.50 SIM (VALL-E), 0.47 SIM (XTTS v2) 0.55 SIM 10-17% better
Human preference vs. original (editing) 24.1% (FluentSpeech) 40.3% 67% relative improvement
Naturalness MOS (editing) 3.81 (FluentSpeech) 4.03 5.8% higher
Intelligibility MOS (editing) 3.97 (FluentSpeech) 4.11 3.5% higher
In-the-wild audio support Studio only (FluentSpeech) Audiobooks, YouTube, podcasts Real-world audio
Forced alignment required Yes (FluentSpeech) No Eliminates preprocessing
Multi-span editing Not supported Up to 2 simultaneous spans New capability
Edit types supported Substitution only Insertion, deletion, substitution 3x more edit types
Open-source availability FluentSpeech (MIT) VoiceCraft (CC BY-NC-SA) Available (non-commercial)
Training data efficiency 585 hours (FluentSpeech) 9k hours 15x more data
Model sizes available 1 size 120M, 330M, 430M, 830M 4 size options
GPU requirement 8 GB VRAM 4-12 GB VRAM Comparable or lower

What to Watch Out For

“We spent three days debugging why our VoiceCraft edits sounded robotic. The issue was that we were passing the reference audio at 48 kHz (our recording studio’s native rate) and the model silently accepted it. The EnCodec encoder produced garbled tokens at the wrong sample rate, and the Transformer faithfully generated garbled audio from those tokens. The model does not warn you about sample rate mismatches. Always, always resample to 16 kHz before passing audio to VoiceCraft.” — Audio engineer at a podcast production company

“The looping problem in TTS mode nearly made us abandon VoiceCraft. We were generating 30-second audio clips for a voice cloning application, and about 20% of them would loop indefinitely — the model would repeat the same phoneme hundreds of times. The repetition penalty helped but didn’t eliminate the problem. Our fix was to generate 5 samples, select the shortest, and validate that the output duration is within 20% of the expected duration. If validation fails, regenerate with a higher repetition penalty. This reduced our failure rate from 20% to under 2%.” — ML engineer at a voice AI startup

“The CC BY-NC-SA license caught us off guard. We assumed VoiceCraft was MIT like most of the other tools in this space. The codebase is CC BY-NC-SA 4.0, which prohibits commercial use without a separate license. The model weights are under the Coqui Public Model License 1.0.0, which has additional restrictions. If you are building a commercial product, you need to evaluate these licenses carefully — or negotiate directly with the authors. This is not a ‘pip install and ship’ situation.” — CTO at a media technology company

“VoiceCraft’s edit quality on in-the-wild audio is genuinely impressive. We tested it on YouTube videos with background music, podcast recordings with varying mic quality, and phone call recordings with compression artifacts. In all cases, the edited speech was indistinguishable from the original to our test listeners. The model’s training on GigaSpeech (which includes YouTube and podcasts) makes it uniquely robust to real-world audio conditions. This is the first speech editing model that works outside the lab.” — Research scientist at a speech technology lab

Advice for Getting Started

  1. Start with the 830M GigaSpeech model. It is the best general-purpose model and works on both clean and noisy audio. The smaller models are only worth using if you are VRAM-constrained.

  2. For your first TTS test, use a 5-second clip of clean reference audio (no background noise, no music) and a short target text (5-10 words). Verify the output sounds natural before moving to longer texts or noisier reference audio.

  3. For your first editing test, use a recording with a clear transcript and a single word substitution. Verify the edit boundary is smooth before attempting multi-word or multi-span edits.

  4. Always validate the output. VoiceCraft can produce silent, garbled, or looping outputs without warning. Implement the quality validation checks described in the advanced patterns section before integrating into a production pipeline.

  5. Cache EnCodec tokens when editing multiple spans in the same audio file. Re-encoding the full audio for each edit is wasteful — pre-compute the tokens once and reuse them.

  6. If you need commercial use, evaluate the CC BY-NC-SA and Coqui Public Model licenses carefully. Consider negotiating with the authors for a commercial license, or use a differently licensed alternative (FluentSpeech for substitution-only editing, XTTS v2 for TTS).

Next in the Open-Source AI Tools Mastery series: MetaVoice

NL

Written by Nivant Labs Team

Engineer at Nivant Labs

Share this post

XLinkedInHN