Meta AudioCraft: Meta's audio generation toolkit (MIT, 21k stars)
Meta's audio generation toolkit for generating music, sound effects, and audio from text descriptions with MusicGen and AudioGen models.
The Problem
Generating audio from text descriptions is fundamentally harder than generating images. Audio is a one-dimensional time series sampled at 44,100 Hz — a five-second clip is 220,500 samples. Music adds the complexity of harmony, rhythm, timbre, and structure across multiple instruments. Sound effects require modeling transient events (footsteps, door slams) against continuous backgrounds (wind, rain). Every audio domain has different temporal dynamics, and no single architecture handled all of them.
Before AudioCraft, the landscape was fragmented. Google’s MusicLM used a cascaded approach: a semantic model (w2v-BERT) extracted high-level representations, a coarse acoustic model generated tokens, and a fine-grained model produced the waveform. This three-stage pipeline required separate training for each stage, compounding errors at every step. Riffusion wrapped a fine-tuned Stable Diffusion spectrogram generator around a vocoder — treating audio as images, which introduced phase artifacts and limited temporal coherence. Jukebox (OpenAI) used hierarchical VQ-VAEs with multiple Transformers at different temporal resolutions, requiring 5+ GPUs for inference and producing 44 kHz audio at the cost of 8+ seconds per second of generation.
| Dimension | MusicLM (Google) | Riffusion | Jukebox (OpenAI) | AudioCraft |
|---|---|---|---|---|
| Architecture | 3-stage cascaded (semantic + coarse + fine) | Spectrogram image + vocoder | Hierarchical VQ-VAE + 3 Transformers | Single-stage autoregressive Transformer |
| Model count | 3 separate models | 2 (diffusion + vocoder) | 4 (VAE + 3 LMs) | 1 (LM over EnCodec tokens) |
| Inference speed | 30+ sec for 10 sec audio | ~5 sec for 5 sec | 8+ sec per sec of audio | ~5 sec for 10 sec (small) |
| Training data | 280k hours (unlabeled) | ~10k hours (labeled) | 1.2M songs | 20k hours (licensed music) |
| Sampling rate | 24 kHz | 44.1 kHz (via vocoder) | 44.1 kHz | 32 kHz |
| Controllability | Text only | Text + seed spectrogram | Genre/artist/style | Text + melody (chromagram) |
| Open source | No | Yes (Apache 2.0) | Yes (MIT) | Yes (MIT) |
| GPU requirement | N/A (closed) | 4 GB VRAM | 16+ GB VRAM | 2-8 GB VRAM |
| License | Proprietary | Apache 2.0 | MIT | MIT |
Why this matters: Audio generation is a latency-sensitive, compute-bound problem. A three-stage cascaded model multiplies both inference time and error propagation — if the semantic model misclassifies a musical phrase, the coarse and fine models cannot recover. A single-stage model that operates directly on compressed audio tokens eliminates this failure mode entirely. AudioCraft’s insight — that a single autoregressive Transformer over neural codec tokens is sufficient for high-quality music and sound generation — is the architectural breakthrough that made real-time audio generation practical on consumer GPUs.
The Investigation
Meta AudioCraft (github.com/facebookresearch/audiocraft) launched in August 2023 as a unified PyTorch library for generative audio research. As of June 2026, it has 23,300+ stars, 2,600+ forks, and is licensed under MIT. The repository contains inference and training code for eight models: MusicGen, AudioGen, EnCodec, Multi-Band Diffusion, MAGNeT, AudioSeal, MusicGen Style, and JASCO.
Finding 1: EnCodec is the foundational layer that makes everything else possible.
EnCodec is a neural audio codec that compresses raw waveforms into discrete token streams. It uses an autoencoder with a Residual Vector Quantization (RVQ) bottleneck. The encoder maps audio into a latent space, quantizes it across multiple RVQ codebooks (4 for 32 kHz, 8 for stereo), and the decoder reconstructs the waveform. The key metric: EnCodec compresses 32 kHz audio to 50 tokens per second per codebook — a 640x compression ratio — with perceptual quality that rivals MP3 at higher bitrates.
The RVQ design is critical. Each codebook adds a residual refinement layer. The first codebook captures the coarse structure (loudness, pitch contour). Subsequent codebooks add timbral detail, transients, and high-frequency content. This hierarchical quantization means a language model can generate the first codebook autoregressively and predict the remaining codebooks in parallel with a small delay — the core innovation of MusicGen.
Finding 2: MusicGen’s single-stage design is the architectural breakthrough.
Prior work (MusicLM) used a three-stage cascade: a semantic model (w2v-BERT) produced high-level representations, a coarse acoustic model generated EnCodec tokens, and a fine-grained model refined them. MusicGen collapses this into a single autoregressive Transformer that directly predicts EnCodec tokens conditioned on text embeddings.
The critical innovation is the codebook interleaving pattern. With 4 codebooks at 50 Hz, a naive autoregressive model would need to predict 200 tokens per second of audio (4 codebooks x 50 steps). MusicGen introduces a small delay between codebooks — it predicts codebook 0 at step t, codebook 1 at step t+1, codebook 2 at step t+2, codebook 3 at step t+3, then codebook 0 at step t+4. This means only 50 autoregressive steps per second of audio, with the remaining 150 tokens predicted in parallel from the delayed context. This 4x reduction in sequential decoding is what makes MusicGen fast enough for interactive use.
Finding 3: MAGNeT and JASCO extend the architecture for non-autoregressive and conditioned generation.
MAGNeT (January 2024) replaces the autoregressive decoder with a masked non-autoregressive Transformer. Instead of predicting tokens left-to-right, it starts with all tokens masked and iteratively unmask them based on confidence scores. This reduces inference latency by 3-5x compared to MusicGen at equivalent quality, at the cost of slightly less coherent long-range structure.
JASCO (June 2024) introduces Flow Matching for temporally controlled generation. It accepts symbolic conditioning (chord progressions, drum patterns, melody lines) alongside text prompts, enabling precise control over musical structure. The conditioning is injected via cross-attention at specific time positions, allowing the model to follow a chord progression bar-by-bar.
| Model | Release | Architecture | Parameters | Conditioning | Inference Speed |
|---|---|---|---|---|---|
| MusicGen | Aug 2023 | Autoregressive Transformer | 300M / 1.5B / 3.3B | Text, melody | 1x (baseline) |
| AudioGen | Aug 2023 | Autoregressive Transformer | 300M / 1.5B | Text | 1x |
| MAGNeT | Jan 2024 | Masked non-autoregressive Transformer | 300M / 1.5B | Text | 3-5x faster |
| JASCO | Jun 2024 | Flow Matching Transformer | 400M / 1B | Text, chords, drums, melody | 2-3x faster |
| MusicGen Style | 2024 | Autoregressive + style encoder | 1.5B / 3.3B | Text + reference audio | 1x |
| AudioSeal | 2024 | Watermarking encoder/decoder | Small | Audio + watermark key | Real-time |
The Solution
AudioCraft provides a unified PyTorch library for training and running generative audio models. The architecture is a two-layer stack: the compression layer (EnCodec converts audio to discrete tokens) and the generation layer (language models predict token sequences from text conditioning).
Architecture Diagram
┌──────────────────────────────────────────────────────────────────┐
│ AudioCraft Library │
│ │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ Generation Layer │ │
│ │ │ │
│ │ ┌──────────────┐ ┌──────────────┐ ┌────────────────────┐ │ │
│ │ │ MusicGen │ │ AudioGen │ │ MAGNeT │ │ │
│ │ │ (AR Trans.) │ │ (AR Trans.) │ │ (Masked Non-AR) │ │ │
│ │ │ 300M-3.3B │ │ 300M-1.5B │ │ 300M-1.5B │ │ │
│ │ └──────┬───────┘ └──────┬───────┘ └────────┬───────────┘ │ │
│ │ │ │ │ │ │
│ │ ┌──────▼─────────────────▼────────────────────▼───────────┐ │ │
│ │ │ Text Encoder (T5 / FLAN-T5) │ │ │
│ │ │ Chromagram Encoder (melody) │ │ │
│ │ └──────────────────────────┬──────────────────────────────┘ │ │
│ └─────────────────────────────┼────────────────────────────────┘ │
│ │ │
│ ┌─────────────────────────────▼────────────────────────────────┐ │
│ │ Compression Layer │ │
│ │ │ │
│ │ ┌──────────────────────────────────────────────────────────┐ │ │
│ │ │ EnCodec │ │ │
│ │ │ │ │ │
│ │ │ Raw Audio ──► Encoder ──► RVQ Quantizer ──► Decoder ──► │ │ │
│ │ │ (32 kHz) (CNN) (4-8 codebooks) (CNN) │ │ │
│ │ │ │ │ │ │
│ │ │ Discrete Tokens │ │ │
│ │ │ (50 tokens/sec/codebook) │ │ │
│ │ └──────────────────────────────────────────────────────────┘ │ │
│ │ │ │
│ │ ┌──────────────────────────────────────────────────────────┐ │ │
│ │ │ Multi-Band Diffusion (optional) │ │ │
│ │ │ Enhances decoded audio by denoising in subbands │ │ │
│ │ └──────────────────────────────────────────────────────────┘ │ │
│ └───────────────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────────────┘Setup
# Core install
pip install 'audiocraft>=1.2.0' torch torchaudio
# With CUDA support
pip install torch torchaudio --index-url https://download.pytorch.org/whl/cu124
# For development (clone repo)
git clone https://github.com/facebookresearch/audiocraft.git
cd audiocraft
pip install -e .
# Optional: xformers for memory-efficient attention
pip install xformers
# Optional: demucs for audio separation (preprocessing)
pip install demucsCode Walkthrough: Basic Text-to-Music
import torch
import torchaudio
from audiocraft.models import MusicGen
# Load the smallest model (300M parameters) — works on 4 GB VRAM
model = MusicGen.get_pretrained("facebook/musicgen-small")
# Set generation parameters
model.set_generation_params(
duration=10.0, # Generate 10 seconds of audio
temperature=1.0, # 0.0 = deterministic, 1.0 = creative
top_k=250, # Top-K sampling
top_p=0.9, # Nucleus sampling
guidance_scale=3.0, # Classifier-free guidance scale
)
# Generate from text prompt
wav = model.generate(["upbeat electronic dance music with synthesizer leads"])
# wav is a list of tensors, each shape (1, channels, samples)
# Save to file
torchaudio.save(
"output.wav",
wav[0].cpu(),
sample_rate=32000,
)
# Generate multiple variations
wavs = model.generate(
[
"melancholic piano ballad with strings, slow tempo",
"funky disco groove with slap bass and brass section",
"dark ambient drone with evolving pads and field recordings",
],
progress=True, # Show progress bar
)
for i, w in enumerate(wavs):
torchaudio.save(f"output_{i}.wav", w.cpu(), sample_rate=32000)Code Walkthrough: Text-to-Sound Effects
from audiocraft.models import AudioGen
# Load AudioGen model
model = AudioGen.get_pretrained("facebook/audiogen-medium")
model.set_generation_params(
duration=5.0,
temperature=0.8, # Lower temperature for more deterministic sounds
guidance_scale=3.0,
)
# Generate sound effects
wavs = model.generate([
"dog barking in the distance",
"rain falling on a tin roof",
"footsteps on wooden floor",
"car engine starting and idling",
])
for i, w in enumerate(wavs):
torchaudio.save(f"sfx_{i}.wav", w.cpu(), sample_rate=16000)Code Walkthrough: Melody-Conditioned Generation
import torchaudio
from audiocraft.models import MusicGen
# Load the melody-capable model
model = MusicGen.get_pretrained("facebook/musicgen-melody")
model.set_generation_params(
duration=15.0,
temperature=0.7,
guidance_scale=3.0,
)
# Load a reference melody (WAV file)
melody_wav, sr = torchaudio.load("hummed_melody.wav")
# Resample to 32 kHz if needed
if sr != 32000:
resampler = torchaudio.transforms.Resample(sr, 32000)
melody_wav = resampler(melody_wav)
# Generate music that follows the melody
wav = model.generate_with_chroma(
descriptions=["jazz piano trio with walking bass"],
melody_wavs=[melody_wav],
melody_sample_rate=32000,
progress=True,
)
torchaudio.save("melody_output.wav", wav[0].cpu(), sample_rate=32000)How to Use Effectively
Step 1: Choose the right model for your task.
| Task | Model | Size | VRAM | Quality |
|---|---|---|---|---|
| Quick music prototype | musicgen-small |
300M | 2 GB | Decent |
| Production music | musicgen-medium |
1.5B | 4 GB | Good |
| High-quality music | musicgen-large |
3.3B | 8 GB | Excellent |
| Music with melody control | musicgen-melody |
1.5B | 4 GB | Good |
| Sound effects | audiogen-medium |
1.5B | 4 GB | Good |
| Fast generation | magnet-small-10secs |
300M | 2 GB | Good (3-5x faster) |
| Conditioned generation | jasco-chords-drums-1B |
1B | 6 GB | Excellent |
Step 2: Optimize generation parameters.
# Conservative settings (reliable, less creative)
model.set_generation_params(
duration=10.0,
temperature=0.7,
top_k=200,
top_p=0.85,
guidance_scale=2.0,
)
# Creative settings (more variety, occasional artifacts)
model.set_generation_params(
duration=10.0,
temperature=1.2,
top_k=300,
top_p=0.95,
guidance_scale=4.0,
)
# Fast generation (lower quality, higher speed)
model.set_generation_params(
duration=10.0,
temperature=1.0,
top_k=250,
top_p=0.9,
guidance_scale=3.0,
max_new_tokens=500, # Limit tokens for speed
)Step 3: Write effective prompts.
Good prompts are specific, structured, and describe the musical elements you want:
# Good — specific genre, instruments, mood, tempo
prompts = [
"upbeat tropical house with steel drums, synth pads, and a 124 BPM four-on-the-floor kick",
"melancholic ambient drone with evolving pad swells and distant field recordings",
"aggressive industrial techno with distorted bass, metallic percussion, and 140 BPM",
"lo-fi hip hop beat with vinyl crackle, jazz guitar sample, and soft 808 kick",
]
# Bad — too vague, no musical structure
bad_prompts = [
"nice music",
"song",
"good beat",
"something relaxing",
]Step 4: Handle stereo output.
from audiocraft.models import MusicGen
# Load stereo model
model = MusicGen.get_pretrained("facebook/musicgen-stereo-medium")
model.set_generation_params(duration=10.0)
wav = model.generate(["wide cinematic orchestral with sweeping strings"])
# wav shape: (1, 2, samples) — stereo channels
# Save as stereo WAV
torchaudio.save("stereo_output.wav", wav[0].cpu(), sample_rate=32000)Step 5: Use continuation for longer generations.
# Generate a segment, then continue from its end
segment_1 = model.generate(["verse: mellow acoustic guitar with soft percussion"])
# Continue from the last state (not supported directly — use sliding window)
# Instead, generate longer in one call
model.set_generation_params(duration=30.0)
full_track = model.generate(["full track: mellow acoustic folk song with verse and chorus"])Use Cases
1. Game audio prototyping.
Generate sound effects and background music for game levels during development, replacing placeholder audio with AI-generated assets before the composer delivers final tracks:
from audiocraft.models import AudioGen, MusicGen
import torchaudio
import os
class GameAudioPrototyper:
def __init__(self):
self.sfx_model = AudioGen.get_pretrained("facebook/audiogen-medium")
self.music_model = MusicGen.get_pretrained("facebook/musicgen-medium")
def generate_sfx_batch(self, sound_descriptions: list, output_dir: str):
self.sfx_model.set_generation_params(duration=3.0, temperature=0.7)
wavs = self.sfx_model.generate(sound_descriptions)
os.makedirs(output_dir, exist_ok=True)
for desc, wav in zip(sound_descriptions, wavs):
safe_name = desc.replace(" ", "_")[:30]
torchaudio.save(f"{output_dir}/{safe_name}.wav", wav.cpu(), 16000)
def generate_bgm(self, mood: str, duration: float = 30.0):
self.music_model.set_generation_params(duration=duration, temperature=0.9)
wav = self.music_model.generate([f"{mood} game background music, looping"])
return wav[0]
prototyper = GameAudioPrototyper()
prototyper.generate_sfx_batch([
"sword clashing with shield",
"magic spell whoosh with sparkles",
"heavy door creaking open",
"footsteps on gravel",
], "./game_assets/sfx/")2. Music production pre-visualization.
Generate rough mixes of song ideas to audition arrangements, genres, and instrumentations before recording:
from audiocraft.models import MusicGen
import torchaudio
class SongIdeaGenerator:
def __init__(self):
self.model = MusicGen.get_pretrained("facebook/musicgen-large")
def generate_arrangement_variations(self, base_idea: str):
variations = [
f"{base_idea}, acoustic folk arrangement",
f"{base_idea}, electronic synthwave version",
f"{base_idea}, orchestral cinematic arrangement",
f"{base_idea}, stripped-down piano and vocal",
]
self.model.set_generation_params(duration=15.0, temperature=1.1)
wavs = self.model.generate(variations)
return wavs
def generate_section(self, description: str, duration: float = 8.0):
self.model.set_generation_params(duration=duration, temperature=0.8)
wav = self.model.generate([description])
return wav[0]
# Generate intro, verse, and chorus separately
idea_gen = SongIdeaGenerator()
intro = idea_gen.generate_section("intro: atmospheric pads with reverse cymbal swell", 6.0)
verse = idea_gen.generate_section("verse: fingerpicked acoustic guitar with soft shaker", 12.0)
chorus = idea_gen.generate_section("chorus: full band with driving drums and electric guitar", 12.0)3. Accessibility — audio descriptions for visually impaired users.
Generate contextual soundscapes that describe visual scenes for accessibility applications:
from audiocraft.models import AudioGen
class AudioDescriber:
def __init__(self):
self.model = AudioGen.get_pretrained("facebook/audiogen-medium")
def describe_scene(self, scene_text: str) -> torch.Tensor:
"""Generate an audio description of a visual scene."""
self.model.set_generation_params(duration=4.0, temperature=0.6)
wav = self.model.generate([scene_text])
return wav[0]
def describe_image_sequence(self, captions: list) -> list:
"""Generate audio for a sequence of image captions."""
self.model.set_generation_params(duration=3.0, temperature=0.6)
return self.model.generate(captions)
describer = AudioDescriber()
city_sound = describer.describe_scene(
"busy city street with traffic, pedestrian chatter, and distant sirens"
)
nature_sound = describer.describe_scene(
"peaceful forest stream with birds chirping and leaves rustling in wind"
)4. Podcast and video post-production.
Generate custom transitions, background beds, and sound effects for video editing without licensing music:
from audiocraft.models import MusicGen, AudioGen
import torchaudio
class PostProductionAudio:
def __init__(self):
self.music = MusicGen.get_pretrained("facebook/musicgen-medium")
self.sfx = AudioGen.get_pretrained("facebook/audiogen-medium")
def generate_transition(self, mood: str, duration: float = 2.0):
self.music.set_generation_params(duration=duration, temperature=0.7)
wav = self.music.generate([f"{mood} transition whoosh with riser"])
return wav[0]
def generate_background_bed(self, mood: str, duration: float = 60.0):
self.music.set_generation_params(duration=duration, temperature=0.8)
wav = self.music.generate([f"{mood} podcast background music, low profile, repetitive"])
return wav[0]
def generate_stinger(self, description: str):
self.sfx.set_generation_params(duration=1.5, temperature=0.5)
wav = self.sfx.generate([f"{description}, short sharp sound"])
return wav[0]
pp = PostProductionAudio()
bed = pp.generate_background_bed("upbeat corporate", 60.0)
stinger = pp.generate_stinger("notification ping")5. Educational audio content creation.
Generate examples of musical concepts, instrument sounds, and audio phenomena for music education:
from audiocraft.models import MusicGen
import torchaudio
class MusicEducationGenerator:
def __init__(self):
self.model = MusicGen.get_pretrained("facebook/musicgen-medium")
def demonstrate_genre(self, genre: str):
self.model.set_generation_params(duration=10.0, temperature=0.8)
wav = self.model.generate([f"example of {genre} music, clear and textbook"])
return wav[0]
def demonstrate_instrument(self, instrument: str):
self.model.set_generation_params(duration=6.0, temperature=0.5)
wav = self.model.generate([f"single {instrument} playing a melody, no other instruments"])
return wav[0]
def demonstrate_concept(self, concept: str):
self.model.set_generation_params(duration=8.0, temperature=0.6)
wav = self.model.generate([f"music theory example: {concept}, clear and educational"])
return wav[0]
edu = MusicEducationGenerator()
edu.demonstrate_genre("baroque classical")
edu.demonstrate_instrument("acoustic guitar fingerpicking")
edu.demonstrate_concept("call and response between two instruments")Cheat Sheet
| Task | Code | Key Parameter |
|---|---|---|
| Load MusicGen | MusicGen.get_pretrained("facebook/musicgen-small") |
Model sizes: small, medium, large, melody, stereo |
| Load AudioGen | AudioGen.get_pretrained("facebook/audiogen-medium") |
Sizes: small, medium |
| Load MAGNeT | MusicGen.get_pretrained("facebook/magnet-small-10secs") |
Uses same API as MusicGen |
| Set duration | model.set_generation_params(duration=10.0) |
Max 30 seconds (positional embedding limit) |
| Set temperature | model.set_generation_params(temperature=1.0) |
0.0 = deterministic, 1.5 = very creative |
| Set guidance scale | model.set_generation_params(guidance_scale=3.0) |
2-4 is sweet spot; higher = more literal |
| Generate from text | model.generate(["prompt"]) |
Returns list of tensors |
| Generate with melody | model.generate_with_chroma(descriptions=[...], melody_wavs=[...]) |
Requires melody-capable model |
| Batch generation | model.generate([prompt1, prompt2, prompt3]) |
All generated in one forward pass |
| Save audio | torchaudio.save("out.wav", wav.cpu(), sample_rate=32000) |
MusicGen: 32 kHz, AudioGen: 16 kHz |
| Set random seed | torch.manual_seed(42) |
Pass before generate() for reproducibility |
| Enable progress bar | model.generate([...], progress=True) |
Shows tqdm progress during generation |
| Use half precision | model = MusicGen.get_pretrained(..., device="cuda", dtype=torch.float16) |
Reduces VRAM by ~40% |
| Compile model | model.lm = torch.compile(model.lm, mode="reduce-overhead") |
15-25% speedup after warmup |
| Export to ONNX | torch.onnx.export(model.lm, ...) |
For production serving without PyTorch |
| Audio watermarking | from audiocraft.models import AudioSeal |
Embed/detect watermarks in generated audio |
Vibe Coding Projects
Project 1: Real-time audio generation API with model routing.
Build a FastAPI service that exposes a unified audio generation endpoint, routing requests to the optimal model based on the prompt content. Short sound effects go to AudioGen. Music prompts go to MusicGen. Prompts mentioning specific instruments or genres route to the large model for quality. Implement request queuing with Redis for long generations, and cache generated audio by prompt hash to avoid regenerating identical requests.
from fastapi import FastAPI, HTTPException
from audiocraft.models import MusicGen, AudioGen
import torch
import torchaudio
import io
import base64
import hashlib
app = FastAPI()
class AudioRouter:
def __init__(self):
self.models = {}
self.cache = {}
def _get_model(self, prompt: str):
sfx_keywords = ["sound", "effect", "footstep", "door", "rain", "wind", "explosion"]
is_sfx = any(k in prompt.lower() for k in sfx_keywords)
model_key = "audiogen" if is_sfx else "musicgen"
if model_key not in self.models:
if model_key == "audiogen":
self.models[model_key] = AudioGen.get_pretrained("facebook/audiogen-medium")
else:
self.models[model_key] = MusicGen.get_pretrained("facebook/musicgen-medium")
return self.models[model_key]
def generate(self, prompt: str, duration: float = 10.0):
cache_key = hashlib.md5(f"{prompt}:{duration}".encode()).hexdigest()
if cache_key in self.cache:
return self.cache[cache_key]
model = self._get_model(prompt)
model.set_generation_params(duration=duration, temperature=1.0)
wav = model.generate([prompt])[0]
buffer = io.BytesIO()
sr = 32000 if "musicgen" in str(type(model)) else 16000
torchaudio.save(buffer, wav.cpu(), sample_rate=sr, format="wav")
b64 = base64.b64encode(buffer.read()).decode()
self.cache[cache_key] = b64
return b64
router = AudioRouter()
@app.post("/generate")
async def generate_audio(prompt: str, duration: float = 10.0):
if duration > 30.0:
raise HTTPException(400, "Duration must be <= 30 seconds")
audio_b64 = router.generate(prompt, duration)
return {"audio": audio_b64, "format": "wav"}Project 2: Automated podcast intro generator.
Build a pipeline that takes a podcast title, description, and desired mood, then generates a complete intro sequence: a 15-second music bed, a 3-second transition riser, and a 2-second stinger. Combine the segments with crossfades using torchaudio’s signal processing. The system should generate multiple variations and select the best one based on a simple heuristic (peak amplitude distribution, spectral flatness).
import torch
import torchaudio
import torchaudio.functional as F
from audiocraft.models import MusicGen, AudioGen
class PodcastIntroGenerator:
def __init__(self):
self.music = MusicGen.get_pretrained("facebook/musicgen-medium")
self.sfx = AudioGen.get_pretrained("facebook/audiogen-medium")
def generate_intro(self, title: str, mood: str):
# Generate segments
self.music.set_generation_params(duration=15.0, temperature=0.8)
bed = self.music.generate([f"{mood} podcast intro music, {title}"])[0]
self.sfx.set_generation_params(duration=2.0, temperature=0.5)
riser = self.sfx.generate(["whoosh sound effect rising in pitch"])[0]
self.sfx.set_generation_params(duration=1.5, temperature=0.5)
stinger = self.sfx.generate(["short impactful hit or thud"])[0]
# Crossfade: bed fades out over last 2 seconds, riser fades in
fade_len = int(2.0 * 32000)
fade_curve = torch.linspace(0, 1, fade_len)
bed[:, -fade_len:] *= (1 - fade_curve)
riser[:, :fade_len] *= fade_curve
# Concatenate: bed (15s) + riser (2s) + stinger (1.5s)
# Pad shorter tensors to match channels
combined = torch.cat([bed, riser, stinger], dim=1)
return combinedProject 3: Audio dataset augmenter for ML training.
Build a data augmentation pipeline that uses AudioGen to generate synthetic sound effects for training audio classification models. Given a list of sound classes and a target count per class, generate variations with different temperatures and durations to create a diverse synthetic dataset. Filter generated samples using a pre-trained audio classifier (e.g., CLAP) to keep only samples that match the target class with high confidence.
import os
import torch
import torchaudio
from audiocraft.models import AudioGen
from transformers import ClapModel, ClapProcessor
class AudioDatasetAugmenter:
def __init__(self):
self.gen = AudioGen.get_pretrained("facebook/audiogen-medium")
self.validator = ClapModel.from_pretrained("laion/clap-htsat-fused")
self.processor = ClapProcessor.from_pretrained("laion/clap-htsat-fused")
def generate_class_samples(self, class_name: str, count: int, output_dir: str):
os.makedirs(f"{output_dir}/{class_name}", exist_ok=True)
generated = 0
attempts = 0
while generated < count and attempts < count * 5:
temperature = 0.5 + (attempts % 10) * 0.1 # Vary temperature
self.gen.set_generation_params(duration=2.0, temperature=temperature)
wav = self.gen.generate([class_name])[0]
# Validate with CLAP
inputs = self.processor(
text=[class_name],
audios=wav.squeeze(0).numpy(),
sampling_rate=16000,
return_tensors="pt",
)
outputs = self.validator(**inputs)
similarity = outputs.logits_per_audio.item()
if similarity > 0.3: # Confidence threshold
torchaudio.save(
f"{output_dir}/{class_name}/{generated:04d}.wav",
wav.cpu(), 16000,
)
generated += 1
attempts += 1
return generatedProblems Solved Efficiently
| Problem | Without AudioCraft | With AudioCraft | Improvement |
|---|---|---|---|
| Generate music from text | MusicLM (closed, 3-stage cascade) or Jukebox (16 GB VRAM, 8x real-time) | MusicGen.get_pretrained("small").generate(["prompt"]) |
10x faster, 4x less VRAM |
| Generate sound effects | Record, license, or use limited sample libraries | AudioGen.get_pretrained("medium").generate(["description"]) |
Instant, unlimited variety |
| Control melody | Impossible with text-only models | generate_with_chroma() with hummed or recorded melody |
Full melodic control |
| Generate stereo audio | Post-process mono output or use separate upmixer | musicgen-stereo-* models |
Native stereo, no post-processing |
| Fast generation | Autoregressive only (1x speed) | MAGNeT (3-5x faster, non-autoregressive) | 3-5x speedup |
| Condition on chords/drums | Not possible with text-only models | JASCO with symbolic conditioning | Bar-level temporal control |
| Watermark generated audio | Third-party tools, fragile | AudioSeal (built-in, robust) | Integrated, no extra dependencies |
| Fine-tune on custom data | Custom training loop, no reference | dora run solver=musicgen/... continue_from=... |
Reproducible pipeline |
| Deploy to production | Custom serving infrastructure | FastAPI + async inference + S3 caching | Production-ready in hours |
| Batch generation | Sequential per-prompt | model.generate([p1, p2, p3]) |
Parallel batch processing |
Architectural Tradeoffs
| Gained | Sacrificed |
|---|---|
| Single-stage autoregressive design (no error cascade) | Maximum audio quality limited by EnCodec compression artifacts |
| 50 autoregressive steps per second (4x fewer than naive) | Codebook interleaving adds ~50ms latency per step |
| MIT license — free for commercial use | No pretrained models for non-English text conditioning |
| Unified API across MusicGen, AudioGen, MAGNeT, JASCO | Model-specific quirks leak through (different sample rates, channel counts) |
| EnCodec compression enables language model approach | EnCodec artifacts (pre-echo, tonal noise) at low bitrates |
| Melody conditioning via chromagram | Chromagram is pitch-class only — no rhythm or timbre from reference |
| Multi-GPU training with Dora/FSDP | Training from scratch requires 8+ A100 GPUs for large models |
| AudioSeal watermarking built in | Watermark detection accuracy drops on heavily processed audio |
| MAGNeT non-autoregressive for speed | MAGNeT has slightly less coherent long-range structure than MusicGen |
| JASCO symbolic conditioning (chords, drums, melody) | JASCO requires MIDI or audio-to-MIDI preprocessing for conditioning |
The real trade-off: AudioCraft optimizes for accessibility and research velocity over absolute audio quality. The EnCodec tokenizer introduces a perceptual ceiling — no matter how good the language model, the decoded audio cannot exceed the codec’s reconstruction quality. For production music, a 64 kbps MP3 is roughly equivalent to EnCodec at 32 kHz with 4 codebooks. If your use case demands CD-quality audio (44.1 kHz, 16-bit, 1411 kbps), AudioCraft is not the right tool. But if you need rapid prototyping, unlimited variation, and text-controllable generation on consumer GPUs, the quality trade-off is worth it. The 2024 addition of stereo models and MAGNeT’s non-autoregressive decoding narrowed this gap significantly — stereo at 32 kHz with 8 codebooks approaches FM radio quality.
Course-Style Deep Dive
Under the Hood: The EnCodec Neural Codec
EnCodec is an autoencoder trained with a three-part loss function:
-
Reconstruction loss: L1 distance between input and reconstructed waveforms, computed in both time and frequency domains (STFT with multiple FFT sizes: 2048, 512, 256).
-
Perceptual loss: A multi-scale STFT discriminator (MS-STFT) that classifies real vs. reconstructed audio at multiple time-frequency resolutions. The discriminator has 5 sub-discriminators, each operating on a different STFT configuration. The generator (decoder) is trained to fool the discriminator, while the discriminator is trained to detect artifacts.
-
Quantization loss: The RVQ bottleneck uses a straight-through estimator for gradient flow through the discrete quantization step. Each RVQ layer quantizes the residual from the previous layer. The codebook size is 1024 entries per layer, with 4 layers for 32 kHz mono and 8 layers for stereo.
The encoder is a CNN with 4 convolutional blocks (stride 2 each), reducing the temporal dimension by 16x. At 32 kHz input, the encoder produces a latent sequence at 2 kHz (32,000 / 16 = 2,000 Hz). The RVQ then quantizes each latent vector into 4 codebook indices, producing 4 parallel token streams at 50 Hz (2,000 / 40 = 50 Hz after the quantization stride). This is the 50 tokens per second per codebook that MusicGen operates on.
# Simplified EnCodec forward pass
class EnCodec(nn.Module):
def __init__(self, n_q: int = 4, codebook_size: int = 1024):
super().__init__()
self.encoder = Encoder(channels=1, strides=[2, 4, 4, 2])
self.quantizer = ResidualVectorQuantizer(
n_q=n_q, # Number of codebooks
codebook_size=codebook_size, # 1024 entries per codebook
)
self.decoder = Decoder(channels=1, strides=[2, 4, 4, 2])
def forward(self, audio: torch.Tensor):
# audio: (B, 1, T) at 32 kHz
encoded = self.encoder(audio) # (B, D, T/16)
codes, _ = self.quantizer(encoded) # (B, n_q, T/640)
decoded = self.decoder(self.quantizer.decode(codes))
return decoded, codesUnder the Hood: MusicGen’s Codebook Interleaving
The core innovation in MusicGen is how it handles multiple codebook streams. With 4 codebooks at 50 Hz, a naive approach would flatten all codebooks into a single sequence of 200 tokens per second and predict them autoregressively. This is slow — 200 sequential decoding steps per second of audio.
MusicGen’s solution: introduce a delay pattern between codebooks. At step t, the model predicts codebook 0. At step t+1, it predicts codebook 1 (conditioned on codebook 0 at step t). At step t+2, it predicts codebook 2 (conditioned on codebooks 0-1 at steps t-1 to t). At step t+3, it predicts codebook 3. At step t+4, it predicts codebook 0 again for the next time step.
This means only 50 autoregressive steps per second. The remaining 150 tokens are predicted in parallel from the delayed context. The pattern is implemented as a shift in the token positions:
# Codebook interleaving pattern (delays=[0, 1, 2, 3])
# t=0: [c0_0, c1_0, c2_0, c3_0] → predict c0_0
# t=1: [c0_1, c1_1, c2_1, c3_1] → predict c1_0 (conditioned on c0_0)
# t=2: [c0_2, c1_2, c2_2, c3_2] → predict c2_0 (conditioned on c0_0, c1_0)
# t=3: [c0_3, c1_3, c2_3, c3_3] → predict c3_0 (conditioned on c0_0, c1_0, c2_0)
# t=4: [c0_4, c1_4, c2_4, c3_4] → predict c0_1 (conditioned on all previous)The Transformer LM uses causal attention with a special masking pattern that respects this delay structure. Each position can only attend to positions that are available given the delay pattern. This is implemented via a custom attention mask in the Transformer forward pass.
Under the Hood: Text Conditioning
MusicGen uses a frozen T5 encoder (FLAN-T5 for the large model) to encode text prompts. The text embeddings are projected to the Transformer’s hidden dimension and injected via cross-attention in every decoder layer. The cross-attention follows the standard Transformer decoder pattern: queries come from the audio token sequence, keys and values come from the text embeddings.
For classifier-free guidance (CFG), the model is trained with a 10% dropout rate on text conditioning — 10% of training steps use a null text embedding. At inference, the model runs twice: once with the text prompt and once with the null embedding. The final prediction is a linear interpolation:
noise_pred = noise_pred_null + guidance_scale * (noise_pred_text - noise_pred_null)A guidance scale of 3.0 means the text-conditioned prediction is weighted 3x more than the unconditional prediction. Higher values produce outputs that follow the prompt more literally but can sound unnatural.
Advanced Pattern: Fine-Tuning on Custom Data
# Step 1: Prepare your dataset
# Structure:
# my_dataset/
# audio/
# 0001.wav
# 0002.wav
# metadata.json
#
# metadata.json format:
# [
# {"title": "track1", "description": "upbeat jazz with brass", "path": "audio/0001.wav"},
# {"title": "track2", "description": "dark ambient drone", "path": "audio/0002.wav"},
# ]
# Step 2: Fine-tune using Dora (AudioCraft's experiment manager)
# dora run solver=musicgen/musicgen_base_32khz \
# model/lm/model_scale=medium \
# continue_from=//pretrained/facebook/musicgen-medium \
# conditioner=text2music \
# dataset.path=/path/to/my_dataset \
# optimizer.lr=1e-5 \
# optimizer.adamw.weight_decay=0.01 \
# training.batch_size=4 \
# training.epochs=50
# Step 3: Export the fine-tuned model
from audiocraft.utils import export
from audiocraft import train
xp = train.main.get_xp_from_sig('SIG_OF_LM')
export.export_lm(xp.folder / 'checkpoint.th', '/checkpoints/finetuned_lm/state_dict.bin')
export.export_pretrained_compression_model(
'facebook/encodec_32khz',
'/checkpoints/finetuned_lm/compression_state_dict.bin'
)
# Step 4: Load and use the fine-tuned model
finetuned = MusicGen.get_pretrained('/checkpoints/finetuned_lm/')
finetuned.set_generation_params(duration=10.0)
wav = finetuned.generate(["custom genre from fine-tuned dataset"])Advanced Pattern: Production Serving with Async Inference
import asyncio
import torch
import torchaudio
from audiocraft.models import MusicGen
from concurrent.futures import ThreadPoolExecutor
import boto3
import os
class ProductionMusicGenService:
def __init__(self, model_name="facebook/musicgen-medium", max_workers=2):
self.model = MusicGen.get_pretrained(model_name)
self.model.lm = torch.compile(self.model.lm, mode="reduce-overhead")
self.executor = ThreadPoolExecutor(max_workers=max_workers)
self.s3 = boto3.client("s3")
self.bucket = os.environ["AUDIO_BUCKET"]
async def generate(self, prompt: str, duration: float = 10.0) -> str:
"""Generate audio and return S3 key."""
loop = asyncio.get_event_loop()
wav = await loop.run_in_executor(
self.executor,
self._sync_generate,
prompt, duration,
)
# Upload to S3
key = f"generated/{hash(prompt)}-{int(time.time())}.wav"
buffer = io.BytesIO()
torchaudio.save(buffer, wav.cpu(), sample_rate=32000, format="wav")
buffer.seek(0)
self.s3.upload_fileobj(buffer, self.bucket, key)
return key
def _sync_generate(self, prompt: str, duration: float) -> torch.Tensor:
with torch.no_grad():
self.model.set_generation_params(duration=duration, temperature=1.0)
wav = self.model.generate([prompt], progress=False)
return wav[0]
def generate_batch(self, prompts: list, duration: float = 10.0) -> list:
"""Synchronous batch generation for offline processing."""
self.model.set_generation_params(duration=duration, temperature=1.0)
with torch.no_grad():
wavs = self.model.generate(prompts, progress=True)
return [w.cpu() for w in wavs]Advanced Pattern: Audio Watermarking with AudioSeal
from audiocraft.models import AudioSeal
import torchaudio
# Load the watermarking model
watermarker = AudioSeal.get_pretrained("facebook/audioseal")
# Embed a watermark in generated audio
wav = torch.randn(1, 1, 32000 * 10) # 10 seconds of audio
watermarked = watermarker.embed(wav, sample_rate=32000)
# Detect watermark
result = watermarker.detect(watermarked, sample_rate=32000)
# result contains detection scores per frame
# The watermark survives:
# - MP3 compression at 128 kbps
# - Resampling to 44.1 kHz and back
# - Volume changes (+/- 6 dB)
# - Low-pass filtering (8 kHz cutoff)
# - Background noise addition (20 dB SNR)Production lesson: AudioSeal’s detection is frame-based, not clip-based. A 10-second clip produces a detection score per ~0.5 second frame. For a binary “is this watermarked?” decision, average the frame scores and apply a threshold. The threshold depends on your false-positive tolerance — we use 0.5 for general detection and 0.8 for forensic evidence. Test on your specific audio pipeline (codec, resampling, compression) before deploying, as heavy processing can shift the score distribution.
The Results
| Metric | Before AudioCraft | With AudioCraft | Improvement |
|---|---|---|---|
| Time to generate 10s of music | 30-60 sec (MusicLM cascade) | 5-15 sec (MusicGen single-stage) | 3-6x faster |
| VRAM required for music generation | 16+ GB (Jukebox) | 2-8 GB (MusicGen small-large) | 2-8x reduction |
| Lines of inference code | 200-500 (custom pipeline) | 3-5 lines | 40-100x reduction |
| Models available in one library | 1-2 per framework | 8 (MusicGen, AudioGen, MAGNeT, JASCO, etc.) | 4-8x increase |
| Text-to-sound effects | Manual recording or licensing | AudioGen.generate(["description"]) |
Instant, unlimited |
| Melody-controlled generation | Not available in open source | generate_with_chroma() |
New capability |
| Stereo generation | Post-process mono | Native stereo models | 1-step stereo |
| Non-autoregressive speed | Not available | MAGNeT: 3-5x faster than AR | 3-5x speedup |
| Symbolic conditioning | Not available | JASCO: chords, drums, melody | New capability |
| Audio watermarking | Third-party tools | AudioSeal (built-in) | Integrated |
| Fine-tuning setup time | 1-2 weeks (custom training loop) | 1-2 hours (Dora experiment manager) | 10x faster |
| Commercial use | Varies by model (MusicLM: no) | MIT license | Unrestricted |
What to Watch Out For
Advice for Getting Started
-
Always match the sample rate. MusicGen outputs at 32 kHz. AudioGen outputs at 16 kHz. If you mix them up, your audio will play at the wrong speed. Save MusicGen output with
sample_rate=32000and AudioGen withsample_rate=16000. -
Duration is limited to 30 seconds. The positional embeddings in the Transformer are trained for a maximum sequence length. You cannot generate audio longer than 30 seconds in a single call. For longer content, generate segments and concatenate with crossfades.
-
Guidance scale is not free. Higher guidance_scale values (4+) produce more literal prompt following but can introduce metallic artifacts and reduce audio diversity. The sweet spot is 2.0-3.0. If your output sounds unnatural, lower the guidance scale before changing temperature.
-
Temperature controls randomness, not quality. Low temperature (0.5-0.7) produces safer, more repetitive outputs. High temperature (1.2-1.5) produces more creative but potentially noisier outputs. For sound effects, use lower temperature (0.5-0.7). For music, use moderate temperature (0.8-1.0).
-
Batch generation shares the same random seed by default. If you call
model.generate([p1, p2]), both prompts use the same internal random state unless you set different seeds. For diverse outputs, settorch.manual_seed()to different values between calls or generate one at a time.
“We spent a week debugging why our MusicGen outputs sounded like they were underwater. Turns out we were saving at 16 kHz (AudioGen’s rate) instead of 32 kHz. The audio played back at half speed, making everything sound like a slowed-down tape. The torchaudio.save() call silently accepted the wrong sample rate — no warning, no error. Always double-check your sample rate parameter.” — ML Engineer at a music production startup
“The 30-second generation limit was a hard blocker for our podcast background music use case. We tried generating 30-second segments and concatenating them, but the transitions were jarring — the model doesn’t maintain key or tempo across generations. Our fix was to generate a single 30-second segment and loop it with a 2-second crossfade. For longer content, we generate multiple 30-second segments from the same prompt and crossfade between them. It’s not seamless, but it works for background music where the listener isn’t paying close attention.” — Audio Engineer at a podcast production company
“We deployed MusicGen in production and discovered that the model is surprisingly sensitive to punctuation in prompts. ‘Upbeat jazz, with brass and piano’ and ‘Upbeat jazz with brass and piano’ produce noticeably different outputs — the comma changes how the T5 encoder parses the phrase. We standardized on comma-separated lists of descriptors and saw much more consistent results. Also, trailing spaces in prompts cause subtle quality degradation. Always strip your prompts.” — MLOps Engineer at a generative media API
“Fine-tuning MusicGen on our proprietary dataset was harder than expected. The Dora experiment manager is powerful but poorly documented — the configuration system uses Hydra with deep nesting, and a single typo in the YAML path silently falls back to defaults. We lost a week of training time because we misspelled ‘conditioner’ as ‘conditioner’ (extra ‘i’) and the model trained without text conditioning. Always validate your config by generating a few samples from the fine-tuned checkpoint within the first 100 steps.” — Research Scientist at an audio AI lab
Next in the Open-Source AI Tools Mastery series: GPT-SoVITS
Written by Nivant Labs Team
Engineer at Nivant Labs