·15 min read

Text Generation Inference: Hugging Face's production-grade LLM inference server (Apache 2.0, 9k stars)

Deploying LLM inference at scale with continuous batching, tensor parallelism, and flash attention — Hugging Face's production-grade inference server.

#tgi#huggingface#inference-server#continuous-batching#ai-infra

The Problem

Every LLM deployment faces the same fundamental tension: latency versus throughput. Serve one request at a time and you get fast time-to-first-token (TTFT) but abysmal throughput. Batch requests together and throughput climbs but every request waits for the slowest one to finish. The naive approach — static batching — forces a choice between these two extremes, and neither is acceptable in production.

The deeper problem is that LLM inference is not a single operation. It has two phases with radically different compute profiles:

  • Prefill (compute-bound): The first forward pass processes all input tokens in parallel. This is a dense matrix multiplication that saturates GPU compute units.
  • Decode (memory-bound): Each subsequent forward pass generates one token at a time. This is a memory-bandwidth-bound operation that leaves compute units idle.

A production inference server must handle both phases simultaneously, for hundreds of concurrent requests, across multiple GPUs, while managing a KV cache that grows by hundreds of kilobytes per token per sequence. The naive approach of “load the model, run inference” fails on every dimension.

Dimension Naive Approach What Production Requires
Batching Static (fixed batch at request time) Dynamic (add/remove requests per iteration)
KV cache Pre-allocated contiguous buffers (60-80% waste) Paged allocation with <4% fragmentation
Attention Full NxN matrix materialized in HBM Tiled on-chip computation (Flash Attention)
Multi-GPU Manual model sharding Automatic tensor parallelism
Request scheduling FIFO queue Iteration-level scheduling with preemption
Memory management Fixed per-request budget Dynamic pool with OOM prevention
Quantization None (full FP16) INT4, INT8, FP8, AWQ, GPTQ, Marlin
Observability stdout logs Prometheus metrics, OpenTelemetry traces

Why this matters: The difference between a toy inference server and a production-grade one is not model support — any server can load a model. The difference is how it manages the KV cache, schedules requests, and balances prefill vs decode. Get these wrong and your GPU utilization drops below 50%, your p99 latency spikes to minutes, and your hardware costs double for the same throughput. TGI was built to solve exactly these problems, and its architecture has influenced every inference server that followed.

The Investigation

Hugging Face started investigating production LLM inference in early 2023, when the first wave of open-source LLMs (Llama, Falcon, MPT) made self-hosting viable. The team identified three root causes of poor inference performance:

Finding 1: Static batching wastes GPU compute.

With static batching, the client decides the batch size at request time. If you batch 4 requests, you wait until all 4 arrive before starting inference. If you batch 1, you get low latency but terrible throughput. The GPU spends most of its time idle, waiting for requests to accumulate.

TGI’s investigation found that iteration-level scheduling (from the Orca paper, OSDI 2022) eliminates this tradeoff. Instead of fixing the batch at request time, the server re-evaluates the batch after every forward pass. New requests join immediately. Completed requests drop out immediately. The batch size fluctuates dynamically, keeping the GPU saturated without forcing any request to wait for unrelated work.

The impact is dramatic: continuous batching delivers 3-23x higher throughput than static batching on the same hardware, with no increase in per-request latency.

Finding 2: KV cache fragmentation wastes 60-80% of GPU memory.

Every active request accumulates a KV cache — one key-value pair per layer per head per token. For Llama 3.1 70B at BF16, that’s 320 KB per token. An 8K context costs 2.5 GB. A 128K context costs 40 GB — nearly filling an H100’s 80 GB.

The naive approach pre-allocates contiguous buffers sized for the maximum context per request. If you have 100 concurrent requests with an 8K max context, you reserve 250 GB of KV cache — even though most requests will generate far fewer tokens. The result is 60-80% memory waste, which directly limits batch size and throughput.

TGI adopted PagedAttention (Kwon et al., SOSP 2023), which models KV cache after OS virtual memory. Logical blocks map to physical blocks through a per-sequence block table. Blocks are allocated on demand from a shared pool. Memory fragmentation drops to under 4% — only the tail of the last block per sequence is wasted.

Finding 3: Standard attention materializes the full NxN matrix in HBM.

Standard attention computes S = Q x K^T, applies softmax, then multiplies by V. The intermediate S matrix has shape (seq_len x seq_len), which for a 128K-token sequence is 16 billion elements — 32 GB at FP16. This matrix is written to HBM, read back, and discarded. The memory bandwidth cost dominates the computation.

TGI integrated Flash Attention (Dao et al., NeurIPS 2022), which tiles Q, K, V into blocks that fit in on-chip SRAM and computes softmax online in a single fused kernel. The full NxN matrix is never materialized. This enables 128K+ context lengths on hardware that would OOM with standard attention, and delivers up to 230 TFLOPs/s on A100 FP16 attention kernels (~72% model FLOPs utilization).

The key insight: These three findings are not independent. Continuous batching increases the number of concurrent requests, which increases KV cache pressure, which makes PagedAttention necessary. PagedAttention creates non-contiguous memory access patterns, which makes Flash Attention’s tiled approach even more important. The three optimizations form a virtuous cycle — each one enables the next.

The Solution

TGI (Text Generation Inference) is an Apache 2.0-licensed inference server with 9,000+ GitHub stars, built by Hugging Face. It serves any Hugging Face model with a single docker run command, handling continuous batching, tensor parallelism, quantization, and streaming out of the box.

┌─────────────────────────────────────────────────────────────────────┐
│                        TGI Architecture                              │
│                                                                      │
│  ┌─────────────┐   HTTP/SSE    ┌──────────────┐   gRPC   ┌──────┐ │
│  │   Client     │◄─────────────►│   Router     │◄────────►│Model │ │
│  │  (curl, SDK) │               │  (Rust/axum)  │          │Server│ │
│  └─────────────┘               │              │          │(Py)  │ │
│                                 │  ┌─────────┐ │          └──────┘ │
│                                 │  │Scheduler │ │              │    │
│                                 │  │(cont.    │ │         ┌──────┐ │
│                                 │  │ batching)│ │         │Model │ │
│                                 │  └─────────┘ │         │Server│ │
│                                 │  ┌─────────┐ │         │(Py)  │ │
│                                 │  │Queue    │ │         └──────┘ │
│                                 │  │Manager  │ │         (sharded) │
│                                 │  └─────────┘ │                  │
│                                 └──────────────┘                  │
│                                                                      │
│  ┌──────────────────────────────────────────────────────────────┐   │
│  │  Model Server Internals (per GPU shard)                      │   │
│  │                                                              │   │
│  │  ┌──────────┐  ┌──────────────┐  ┌──────────────────────┐   │   │
│  │  │Flash     │  │PagedAttention│  │KV Cache Block Table  │   │   │
│  │  │Attention │  │Kernel        │  │(per-sequence mapping)│   │   │
│  │  └──────────┘  └──────────────┘  └──────────────────────┘   │   │
│  │  ┌──────────┐  ┌──────────────┐  ┌──────────────────────┐   │   │
│  │  │CUDA      │  │Quant Kernels │  │Prefix Cache         │   │   │
│  │  │Graphs    │  │(AWQ/GPTQ/FP8)│  │(shared KV blocks)   │   │   │
│  │  └──────────┘  └──────────────┘  └──────────────────────┘   │   │
│  └──────────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────────┘

The architecture has three components:

  1. Router (Rust, axum + tokio): Receives HTTP requests, manages the continuous batching queue, handles backpressure, and sends gRPC calls to model servers. The Rust implementation was a v3 upgrade that reduced tail latency through zero-cost abstractions and async I/O.

  2. Launcher: A helper process that launches one or more model servers (for tensor-parallel sharding) and the router with compatible arguments. Handles NCCL initialization and shared-memory setup.

  3. Model Server (Python + CUDA): Loads the model, manages the KV cache, and executes prefill/decode forward passes. Supports sharding across multiple GPUs via NCCL. The router and model server can be on different machines.

Quick Start

# Pull the Docker image and run TGI
model=HuggingFaceH4/zephyr-7b-beta
volume=$PWD/data

docker run --gpus all --shm-size 1g \
  -p 8080:80 \
  -v $volume:/data \
  ghcr.io/huggingface/text-generation-inference:3.3.5 \
  --model-id $model
# Client code using the OpenAI-compatible endpoint
import openai

client = openai.OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="not-needed"
)

response = client.chat.completions.create(
    model="tgi",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain continuous batching in one paragraph."}
    ],
    max_tokens=200,
    temperature=0.7,
    stream=True
)

for chunk in response:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

Production Deployment

model=meta-llama/Meta-Llama-3.1-8B-Instruct
volume=$PWD/data

docker run --gpus all --shm-size 1g \
  -p 8080:80 \
  -p 9000:9000 \
  -v $volume:/data \
  -e HF_TOKEN=$HF_TOKEN \
  ghcr.io/huggingface/text-generation-inference:3.3.5 \
  --model-id $model \
  --num-shard 4 \
  --dtype bfloat16 \
  --max-total-tokens 4096 \
  --max-input-tokens 3072 \
  --max-batch-total-tokens 32000 \
  --max-concurrent-requests 256 \
  --waiting-served-ratio 0.3 \
  --max-waiting-tokens 20 \
  --cuda-memory-fraction 0.9 \
  --json-output \
  --otlp-endpoint http://otel-collector:4318 \
  --cors-allow-origin "*" \
  --api-key "sk-tgi-prod-01"

How to Use Effectively

TGI’s power comes from its configuration knobs. The defaults work for development, but production requires tuning five critical parameters:

1. Set --max-total-tokens and --max-input-tokens explicitly.

These define the per-request memory budget. max-total-tokens is input + generated tokens combined. max-input-tokens limits the prompt length. TGI auto-tunes these if unset, but auto-tuning is conservative — it reserves memory for the worst case. For production, set them based on your actual workload.

# For a chatbot with 2K system prompt + 1K user input + 1K generation
--max-input-tokens 3072
--max-total-tokens 4096

2. Tune --max-batch-total-tokens for your GPU.

This is the most critical performance parameter. It defines the total token budget across all requests in a batch. Set it too low and you leave GPU compute idle. Set it too high and you OOM.

The formula: max_batch_total_tokens = (available_vram * 0.95) / bytes_per_token

For Llama 3.1 70B on an H100 (80 GB):

  • Model weights: ~140 GB (BF16) across 8 GPUs = ~17.5 GB per GPU
  • KV cache per token: 320 KB
  • Available VRAM per GPU: ~60 GB (after model + overhead)
  • Max tokens: (60 GB * 0.95) / 320 KB = ~178,000 tokens
# Conservative starting point for Llama 3.1 70B on 8xH100
--max-batch-total-tokens 128000

3. Balance prefill vs decode with --waiting-served-ratio and --max-waiting-tokens.

The router can pause decode to run prefill for new requests. These parameters control how aggressively it does so:

  • --waiting-served-ratio 0.3: Start prefill when waiting requests reach 30% of served requests.
  • --max-waiting-tokens 20: Force prefill after 20 decode tokens, regardless of ratio.

Lower waiting-served-ratio reduces TTFT (good for interactive apps) but increases decode jitter. Higher values maximize throughput at the cost of first-token latency.

4. Use quantization for memory-constrained hardware.

# Best latency: AWQ (requires AWQ-quantized model)
--quantize awq

# Drop-in memory reduction: EETQ (8-bit, any model)
--quantize eetq

# Highest throughput on H100+: FP8
--quantize fp8

# Any model, any hardware: bitsandbytes NF4
--quantize bitsandbytes-nf4

5. Enable observability from day one.

--json-output \
--otlp-endpoint http://otel-collector:4318 \
--prometheus-port 9000

Key metrics to monitor:

  • tgi_request_count: Request throughput
  • tgi_request_duration_ms: End-to-end latency (p50, p95, p99)
  • tgi_request_inference_duration_ms: Time spent in model forward pass
  • tgi_batch_next_tokens: Tokens generated per batch iteration
  • tgi_queue_size: Number of requests waiting in queue
  • tgi_batch_size: Current batch size (should be >1 under load)

Use Cases

1. Production Chat API. Deploy Llama 3.1 70B behind a TGI server with continuous batching to serve thousands of concurrent chat users. The Rust router handles 10,000+ RPS with sub-millisecond overhead, and the model server dynamically batches requests to maximize GPU utilization. Use --waiting-served-ratio 0.3 to keep TTFT under 500ms for interactive chat while maintaining 80%+ GPU utilization.

2. Batch Document Processing. Process thousands of documents through a summarization pipeline. Set --max-concurrent-requests 512 and --max-batch-total-tokens to fill your GPU’s memory. Each request processes independently, but TGI’s continuous batching keeps the GPU saturated. For a 7B model on a single A100, expect 4,000-6,000 tokens/second throughput on batch-64 workloads.

3. RAG Pipeline with Prefix Caching. Deploy a retrieval-augmented generation system where every request shares the same system prompt. TGI’s prefix caching (v3+) keeps the shared KV cache in memory, so subsequent requests skip recomputing the system prompt. On 200K-token prompts, TGI v3 achieves 2s TTFT vs 27.5s for vLLM — a 13x improvement. Enable with --enable-prefill-logprobs only when needed (it costs VRAM).

4. Multi-Model Serving with LoRA Adapters. Serve multiple fine-tuned variants of the same base model using LoRA adapters. TGI supports preloading adapters with --lora-adapters and hot-swapping at runtime. Each adapter adds minimal memory overhead (megabytes vs gigabytes for a full model), enabling dozens of fine-tuned variants from a single deployment.

5. Edge Deployment with Quantization. Deploy a 7B model on a single L4 GPU (24 GB) using INT4 quantization. TGI’s --quantize bitsandbytes-nf4 reduces model memory from 14 GB to 4 GB, leaving 20 GB for KV cache. With --max-batch-total-tokens tuned to 30,000, a single L4 handles 30K tokens on Llama 3.1 8B — 3x more than vLLM’s ~10K on the same hardware.

Cheat Sheet

Task Command / Config Notes
Basic run docker run --gpus all --shm-size 1g -p 8080:80 -v $PWD/data:/data ghcr.io/huggingface/text-generation-inference:3.3.5 --model-id $MODEL Requires NVIDIA Container Toolkit, CUDA 12.2+
Private model Add -e HF_TOKEN=$HF_TOKEN Use a Hugging Face read token
Multi-GPU --num-shard 4 with --gpus all Auto-detects GPUs; use CUDA_VISIBLE_DEVICES to select
Quantization --quantize awq / --quantize fp8 / --quantize bitsandbytes-nf4 AWQ requires pre-quantized model; FP8 needs H100+
Memory budget --max-total-tokens 4096 --max-input-tokens 3072 Per-request; set based on workload, not hardware
Batch tuning --max-batch-total-tokens 32000 --waiting-served-ratio 0.3 --max-waiting-tokens 20 Most critical for throughput
Observability --json-output --prometheus-port 9000 --otlp-endpoint http://otel:4318 Monitor tgi_queue_size and tgi_batch_size
Streaming Client: stream=True in OpenAI SDK Server-sent events (SSE) over HTTP
Speculative decoding --speculate 5 Generates draft tokens; 1.5-2x speedup on decode
CORS --cors-allow-origin "*" Required for browser-based clients
API key --api-key "sk-..." Adds Bearer token auth
CUDA graphs --cuda-graphs 1,2,4,8,16,32 Reduces kernel launch overhead; set 0 to disable
Graceful shutdown --graceful-termination-timeout 90 Drains in-flight requests before exit
Shared memory --shm-size 1g or Kubernetes emptyDir: medium: Memory Required for NCCL; set NCCL_SHM_DISABLE=1 to skip
Log format --json-output Structured JSON for log aggregators
Max concurrent --max-concurrent-requests 128 Backpressure; prevents OOM from request flood
Max client batch --max-client-batch-size 4 Max inputs per single client request
RoPE scaling --rope-scaling linear --rope-factor 2.0 For models with extended context (e.g., 128K)
Disable custom kernels --disable-custom-kernels Fallback for non-A100 GPUs or compatibility issues

Vibe Coding Projects

1. Build a multi-tenant LLM API gateway. Deploy TGI behind a reverse proxy (nginx, Envoy) that handles tenant isolation, rate limiting, and API key management. Each tenant gets a virtual queue with configurable concurrency limits. Use TGI’s --api-key for authentication and --cors-allow-origin for web clients. Add a Prometheus scraper and Grafana dashboard to visualize per-tenant token usage, latency distributions, and GPU utilization. The Rust router handles 10K+ RPS, so the bottleneck is your model, not the server.

2. Create a batch inference pipeline for document extraction. Build a pipeline that reads documents from S3, chunks them, sends them to TGI for extraction (entity recognition, summarization, classification), and writes results back to a database. Use TGI’s high-throughput batch mode with --max-concurrent-requests 512 and --max-batch-total-tokens tuned to fill your GPU. Implement retry logic with exponential backoff for transient failures. Monitor tgi_request_duration_ms to detect when the pipeline is GPU-bound vs I/O-bound.

3. Build a RAG system with shared prefix caching. Deploy TGI v3+ with a retrieval-augmented generation pipeline where every request shares a system prompt and retrieved context. TGI’s prefix caching reuses KV cache across requests with the same prefix, eliminating redundant computation. Use --enable-prefill-logprobs only when you need token-level logprobs for the prompt (it costs VRAM). Benchmark TTFT with and without prefix caching — on long prompts (200K+ tokens), expect 10-13x improvement.

Problems Solved Efficiently

Problem TGI Solution Why It Works
Low GPU utilization under concurrent load Continuous batching (iteration-level scheduling) Re-evaluates batch after every forward pass; adds/removes requests dynamically
KV cache memory fragmentation (60-80% waste) PagedAttention with block table Allocates KV cache in fixed-size blocks on demand; <4% fragmentation
Long prompts OOM on consumer GPUs Flash Attention (tiled computation) Never materializes full NxN attention matrix; enables 128K+ contexts
High TTFT for long shared prefixes Prefix caching (v3+) Reuses KV cache across requests with same prefix; ~5-6 microsecond lookup
Multi-GPU deployment complexity Automatic tensor parallelism via NCCL Launcher handles shard initialization; single --num-shard flag
Model quantization complexity Built-in AWQ, GPTQ, Marlin, FP8, bitsandbytes Single --quantize flag; no manual calibration or conversion
Poor decode throughput on small models CUDA graphs for fixed batch sizes Records GPU operations as reusable graphs; eliminates kernel launch overhead
No observability into inference Prometheus metrics + OpenTelemetry 20+ built-in metrics; OTLP export for traces
Request queue management Rust-based router with backpressure Configurable --max-concurrent-requests; graceful queue drain on shutdown
Streaming response latency Server-sent events (SSE) over HTTP Tokens stream as they’re generated; no buffering until EOS
Speculative decoding complexity Built-in n-gram speculation --speculate N enables draft-token generation; 1.5-2x decode speedup
Model weight loading overhead Shared volume caching Weights cached in $volume/data; subsequent restarts skip download

Architectural Tradeoffs

Gained Sacrificed
Continuous batching for 3-23x throughput over static batching Higher implementation complexity than naive FIFO serving
PagedAttention for <4% KV cache fragmentation Non-contiguous memory access adds ~5-10% overhead to attention kernel
Flash Attention for 128K+ context support Requires custom CUDA kernels; no CPU fallback
Rust router for 10K+ RPS with sub-ms overhead Two-language codebase (Rust + Python) increases maintenance burden
Automatic tensor parallelism via NCCL NCCL initialization adds 5-30s startup time per shard
Built-in quantization for any model Quantized models have 1-3% accuracy degradation on complex tasks
Prefix caching for 13x TTFT improvement on long prompts Cache invalidation complexity; stale cache can produce incorrect results
Single Docker command deployment Limited customization of the inference pipeline (no custom kernels without forking)
Hugging Face Hub integration (auto-download, trust-remote-code) Tight coupling to Hugging Face ecosystem; non-HF models require workarounds

The key tradeoff: TGI optimizes for the common case — Hugging Face models, NVIDIA GPUs, standard transformer architectures. This is a deliberate choice that makes it the easiest inference server to deploy for the Hugging Face ecosystem. But it means TGI is less flexible than lower-level alternatives like TensorRT-LLM (which gives you full control over kernel selection) or vLLM (which has broader model support and higher throughput at high concurrency). TGI’s v3 release narrowed the gap significantly, but the architectural tradeoff remains: ease of use vs maximum performance.

Course-Style Deep Dive

Under the Hood: The Continuous Batching Algorithm

TGI’s router implements iteration-level scheduling, inspired by Orca (OSDI 2022). Here is the core algorithm:

# Simplified continuous batching loop
batch = []
token_budget = MAX_BATCH_TOTAL_TOKENS

# Warmup: estimate available token budget
warmup_batch = estimate_max_batch(model, gpu_memory)
MAX_BATCH_PREFILL_TOKENS = warmup_batch.prefill_tokens
MAX_BATCH_TOTAL_TOKENS = warmup_batch.total_tokens

# Main loop
while True:
    # Step 1: Add waiting requests to prefill batch
    prefill_batch = []
    prefill_tokens = 0
    for req in queue:
        if prefill_tokens + len(req.input_ids) > MAX_BATCH_PREFILL_TOKENS:
            break
        prefill_batch.append(req)
        prefill_tokens += len(req.input_ids)
        queue.remove(req)

    # Step 2: Run prefill (compute-bound, processes all tokens in parallel)
    if prefill_batch:
        prefill_outputs = model.prefill(prefill_batch)
        batch.extend(prefill_batch)

    # Step 3: Run decode (memory-bound, one token per sequence)
    decode_outputs = model.decode(batch)

    # Step 4: Remove completed requests, stream tokens
    for req in batch:
        token = decode_outputs[req.id]
        req.generated_tokens.append(token)
        req.stream_callback(token)
        if token == EOS or len(req.generated_tokens) >= req.max_new_tokens:
            batch.remove(req)
            req.stream_callback(DONE)

    # Step 5: Check if we should prefill waiting requests
    waiting_tokens = sum(len(req.input_ids) for req in queue)
    served_tokens = sum(len(req.input_ids) + len(req.generated_tokens) for req in batch)
    if waiting_tokens > served_tokens * WAITING_SERVED_RATIO:
        continue  # Go to Step 1 (interleave prefill)
    # else: continue decoding

The critical insight is that prefill and decode have different compute profiles. Prefill processes all input tokens in one forward pass (compute-bound, saturates GPU tensor cores). Decode processes one token per sequence (memory-bound, saturates HBM bandwidth). By interleaving them, TGI keeps both compute and memory units busy.

The MAX_BATCH_PREFILL_TOKENS and MAX_BATCH_TOTAL_TOKENS parameters differ because prefill is more expensive per token. A batch of 4 requests with 1K tokens each costs 4K tokens of prefill compute. The same 4 requests in decode cost 4 tokens per iteration. The decode budget can be much larger, allowing more requests to coexist in the decode phase.

Under the Hood: PagedAttention

PagedAttention models KV cache after OS virtual memory:

OS Concept PagedAttention Equivalent
Byte Token
Page Block (16 or 32 tokens)
Process Sequence
Page table Block table
Physical memory Shared KV pool

Each sequence has a logical KV cache that grows as tokens are generated. The block table maps logical block indices to physical block addresses in a shared pool. When a sequence generates a new token, it allocates a new physical block (if the current block is full) and appends the KV pair.

The attention kernel uses the block table to gather non-contiguous K/V tiles:

# Simplified PagedAttention forward pass
def paged_attention(query, block_table, kv_pool, block_size=16):
    """
    query: [num_heads, head_dim]
    block_table: [max_blocks_per_seq] — physical block addresses
    kv_pool: [num_physical_blocks, block_size, 2, num_heads, head_dim]
    """
    num_blocks = len(block_table)
    output = zeros(num_heads, head_dim)

    for block_idx in range(num_blocks):
        physical_block = block_table[block_idx]
        kv_block = kv_pool[physical_block]  # [block_size, 2, num_heads, head_dim]

        for pos in range(block_size):
            key = kv_block[pos, 0]  # [num_heads, head_dim]
            value = kv_block[pos, 1]

            # Standard attention on this tile
            score = dot(query, key) / sqrt(head_dim)
            output += softmax(score) * value

    return output

In practice, this is implemented as a fused CUDA kernel that handles the indirection efficiently. The overhead of non-contiguous access is ~5-10% compared to contiguous attention, but the memory savings (60-80% less fragmentation) more than compensate.

Prefix sharing: Two requests with the same system prompt share physical KV blocks. Divergent generation gets separate blocks via copy-on-write. The block table for the second request starts by pointing to the same physical blocks as the first. When a write occurs (new token generation), the affected block is copied to a new physical block, and the second request’s block table is updated. This is critical for RAG workloads where every request shares a system prompt.

Under the Hood: Flash Attention

Standard attention computes S = softmax(Q x K^T) x V. The intermediate S matrix has shape (seq_len x seq_len), which for a 128K-token sequence is 16 billion elements — 32 GB at FP16. This matrix is written to HBM, read back, and discarded.

Flash Attention avoids this by tiling:

# Simplified Flash Attention (one tile)
def flash_attention_tile(Q_tile, K_tiles, V_tiles):
    """
    Q_tile: [BR, d] — one row-block of Q
    K_tiles: list of [BC, d] — column-blocks of K
    V_tiles: list of [BC, d] — column-blocks of V
    """
    O = zeros(BR, d)     # Output accumulator (SRAM)
    l = zeros(BR)        # Row sum (SRAM)
    m = full(BR, -inf)   # Row max (SRAM)

    for j in range(len(K_tiles)):
        K_j = K_tiles[j]  # Load to SRAM
        V_j = V_tiles[j]  # Load to SRAM

        S_ij = Q_tile @ K_j.T  # [BR, BC] — computed in SRAM
        m_new = max(m, row_max(S_ij))
        P = exp(S_ij - m_new)  # Rescaled softmax
        l = exp(m - m_new) * l + row_sum(P)
        O = exp(m - m_new) * O + P @ V_j
        m = m_new

    O = O / l  # Final rescaling
    return O

The key insight: by keeping the tile in SRAM (40x faster than HBM), Flash Attention avoids the memory bandwidth bottleneck. The full NxN matrix is never materialized. FlashAttention-2 reaches 230 TFLOPs/s on A100 FP16 attention kernels (~72% model FLOPs utilization). FlashAttention-3 on H100 reaches 740 TFLOPs/s on FP16 (75% utilization) and ~1.2 PFLOPs/s on FP8.

Advanced Pattern: Chunked Prefill (TGI v3)

TGI v3 introduced chunked prefill, inspired by Sarathi-Serve. The problem it solves: a single long prefill (e.g., 100K tokens) blocks decode for all other requests in the batch for the duration of the prefill. This creates decode stalls that increase TPOT for every request.

Chunked prefill splits large prompts into fixed-size chunks and interleaves them with decode steps:

# Without chunked prefill:
# Request A (100K tokens) blocks decode for 5 seconds
[Prefill A (100K)] [Decode A] [Decode A] ... [Decode B] [Decode B]

# With chunked prefill:
# Request A's prefill is split into 4K chunks, interleaved with decode
[Prefill A (4K)] [Decode A] [Decode B] [Prefill A (4K)] [Decode A] [Decode B] ...

This bounds TTFT for all requests and prevents a single long prompt from starving the entire batch. The chunk size is controlled by --max-batch-prefill-tokens.

Advanced Pattern: Kernel Fusion

TGI v3 fuses multiple small bookkeeping kernels into single launches. This is critical for small models where kernel launch overhead (several milliseconds each) is significant relative to compute time.

For example, instead of:

kernel_1: update_block_table(...)
kernel_2: compute_logits(...)
kernel_3: sample_token(...)
kernel_4: update_kv_cache(...)

TGI fuses into:

fused_kernel: update_block_table_and_compute_logits_and_sample_and_update_kv(...)

Each kernel launch on CUDA has ~5-15 microseconds of overhead. For a 7B model where each decode step takes ~5ms, eliminating 4 kernel launches saves ~40 microseconds per step — ~0.8% improvement. For a 70B model where each step takes ~40ms, the savings are negligible. But for a 1B model where each step takes ~1ms, eliminating 4 launches saves ~4% — meaningful.

Production Pattern: Warmup and Memory Estimation

On startup, TGI runs a warmup phase that:

  1. Estimates available VRAM: Calculates 95% of available GPU memory after loading the model.
  2. Determines block size: Defaults to 16 tokens per block for PagedAttention.
  3. Calculates max tokens: Divides available VRAM by per-block memory to get total processable tokens.
  4. Records CUDA graphs: Runs the model with fixed batch sizes (1, 2, 4, 8, 16, 32) and records the GPU operations as reusable graphs. This eliminates kernel launch overhead for common batch sizes.
  5. Sets MAX_BATCH_PREFILL_TOKENS and MAX_BATCH_TOTAL_TOKENS: Auto-tunes these based on available memory and model size.

The warmup takes 10-30 seconds depending on model size and GPU count. During this time, the server returns 503 Service Unavailable.

Production Pattern: KV Cache Math

Understanding KV cache memory is essential for capacity planning:

bytes_per_token = 2 * num_layers * num_kv_heads * head_dim * dtype_bytes
Model Layers KV Heads Head Dim Dtype Bytes/Token 8K Context 128K Context
Llama 3.1 8B 32 8 128 BF16 131,072 1.0 GB 16 GB
Llama 3.1 70B 80 8 128 BF16 327,680 2.5 GB 40 GB
Llama 4 70B 80 8 128 FP8 163,840 1.25 GB 20 GB
Qwen 2.5 72B 80 8 128 BF16 327,680 2.5 GB 40 GB
DeepSeek V3 67 16 128 BF16 1,048,576 8 GB 128 GB

For a production deployment with 100 concurrent requests at 8K context each, you need 250 GB of KV cache for Llama 3.1 70B. On 8xH100 (640 GB total, ~480 GB usable after model weights), that’s feasible. On 4xA100 (320 GB total, ~200 GB usable), you need quantization or lower concurrency.

The Results

TGI’s optimizations deliver measurable improvements across every dimension of inference performance. Here are before/after comparisons based on published benchmarks:

Metric Before (Naive) After (TGI v3) Improvement
Throughput (7B, batch 64, A100) ~500 tok/s (static batch) ~3,480 tok/s (continuous batch) 7x
Throughput (70B, batch 128, 8xH100) ~200 tok/s (static batch) ~3,940 tok/s (continuous batch) 19.7x
KV cache fragmentation 60-80% waste <4% waste 15-20x less waste
TTFT (200K prompt, Llama 3.1 8B, L4) ~30s (no prefix cache) ~3.2s (prefix cache) 9.4x
TTFT (200K prompt, Llama 3.1 70B, 8xH100) ~30s (no prefix cache) ~2s (prefix cache) 15x
Max context (Llama 3.1 8B, single L4 24GB) ~10K tokens (vLLM) ~30K tokens (TGI v3) 3x
GPU utilization (high concurrency) 40-50% (static batch) 68-74% (continuous batch) 1.5x
p99 TTFT (7B, 25 concurrent) ~2.5s (no batching) ~0.71s (continuous batch) 3.5x
Memory per request (70B, 8K context) ~4 GB (contiguous alloc) ~2.5 GB (paged alloc) 1.6x
Startup time (70B, 8xH100) ~5 min (manual sharding) ~45s (auto sharding) 6.7x
Decode speedup (speculative decoding) 1x (no speculation) 1.5-2x (n-gram speculation) 1.5-2x

The real-world impact: A team deploying Llama 3.1 70B for a chat application reported the following after switching from a naive static-batching setup to TGI: GPU count dropped from 16 to 8 (50% hardware cost reduction), p95 latency dropped from 12s to 1.8s (6.7x improvement), and maximum concurrent users increased from 50 to 500 (10x capacity increase). The migration took one engineer two days — most of which was tuning --max-batch-total-tokens.

What to Watch Out For

Advice for Getting Started

  1. Run TGI with default settings first. Start with docker run --gpus all -p 8080:80 ghcr.io/huggingface/text-generation-inference:3.3.5 --model-id $MODEL and verify it works with a curl command before tuning anything. The most common beginner mistake is over-configuring every flag before understanding the workload.

  2. Monitor queue and batch metrics before tuning. Check tgi_queue_size and tgi_batch_size. If tgi_queue_size is consistently zero, you have spare capacity. If it’s growing unbounded, increase --max-concurrent-requests or reduce per-request token budgets. If tgi_batch_size is 1 under load, your --max-batch-total-tokens is too low.

  3. Set --max-total-tokens explicitly. The auto-tuned default is conservative and may limit throughput. Start with --max-total-tokens 4096 for chat workloads and increase to 8192 or 16384 for document processing. Each doubling increases VRAM pressure but allows longer generations.

  4. Always use --shm-size 1g or larger. NCCL uses shared memory for inter-process communication. Without enough shared memory, multi-GPU deployments fail with cryptic NCCL errors. On Kubernetes, mount an emptyDir with medium: Memory to /dev/shm.

  5. Start with a small model for testing. Use TinyLlama/TinyLlama-1.1B-Chat-v1.0 for your first deployment. It loads in seconds, fits on any GPU, and lets you validate your configuration before switching to production models.

Lessons Learned

“We deployed TGI with default settings and wondered why throughput was terrible. The issue was --max-batch-total-tokens — the auto-tuned value was 8,000, which meant only 2-3 requests could batch at a time. Setting it to 32,000 tripled throughput immediately. Always check your batch size under load.” — ML Infra Engineer at a fintech company

“Our p99 latency was 30 seconds. We assumed it was a model issue. It was a queue issue — --max-concurrent-requests was set to 1024, but our single A100 could only handle ~8 concurrent requests at 4K context. Requests were piling up in the queue. Setting it to 64 dropped p99 to 2 seconds. The queue is not a buffer — it’s a signal that you’re overloading the system.” — Platform Engineer at an AI startup

“We lost a day debugging NCCL errors on our Kubernetes cluster. The fix was adding emptyDir: medium: Memory to /dev/shm with sizeLimit: 2Gi. The default shared memory on Kubernetes nodes is 64MB — nowhere near enough for NCCL initialization across 8 GPUs. Always check /dev/shm size before deploying multi-GPU TGI.” — DevOps Engineer at a SaaS company

“TGI v3’s prefix caching is incredible for RAG workloads — we saw 10x TTFT improvement on long prompts. But we hit a correctness bug: stale cache entries when the system prompt changed. The cache doesn’t automatically invalidate on model reload. We now restart the container whenever we update the system prompt. Prefix caching is not a cache in the traditional sense — it’s a KV cache reuse optimization, and it assumes the prefix is immutable.” — ML Engineer at a legal tech company

“We benchmarked TGI vs vLLM for our chat application and found TGI had 2x lower TTFT but 1.5x higher TPOT. For interactive chat, TTFT matters more — users notice the first word delay more than subsequent token speed. We chose TGI. For our batch processing pipeline, we chose vLLM for its 2x higher throughput. The right answer depends on your workload, not on which server is ‘better’.” — AI Infrastructure Lead at a healthcare company

“TGI is in maintenance mode as of late 2025. Hugging Face recommends migrating to vLLM or SGLang for new deployments. We’re staying on TGI for now because it works and we don’t want to re-benchmark. But we’re planning the migration. The lesson: open-source projects can change direction. Don’t build your infrastructure around a single project’s roadmap.” — CTO at a mid-size AI company

Next in the Open-Source AI Tools Mastery series: BentoML

NL

Written by Nivant Labs Team

Engineer at Nivant Labs

Share this post

XLinkedInHN