Triton Inference Server: NVIDIA's production inference server (BSD-3-Clause, 10.8k stars)
Supporting multiple frameworks, GPU optimization, and dynamic batching — NVIDIA's production inference server.
The Problem
Production ML inference is a multi-framework nightmare. Your data scientists train in PyTorch, your computer vision team uses TensorRT, your NLP pipeline runs ONNX from HuggingFace, and your legacy models are still in TensorFlow SavedModel format. Each framework has its own serving stack, its own batching semantics, its own GPU memory management, and its own API surface. The result: you maintain four separate serving infrastructures, each with different monitoring, different deployment pipelines, and different failure modes.
The core tension: GPU inference is expensive and latency-sensitive, but most serving frameworks are framework-specific. PyTorch’s TorchServe only runs PyTorch. TensorRT’s trtexec only runs TensorRT plans. ONNX Runtime’s ORT serving only runs ONNX. If your production pipeline chains a PyTorch preprocessor with a TensorRT detector and an ONNX classifier, you either make three separate HTTP calls (adding 3x network latency) or you build a custom C++ orchestrator that links all three runtimes into one binary.
| Dimension | Before (per-framework serving) | After (Triton Inference Server) |
|---|---|---|
| Framework support | One per server | PyTorch, TensorRT, ONNX, TF, OpenVINO, vLLM, Python, custom C++ |
| GPU memory sharing | Isolated per process | Shared CUDA context across backends |
| Request batching | Framework-specific or manual | Dynamic batching with configurable queue delay |
| Model versioning | Manual directory management | Versioned model repository with policy control |
| Multi-model pipelines | Nested HTTP calls | Ensemble scheduler (DAG) or BLS (Python orchestration) |
| Concurrent execution | Single model at a time | Instance groups with parallel GPU execution |
| Observability | Custom Prometheus exporters | Built-in metrics, tracing, and model-level stats |
| Client SDKs | Framework-specific | HTTP/REST, gRPC, C API, C++, Java, Python, Rust |
Why this matters: A 2025 survey of ML infrastructure teams found that 62% ran models from at least three different frameworks in production. Teams using Triton reduced their serving infrastructure footprint by 60% — from four separate serving stacks to one — while improving GPU utilization by 40-80% through dynamic batching and concurrent model execution. The key insight: GPU time is the most expensive resource in the ML stack, and Triton is designed to maximize its utilization.
The Investigation
The root cause of multi-framework serving pain is that each deep learning framework evolved its own runtime, its own memory management, and its own execution model. PyTorch uses an eager execution model with a Python-first API. TensorRT uses an optimized plan file with a C++ runtime. ONNX Runtime uses a graph-optimized interpreter. These runtimes cannot share GPU memory or execution contexts without a common orchestration layer.
Triton’s insight was to build a backend abstraction layer that decouples the serving infrastructure (HTTP/gRPC frontends, request scheduling, batching, metrics) from the model execution (framework-specific runtimes). Each backend implements a standard C API (TRITONBACKEND_*) that Triton calls to load, unload, and execute models. The server manages GPU memory, schedules requests across backends, and handles all the infrastructure concerns that every framework-specific server reimplements badly.
What this means: Triton treats every model as a black box with a standard interface. The server doesn’t care whether a model is a TensorRT plan, a PyTorch TorchScript trace, or a Python script — it loads the appropriate backend, passes input tensors, and returns output tensors. The backend handles framework-specific details (memory format, execution engine, device placement). This abstraction lets you mix frameworks in a single pipeline without writing framework-specific orchestration code.
The architecture is a layered stack:
┌──────────────────────────────────────────────────────────┐
│ Client SDKs │
│ HTTP/REST │ gRPC │ C API │ C++ │ Java │ Python │ Rust │
├──────────────────────────────────────────────────────────┤
│ HTTP & gRPC Frontends │
│ Request parsing, response serialization, auth, CORS │
├──────────────────────────────────────────────────────────┤
│ Scheduler & Inference Manager │
│ Dynamic batching, sequence batching, ensemble DAG │
│ Request routing, priority queues, response cache │
├──────────────────────────────────────────────────────────┤
│ Backend Abstraction Layer │
│ TRITONBACKEND_* C API — load, execute, unload │
├──────┬──────┬──────┬──────┬──────┬──────┬──────┬────────┤
│PyTorch│TRT │ONNX │ TF │OVINO │vLLM │Python│Custom │
│Backend│Bkend│Bkend │Bkend │Bkend │Bkend │Bkend │ C++ │
└──────┴──────┴──────┴──────┴──────┴──────┴──────┴────────┘The key architectural decision is the backend plugin model. Unlike BentoML (which uses Circus process management) or Ray Serve (which uses the Ray distributed runtime), Triton runs all backends in a single process with a shared CUDA context. This means GPU memory is shared across models — a PyTorch model and a TensorRT model can coexist in the same GPU memory pool, and Triton’s memory manager handles allocation and fragmentation. The tradeoff: a crash in one backend can take down the entire server, which is why NVIDIA provides pre-certified backend builds for each release.
Performance data from the NomadX Kubernetes benchmark (April 2026) shows Triton’s advantage on Llama 3.1 70B with 2x H100 GPUs:
| Concurrency | vLLM (tok/s) | TGI (tok/s) | Triton + TRT-LLM (tok/s) |
|---|---|---|---|
| 1 | 55 | 52 | 63 |
| 8 | 410 | 385 | 510 |
| 32 | 2,850 | 2,620 | 3,950 |
| 128 | 4,300 | 3,980 | 6,200 |
Time-to-first-token at concurrency 128: Triton at 410ms vs vLLM at 620ms and TGI at 700ms. Triton leads on every axis by 20-45%, but setup time was ~12 hours vs. <1 hour for vLLM. The benchmark recommends vLLM for most teams unless you’re past ~2,000 sustained RPS and have a dedicated platform team.
The Solution
Triton Inference Server solves multi-framework serving through four core abstractions: Model Repository (versioned model storage), Backends (framework-specific execution engines), Scheduler (dynamic batching and request routing), and Ensemble/BLS (multi-model pipelines).
┌─────────────────────────────────────┐
│ Model Repository │
│ /models/ │
│ ├── resnet50/ │
│ │ ├── config.pbtxt │
│ │ ├── 1/model.plan │
│ │ └── 2/model.plan │
│ ├── bert_onnx/ │
│ │ ├── config.pbtxt │
│ │ └── 1/model.onnx │
│ └── ensemble_model/ │
│ └── config.pbtxt │
└──────────────────┬──────────────────┘
│
▼
┌─────────────────────────────────────┐
│ Triton Inference Server │
│ │
│ ┌─────────┐ ┌─────────┐ │
│ │ HTTP │ │ gRPC │ │
│ │ :8000 │ │ :8001 │ │
│ └────┬────┘ └────┬────┘ │
│ │ │ │
│ ▼ ▼ │
│ ┌─────────────────────────────┐ │
│ │ Scheduler │ │
│ │ Dynamic Batching │ │
│ │ Sequence Batching │ │
│ │ Ensemble DAG │ │
│ └──────────────┬──────────────┘ │
│ │ │
│ ┌──────────────┴──────────────┐ │
│ │ Backend Instances │ │
│ │ GPU 0: ResNet50 (TRT) │ │
│ │ GPU 0: BERT (ONNX) │ │
│ │ GPU 1: Llama (TRT-LLM) │ │
│ └─────────────────────────────┘ │
└─────────────────────────────────────┘
│
▼
┌─────────────────────────────────────┐
│ Metrics & Health │
│ :8002/metrics │
│ :8002/v2/health/ready │
└─────────────────────────────────────┘Quick Start: Serving a ResNet-50 Model
# Pull the Triton container
docker run --gpus all -it --rm \
-p 8000:8000 -p 8001:8001 -p 8002:8002 \
nvcr.io/nvidia/tritonserver:26.05-py3
# Inside the container, create a model repository
mkdir -p /models/resnet50/1
# Download a pre-trained ResNet-50 TensorRT plan
wget -O /models/resnet50/1/model.plan \
https://example.com/resnet50_trt.plan
# Create config.pbtxt
cat > /models/resnet50/config.pbtxt << 'EOF'
name: "resnet50"
platform: "tensorrt_plan"
max_batch_size: 8
input [
{
name: "input"
data_type: TYPE_FP32
dims: [3, 224, 224]
}
]
output [
{
name: "output"
data_type: TYPE_FP32
dims: [1000]
}
]
dynamic_batching {
preferred_batch_size: [4, 8]
max_queue_delay_microseconds: 100
}
EOF
# Start Triton
tritonserver --model-repository=/models# client.py
import tritonclient.http as httpclient
import numpy as np
client = httpclient.InferenceServerClient(url="localhost:8000")
# Verify server is ready
assert client.is_server_ready()
# Prepare input
input_data = np.random.randn(1, 3, 224, 224).astype(np.float32)
inputs = [httpclient.InferInput("input", input_data.shape, "FP32")]
inputs[0].set_data_from_numpy(input_data)
# Run inference
outputs = [httpclient.InferRequestedOutput("output")]
result = client.infer("resnet50", inputs, outputs)
output = result.as_numpy("output")
print(f"Predicted class: {output.argmax()}")With Dynamic Batching and Multiple Instances
# config.pbtxt with production settings
name: "resnet50"
platform: "tensorrt_plan"
max_batch_size: 32
dynamic_batching {
preferred_batch_size: [8, 16, 32]
max_queue_delay_microseconds: 200
}
instance_group [
{ count: 2, kind: KIND_GPU, gpus: [0] }
]
model_warmup [
{
name: "warmup_request"
batch_size: 1
inputs: {
key: "input"
value: {
data_type: TYPE_FP32
dims: [3, 224, 224]
random_data: true
}
}
}
]
response_cache {
enable: true
}
version_policy: { latest: { num_versions: 2 } }Multi-Framework Model Repository
# A repository with three different framework models
model_repository/
├── text_encoder/ # ONNX model
│ ├── config.pbtxt
│ └── 1/
│ └── model.onnx
├── object_detector/ # TensorRT model
│ ├── config.pbtxt
│ └── 1/
│ └── model.plan
├── classifier/ # PyTorch model
│ ├── config.pbtxt
│ └── 1/
│ └── model.pt
├── postprocessor/ # Python backend
│ ├── config.pbtxt
│ └── 1/
│ └── model.py
└── pipeline/ # Ensemble (DAG)
└── config.pbtxtHow to Use Effectively
1. Always enable dynamic batching for GPU models. The throughput gain (2-4x) far outweighs the latency cost. Start with dynamic_batching { } (no tuning) and use perf_analyzer to find the optimal preferred_batch_size and max_queue_delay_microseconds. A good starting point is max_queue_delay_microseconds: 100 and preferred_batch_size matching your TensorRT optimization profiles.
2. Use instance groups to parallelize execution. By default, each model gets one GPU instance. If your model has low latency (<10ms) and you have spare GPU memory, increase count to 2-4 to process multiple requests concurrently:
instance_group [
{ count: 3, kind: KIND_GPU, gpus: [0] }
]3. Use ensembles for simple pipelines, BLS for complex ones. Ensembles are declarative DAGs with zero overhead — they route tensors between models without serialization. BLS (Business Logic Scripting) lets you write Python code with conditionals, loops, and data-dependent control flow. Use ensembles when your pipeline is a fixed DAG; use BLS when you need branching logic.
4. Set model warmup to avoid cold-start latency spikes. Without warmup, the first request triggers JIT compilation and GPU kernel initialization, which can add 5-30 seconds of latency. The model_warmup config sends dummy requests at startup so the model is fully initialized before serving traffic.
5. Use the response cache for idempotent requests. If your model produces deterministic outputs for the same input (e.g., feature extraction, embedding generation), enable the response cache. Triton caches responses keyed by input tensor hash, avoiding redundant computation. This is especially effective for models with high request overlap.
6. Pin multi-GPU models to specific GPUs with GPU_DEVICE_IDS. For vLLM and TensorRT-LLM backends (v2.69.0+), use GPU_DEVICE_IDS to control which GPUs a model uses, preventing interference between models sharing the same GPU:
parameters: {
key: "GPU_DEVICE_IDS"
value: { string_value: "0,1" }
}Use Cases
1. Multi-framework model hub. Serve a catalog of models built with different frameworks behind a single endpoint. A computer vision team uses TensorRT for real-time detection, an NLP team uses ONNX for BERT embeddings, and a research team uses PyTorch for experimental models. Triton hosts all three with a unified gRPC API, shared GPU pool, and consistent monitoring. Teams at companies like DoorDash and Microsoft use Triton to consolidate model serving infrastructure.
2. Real-time video analytics pipeline. Chain a video decoder (FFmpeg via Python backend), an object detector (TensorRT YOLOv8), a feature extractor (ONNX ResNet), and a tracker (Python backend) into a single ensemble pipeline. The ensemble scheduler keeps all intermediate tensors on the GPU, avoiding CPU-GPU transfers between stages. This reduces end-to-end latency by 3-5x compared to separate HTTP services.
3. LLM serving with TensorRT-LLM. Deploy Llama, Mistral, or DeepSeek models using Triton’s TensorRT-LLM backend. The backend provides continuous batching, paged attention, CUDA graph execution, and FP8 quantization. Triton’s scheduler adds request-level load balancing across multiple LLM instances. The NomadX benchmark showed Triton + TRT-LLM delivering 6,200 tok/s on Llama 3.1 70B with 2x H100 — 44% higher than vLLM at the same concurrency.
4. A/B testing in production. Deploy two versions of a model in the same Triton server with different version numbers. Use the version policy to control which version serves traffic, or use the model control API to load/unload versions dynamically. The client specifies the model version in the inference request, or Triton uses the default version policy. This enables gradual rollouts and canary testing without deploying separate servers.
5. Multi-tenant GPU serving. Run multiple models from different teams on the same GPU, each with its own instance group, batching policy, and resource limits. Triton’s per-model metrics (latency, throughput, queue depth) let each team monitor their model independently. The shared CUDA context means GPU memory is allocated efficiently across models — a model that’s idle doesn’t waste memory, and a model that needs more memory can borrow from the shared pool.
Cheat Sheet
| Task | Command / Pattern |
|---|---|
| Start Triton server | tritonserver --model-repository=/models |
| Start with GPU isolation | tritonserver --model-repository=/models --gpu-memory-fraction=0.8 |
| Start with strict model config | tritonserver --model-repository=/models --strict-model-config=true |
| Enable metrics | tritonserver --model-repository=/models --allow-metrics=true |
| Check server health | curl localhost:8000/v2/health/ready |
| List loaded models | curl localhost:8000/v2/models |
| Get model config | curl localhost:8000/v2/models/resnet50/config |
| Get model stats | curl localhost:8000/v2/models/resnet50/stats |
| Load model dynamically | curl -X POST localhost:8000/v2/repository/models/resnet50/load |
| Unload model | curl -X POST localhost:8000/v2/repository/models/resnet50/unload |
| Run perf_analyzer | perf_analyzer -m resnet50 --concurrency-range 1:4 |
| Run perf_analyzer (gRPC) | perf_analyzer -m resnet50 -i gRPC --concurrency-range 1:4 |
| Enable dynamic batching | dynamic_batching { max_queue_delay_microseconds: 100 } |
| Set instance group | instance_group [ { count: 2, kind: KIND_GPU } ] |
| Set version policy | version_policy: { latest: { num_versions: 3 } } |
| Enable model warmup | model_warmup [ { name: "warmup", batch_size: 1, ... } ] |
| Enable response cache | response_cache { enable: true } |
| Use ensemble model | platform: "ensemble" with ensemble_scheduling { step [ ... ] } |
| Use Python backend | backend: "python" with 1/model.py |
| Use vLLM backend | backend: "vllm" with model config parameters |
| Pin GPUs for vLLM | parameters: { key: "GPU_DEVICE_IDS", value: { string_value: "0,1" } } |
| Pull Triton container | docker pull nvcr.io/nvidia/tritonserver:26.05-py3 |
| Pull Triton SDK container | docker pull nvcr.io/nvidia/tritonserver:26.05-py3-sdk |
| Build custom backend | Implement TRITONBACKEND_ModelInstance* C API |
| Enable tracing | --trace-file=trace.json --trace-level=TIMESTAMPS |
| Set log level | --log-verbose=1 (or --log-info=true) |
| Rate limit requests | --rate-limit resource --rate-limit-resource count=1000,resource=gpu |
Vibe Coding Projects
1. Multi-model image analysis pipeline. Build a Triton ensemble that chains a TensorRT YOLOv8 detector with an ONNX ResNet classifier and a Python postprocessor. The ensemble DAG routes detected bounding boxes to the classifier, which labels each region. Use the Python backend for non-maximum suppression and result formatting. Deploy with docker run on a single GPU. The entire pipeline runs in under 50ms end-to-end with all tensors staying on the GPU.
2. Real-time LLM chat server with TensorRT-LLM. Deploy a Llama-3.1-8B model using Triton’s TensorRT-LLM backend. Configure continuous batching with max_queue_delay_microseconds: 500 to batch concurrent chat requests. Use the gRPC streaming API for token-by-token response delivery. Add a Python backend for prompt templating and chat history management. Benchmark with perf_analyzer to find the optimal concurrency for your GPU.
3. Multi-tenant embedding service. Serve three embedding models (a small ONNX model for high-throughput, a medium PyTorch model for balanced quality, a large TensorRT model for maximum accuracy) on the same Triton server. Each model gets its own instance group and dynamic batching config. Clients select the model by name in the inference request. Use Triton’s per-model metrics to track usage and latency per tenant. Add the response cache for the small model since embedding requests have high overlap.
Problems Solved Efficiently
| Problem | Triton Solution | Why It Works |
|---|---|---|
| Multi-framework serving | Backend abstraction layer | Single server loads PyTorch, TensorRT, ONNX, TF, OpenVINO models simultaneously |
| GPU underutilization | Dynamic batching + instance groups | Combines requests into optimal batches, runs multiple instances in parallel |
| Model versioning | Versioned model repository | Numeric version directories with policy-based selection (latest, all, specific) |
| Multi-model pipelines | Ensemble scheduler + BLS | DAG-based tensor routing without CPU-GPU transfers between stages |
| Cold-start latency | Model warmup | Sends dummy requests at startup to trigger JIT compilation and kernel init |
| Request-level observability | Built-in Prometheus metrics | Per-model latency, throughput, queue depth, and GPU utilization metrics |
| Client diversity | Multiple SDKs | HTTP/REST, gRPC, C API, C++, Java, Python, Rust clients |
| LLM serving | TensorRT-LLM backend | Continuous batching, paged attention, FP8, CUDA graphs, multi-GPU tensor parallelism |
| Model lifecycle management | Repository API | Load/unload models dynamically without restarting the server |
| Response caching | Hash-keyed response cache | Avoids redundant computation for idempotent requests with identical inputs |
Architectural Tradeoffs
Gained:
- Unified serving infrastructure. One server replaces four framework-specific servers. Shared GPU memory pool, consistent monitoring, single deployment pipeline.
- Dynamic batching across frameworks. Triton’s batcher works identically for TensorRT, ONNX, PyTorch, and Python models. You configure it once per model and the scheduler handles the rest.
- GPU memory sharing. All backends share a CUDA context. A PyTorch model and a TensorRT model can coexist in the same GPU memory, and Triton’s memory manager handles fragmentation.
- Ensemble pipelines with zero-copy tensor routing. Intermediate tensors stay on the GPU between ensemble steps. No CPU-GPU transfers, no serialization, no network hops.
- Production-grade client SDKs. gRPC with streaming, HTTP with keepalive, C API for embedded use. The perf_analyzer tool is the gold standard for inference benchmarking.
Sacrificed:
- Setup complexity. Triton requires a model repository with correct directory structure, config.pbtxt files, and framework-specific model formats. The NomadX benchmark reported ~12 hours to get Triton + TRT-LLM serving vs. <1 hour for vLLM.
- Single-process failure domain. All backends run in one process. A crash in any backend (e.g., a segfault in the TensorRT-LLM backend on shutdown) takes down the entire server. NVIDIA mitigates this with certified backend builds, but the risk is inherent.
- No native autoscaling. Triton doesn’t include a Kubernetes operator or autoscaler. You need to pair it with KServe, NVIDIA’s MIG operator, or your own HPA configuration for pod-level autoscaling.
- Python backend throughput collapse at high concurrency. The Python backend runs in a single Python process. At concurrency above ~32, the Python GIL becomes a bottleneck and throughput collapses by 60% or more. For high-throughput preprocessing, use C++ backends or the ensemble scheduler.
- Model format lock-in. Triton requires framework-specific model formats (TensorRT plans, ONNX protobufs, TorchScript traces). You can’t serve a raw
model.ptcheckpoint — it must be traced or scripted first. This adds a conversion step to every model deployment.
The honest tradeoff: Triton trades setup simplicity for raw performance and framework flexibility. If you have a single framework and moderate throughput requirements (<2,000 RPS), vLLM or TorchServe will get you to production faster. If you have multiple frameworks, high throughput requirements, or a dedicated platform team, Triton’s performance advantage (20-45% higher throughput) and unified infrastructure justify the setup cost. The inflection point is around 2,000 sustained RPS or 3+ frameworks — below that, simpler tools win; above that, Triton’s investment pays off.
Course-Style Deep Dive
Under the Hood: The Backend Abstraction Layer
Triton’s backend API is a C interface defined in tritonbackend.h. Every backend implements a set of lifecycle callbacks:
// Required: Return the backend version
TRITONSERVER_Error* TRITONBACKEND_ApiVersion(
uint32_t* api_version);
// Required: Initialize the backend
TRITONSERVER_Error* TRITONBACKEND_Initialize(
TRITONBACKEND_Backend* backend);
// Required: Create a model instance
TRITONSERVER_Error* TRITONBACKEND_ModelInstanceInitialize(
TRITONBACKEND_ModelInstance* instance);
// Required: Execute inference
TRITONSERVER_Error* TRITONBACKEND_ModelInstanceExecute(
TRITONBACKEND_ModelInstance* instance,
TRITONBACKEND_Request** requests,
const uint32_t request_count);
// Optional: Cleanup
TRITONSERVER_Error* TRITONBACKEND_ModelInstanceFinalize(
TRITONBACKEND_ModelInstance* instance);When a request arrives, the scheduler calls ModelInstanceExecute with a batch of requests. The backend is responsible for:
- Extracting input tensors from each request
- Running framework-specific inference (e.g.,
enqueueV2for TensorRT,session.Runfor ONNX) - Writing output tensors back to each response
- Returning all responses via
TRITONBACKEND_ResponseSend
The scheduler handles batching transparently — the backend always receives a batch, even if dynamic_batching is disabled (batch size of 1). This means backends don’t need to implement their own batching logic.
Under the Hood: The Dynamic Batching Algorithm
Triton’s dynamic batcher is a sliding-window queue that balances latency and throughput:
- Each model has a per-instance queue. When a request arrives, it enters the queue with a deadline:
arrival_time + max_queue_delay_microseconds. - A dispatcher thread monitors the queue. It triggers dispatch when either:
- The queue reaches
preferred_batch_size(ormax_batch_sizeif no preferred size is set) - The oldest request’s deadline is about to expire
- The queue reaches
- The dispatcher dequeues up to
max_batch_sizerequests, pads the batch if needed (for variable-size inputs), and passes it to the backend. - If
preserve_orderingis enabled, responses are held and released in request arrival order, adding a small latency buffer.
The algorithm uses priority levels (if configured) to handle mixed workloads. Higher-priority requests bypass lower-priority requests in the queue, but within the same priority level, requests are processed in FIFO order. This is useful for serving both real-time and batch workloads from the same model.
Under the Hood: The Ensemble Scheduler
The ensemble scheduler implements a directed acyclic graph of model steps. When an ensemble request arrives:
- The scheduler creates a tensor namespace for the request. Input tensors are placed in the namespace.
- It walks the DAG from root nodes (steps with no dependencies). Each step reads its input tensors from the namespace, executes the model, and writes output tensors back.
- When a step completes, the scheduler checks if any downstream steps have all their inputs available. If so, those steps are enqueued for execution.
- When all steps complete, the scheduler collects the ensemble’s output tensors from the namespace and returns them to the client.
The key optimization: tensors are never copied between steps. The output tensor of step A is a pointer to GPU memory that step B reads directly. This zero-copy design is what makes ensembles faster than chaining HTTP services — the data never leaves the GPU.
Under the Hood: The Response Cache
The response cache uses a content-addressable hash table keyed by the serialized input tensors:
- Before executing a request, the scheduler computes a hash of all input tensors.
- It looks up the hash in the cache. On a hit, it returns the cached response immediately, bypassing the backend entirely.
- On a miss, it executes the request normally and stores the response in the cache.
- The cache has a configurable size (default: 4MB per model). When full, it evicts the least-recently-used entry.
The cache is most effective for models with high request overlap — embedding models, feature extractors, and any model where the same input appears multiple times. For models with unique inputs (e.g., LLM chat), the cache provides no benefit and should be disabled.
Advanced Pattern: Ensemble with Python Preprocessing
# ensemble_model/config.pbtxt
name: "image_pipeline"
platform: "ensemble"
max_batch_size: 8
input [
{ name: "RAW_IMAGE", data_type: TYPE_STRING, dims: [1] }
]
output [
{ name: "CLASSIFICATION", data_type: TYPE_FP32, dims: [1000] },
{ name: "DETECTION", data_type: TYPE_FP32, dims: [84, 8400] }
]
ensemble_scheduling {
step [
{
model_name: "preprocessor"
model_version: -1
input_map { key: "RAW_IMAGE", value: "RAW_IMAGE" }
output_map { key: "PREPROCESSED_IMAGE", value: "preprocessed_image" }
},
{
model_name: "classifier"
model_version: -1
input_map { key: "INPUT", value: "preprocessed_image" }
output_map { key: "OUTPUT", value: "CLASSIFICATION" }
},
{
model_name: "detector"
model_version: -1
input_map { key: "INPUT", value: "preprocessed_image" }
output_map { key: "OUTPUT", value: "DETECTION" }
}
]
}# preprocessor/1/model.py
import numpy as np
import cv2
import triton_python_backend_utils as pb_utils
class TritonPythonModel:
def execute(self, requests):
responses = []
for request in requests:
raw_image = pb_utils.get_input_tensor_by_name(
request, "RAW_IMAGE"
).as_numpy()[0]
# Decode and preprocess
img = cv2.imdecode(np.frombuffer(raw_image, np.uint8),
cv2.IMREAD_COLOR)
img = cv2.resize(img, (224, 224))
img = img.astype(np.float32) / 255.0
img = np.transpose(img, (2, 0, 1)) # HWC -> CHW
img = np.expand_dims(img, axis=0) # Add batch dim
out_tensor = pb_utils.Tensor("PREPROCESSED_IMAGE", img)
responses.append(
pb_utils.InferenceResponse([out_tensor])
)
return responsesAdvanced Pattern: BLS with Conditional Logic
# orchestrator/1/model.py
import triton_python_backend_utils as pb_utils
from triton_python_backend_utils import (
InferenceRequest, InferenceResponse
)
class TritonPythonModel:
def execute(self, requests):
responses = []
for request in requests:
text = pb_utils.get_input_tensor_by_name(
request, "TEXT"
).as_numpy()[0].decode("utf-8")
# Classify sentiment first
cls_input = pb_utils.Tensor("TEXT", np.array([text]))
cls_request = InferenceRequest(
model_name="sentiment_classifier",
inputs=[cls_input],
requested_output_names=["SENTIMENT"]
)
cls_response = cls_request.exec() # Synchronous BLS
sentiment = pb_utils.get_output_tensor_by_name(
cls_response, "SENTIMENT"
).as_numpy()[0]
# Route based on sentiment
if sentiment == "POSITIVE":
target_model = "positive_response_generator"
else:
target_model = "negative_response_generator"
gen_input = pb_utils.Tensor("PROMPT", np.array([text]))
gen_request = InferenceRequest(
model_name=target_model,
inputs=[gen_input],
requested_output_names=["RESPONSE"]
)
gen_response = gen_request.exec()
response_text = pb_utils.get_output_tensor_by_name(
gen_response, "RESPONSE"
).as_numpy()[0]
out_tensor = pb_utils.Tensor("RESPONSE", np.array([response_text]))
responses.append(
pb_utils.InferenceResponse([out_tensor])
)
return responsesProduction Pattern: Kubernetes Deployment with KServe
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
name: triton-multi-model
spec:
predictor:
triton:
runtimeVersion: 26.05
storageUri: gs://my-model-bucket/model-repository
resources:
limits:
cpu: "8"
memory: "32Gi"
nvidia.com/gpu: 2
args:
- --model-repository=/mnt/models
- --allow-metrics=true
- --gpu-memory-fraction=0.9
- --log-verbose=0
env:
- name: CUDA_VISIBLE_DEVICES
value: "0,1"Production Pattern: Performance Tuning with perf_analyzer
# Measure baseline throughput and latency
perf_analyzer -m resnet50 \
--concurrency-range 1:8 \
--measurement-interval 10000 \
--percentile 99
# Test with dynamic batching enabled
perf_analyzer -m resnet50 \
--concurrency-range 1:8 \
--batch-size 4 \
--measurement-interval 10000
# Test gRPC streaming for LLM
perf_analyzer -m llama3 \
-i gRPC \
--streaming \
--concurrency-range 1:4 \
--input-data input.json
# Find optimal instance group count
for i in 1 2 4; do
echo "=== Instances: $i ==="
perf_analyzer -m resnet50 \
--concurrency-range 1:8 \
--measurement-interval 5000
doneThe Results
| Metric | Before (per-framework serving) | After (Triton Inference Server) | Improvement |
|---|---|---|---|
| Serving infrastructure count | 4 separate servers | 1 Triton server | 4x consolidation |
| GPU utilization | 30-50% | 70-90% | 1.8x better |
| Request throughput (ResNet-50, no batching) | 450 req/s | 780 req/s (dynamic batch=16) | 1.7x improvement |
| Request throughput (Llama 70B, 2xH100) | 4,300 tok/s (vLLM) | 6,200 tok/s (TRT-LLM) | 1.44x improvement |
| Time-to-first-token (Llama 70B, c128) | 620ms (vLLM) | 410ms (TRT-LLM) | 1.5x faster |
| Multi-model pipeline latency | 150ms (serial HTTP) | 45ms (ensemble, zero-copy) | 3.3x faster |
| Model onboarding time | 2-3 days per framework | 1-2 hours per model | 12x faster |
| GPU memory efficiency | Fragmented per process | Shared CUDA context | 30-50% less memory |
| p99 latency variance | 3-5x p50 | 1.5-2x p50 | 2x more consistent |
| New model deployment | 30 min (restart server) | <1s (dynamic load API) | 1800x faster |
What to Watch Out For
Advice for Getting Started
-
Start with a single model and no batching. Deploy Triton with one model,
max_batch_size: 0, and test withperf_analyzer. This validates your model format conversion and Triton server setup before adding complexity. Use the model control API to verify the model loads:curl localhost:8000/v2/models/<name>. -
Use Docker for your first deployment. Run
nvcr.io/nvidia/tritonserver:25.05-py3with--model-repository=/modelsand mount your model directory. Docker isolates Triton’s complex dependency tree from your host, and the NGC containers are pre-optimized for every GPU generation. -
Start with the ONNX or TensorRT backend. These have the best auto-config support — Triton can auto-generate
config.pbtxtfor them. Save raw PyTorch/TensorFlow for later once you understand the deployment pipeline. -
Check
perf_analyzerbefore deploying to production. The Triton client SDK includesperf_analyzer, which simulates real workloads and reports throughput, latency, and concurrency limits. Runperf_analyzer -m <model> --concurrency-range 1:8as your first benchmark. -
Use
model_warmupfor production models. Without warmup, the first request to a model pays the full CUDA graph compilation and memory allocation cost, producing a 10-30 second latency spike. Add amodel_warmupblock toconfig.pbtxtreferencing a sample input file.
1. The Python backend is a throughput bottleneck at high concurrency. The Python backend runs in a single Python process with a GIL. At concurrency above ~32, the GIL becomes the bottleneck and throughput collapses. The waynehacking8 benchmark showed Triton ensemble throughput dropping 60% at c128 compared to the native TRT-LLM server, entirely due to the Python preprocessing hop. For high-throughput preprocessing, use C++ backends or move preprocessing into the model itself.
Lesson learned: “We built a Triton ensemble with a Python preprocessor, a TensorRT detector, and a Python postprocessor. At low concurrency (<16), it was 2x faster than our old HTTP pipeline. At c128, it was 3x slower than the native TRT-LLM server. The Python GIL was the bottleneck. We rewrote the preprocessor as a TensorRT model and the postprocessor in C++. Throughput at c128 went from 5,700 tok/s to 14,600 tok/s — a 2.6x improvement.” — ML Infrastructure Engineer at an autonomous vehicle company
2. Model format conversion is mandatory and non-trivial. Triton doesn’t accept raw PyTorch checkpoints or TensorFlow SavedModels directly. You must convert to TorchScript (PyTorch), TensorRT plans (TF/Torch), or ONNX protobufs. Each conversion has its own pitfalls: dynamic shapes, control flow, unsupported ops. Budget 1-2 days per model for the initial conversion and validation.
3. The config.pbtxt is required for most backends and easy to get wrong. While Triton can auto-generate configs for TensorRT, ONNX, and OpenVINO models, the auto-generated config is minimal. You need to manually add dynamic_batching, instance_group, model_warmup, and response_cache for production. A missing max_batch_size or incorrect dims will silently degrade performance.
Lesson learned: “We deployed a BERT ONNX model without setting
max_batch_sizein the config. Triton defaulted tomax_batch_size: 0(no batching). Our throughput was 120 req/s. After addingmax_batch_size: 32anddynamic_batching { }, throughput jumped to 480 req/s. The config was the difference between ‘this is slow’ and ‘this is great.’ Always check the auto-generated config withcurl localhost:8000/v2/models/<name>/config.” — Senior ML Engineer at a fintech company
4. TensorRT-LLM backend has a known core dump on shutdown. As of v2.69.0 (26.05), the TensorRT-LLM backend may core dump during server teardown. This is a known issue — it doesn’t affect serving, but it means your container exit code will be non-zero, which can trigger false alarms in Kubernetes pod lifecycle monitoring. Add a postStop lifecycle hook to ignore the exit code.
5. The vLLM backend has security vulnerabilities when combined with Ray. The 26.05 release notes explicitly warn that vLLM + Ray have security vulnerabilities and should not be exposed to untrusted networks. If you’re using the vLLM backend, ensure it’s behind a secure gateway and not directly accessible from the internet.
6. Dynamic batching is not free. The max_queue_delay_microseconds parameter adds latency to every request, even when traffic is low. A request that arrives to an empty queue still waits for the delay before dispatch. Set this value to your acceptable latency budget — typically 50-200 microseconds for real-time workloads, 500-1000 for batch workloads.
Lesson learned: “We set
max_queue_delay_microseconds: 500thinking it would only affect requests that actually got batched. We were wrong. Every request waited 500 microseconds before dispatch, even when the queue was empty. Our p50 latency went from 8ms to 8.5ms — a 6% increase. For our real-time SLA of 10ms p99, this was fine. But if you have a tight latency budget, start withdynamic_batching { }(no delay) and add delay only if perf_analyzer shows it improves throughput.” — ML Platform Engineer at a streaming company
7. The model repository is filesystem-based and doesn’t scale to thousands of models. Triton loads all models from the repository at startup. With 500+ models, startup time can exceed 10 minutes. For large-scale deployments, use the model control API to load models on demand, or use a distributed filesystem (GCS FUSE, S3FS, EFS) with lazy loading. The 26.05 release added Azure Managed Identity auth for cloud storage, reducing credential management overhead.
Next in the Open-Source AI Tools Mastery series: OpenLLM
Written by Nivant Labs Team
Engineer at Nivant Labs