OpenLLM: BentoML's LLM serving platform (Apache 2.0, 10k stars)

The Problem

Serving large language models in production is harder than it should be. You have a model checkpoint from HuggingFace, a GPU (or four), and a requirement to expose an OpenAI-compatible /v1/chat/completions endpoint. The naive approach is to wrap the model in FastAPI, load the weights at startup, and call model.generate() on every request. This works for one user on one GPU. It breaks when you have concurrent users, need continuous batching, want to swap models without downtime, or need to scale across multiple GPUs.

The core tension: LLM inference is fundamentally different from standard ML inference. LLMs are memory-bound (KV cache dominates GPU memory), benefit from continuous batching (interleaving prefill and decode phases), and require careful memory management (fragmentation kills throughput on long sequences). A generic web framework like FastAPI gives you none of this — you have to build PagedAttention, continuous batching, KV cache management, and tensor parallelism from scratch.

Dimension	Before (FastAPI + raw Transformers)	After (OpenLLM)
Setup time	2-3 days (Docker, CUDA, model download, API wiring)	2 minutes (pip install openllm && openllm serve)
Concurrent users	1-2 (sequential generation)	256+ (continuous batching via vLLM)
GPU memory efficiency	40-60% (fragmented KV cache)	95%+ (PagedAttention)
Model switching	Code change + redeploy	openllm serve <model>:<tag>
API compatibility	Custom schema	OpenAI-compatible /v1/* endpoints
Deployment targets	Single Docker host	Docker, Kubernetes, BentoCloud
Multi-GPU scaling	Manual tensor parallelism	Declarative --tensor-parallel-size

Why this matters: A 2025 survey of AI infrastructure teams found that 72% spent more time on LLM serving infrastructure than on model development. Teams using purpose-built LLM serving platforms like OpenLLM reduced time-to-production from weeks to hours. OpenLLM specifically eliminates the gap between “I have a model checkpoint” and “I have a production API with OpenAI-compatible endpoints.”

The Investigation

The root cause of LLM serving pain is that transformer inference has unique characteristics that generic web frameworks don’t address. The attention mechanism’s KV cache grows with sequence length and batch size, creating a memory management problem that standard allocators handle poorly. The autoregressive generation loop (one token at a time) means you’re constantly switching between compute-bound prefill phases and memory-bound decode phases.

OpenLLM’s insight was to build on top of BentoML (the unified model serving framework) and vLLM (the high-throughput inference engine), creating a layered architecture that abstracts away the complexity:

┌──────────────────────────────────────────────────────┐
│              OpenAI-Compatible API (/v1/*)             │
│              Built-in Chat UI (/chat)                  │
├──────────────────────────────────────────────────────┤
│              OpenLLM CLI / SDK                         │
│  openllm serve / openllm run / openllm deploy          │
├──────────────────────────────────────────────────────┤
│              BentoML Serving Framework                 │
│  REST API generation, adaptive batching, model store   │
│  Docker containerization, distributed serving          │
├──────────────────────────────────────────────────────┤
│              vLLM Inference Backend                    │
│  PagedAttention, continuous batching, tensor parallel  │
│  Prefix caching, chunked prefill, speculative decoding │
├──────────────────────────────────────────────────────┤
│              Model Weights (HuggingFace / custom)       │
│  Llama, DeepSeek, Mistral, Qwen, Gemma, Phi, Pixtral   │
└──────────────────────────────────────────────────────┘

What this means: OpenLLM is not a standalone inference engine — it’s a serving platform that orchestrates vLLM (the inference backend) through BentoML (the serving framework). The vLLM layer handles the hard problems (PagedAttention, continuous batching, tensor parallelism). The BentoML layer handles the operational concerns (API generation, containerization, deployment, scaling). OpenLLM ties them together with a unified CLI and model repository system.

The key architectural decision is the model repository system. Instead of requiring users to manually download model weights and configure inference engines, OpenLLM maintains a curated catalog of supported models with pre-configured serving parameters. Each model entry specifies the optimal backend, quantization settings, tensor parallelism configuration, and GPU requirements. This means openllm serve deepseek:r1-671b automatically configures 16-way tensor parallelism across 80GB GPUs — no manual tuning required.

Performance data from the vLLM team’s benchmarks shows the impact of the underlying inference engine:

Model	Backend	Throughput (tok/s)	p50 TTFT	p99 TTFT
Llama 3 8B (no batching)	Raw Transformers	45	38ms	85ms
Llama 3 8B (continuous batching)	vLLM	285	78ms	145ms
Llama 3 8B (128 concurrent)	vLLM	4,200	250ms	850ms
Llama 3 70B (FP16, 2x A100)	vLLM	1,800	180ms	520ms
DeepSeek R1 671B (FP8, 16x H100)	vLLM	3,200	450ms	1.8s

Continuous batching delivers 6x throughput improvement over sequential processing — the single most impactful optimization for LLM serving.

The Solution

OpenLLM solves LLM serving through four core abstractions: one-command serving (CLI), OpenAI-compatible API (drop-in replacement), model repository (curated catalog), and multi-target deployment (local, Docker, K8s, BentoCloud).

                    ┌─────────────────────────────┐
                    │     OpenAI SDK Client        │
                    │  (Python / JS / curl)        │
                    └──────────┬──────────────────┘
                               │
                               ▼
                    ┌─────────────────────────────┐
                    │     OpenLLM Server           │
                    │  http://localhost:3000        │
                    │  ┌───────────────────────┐   │
                    │  │ /v1/chat/completions   │   │
                    │  │ /v1/models             │   │
                    │  │ /chat (UI)             │   │
                    │  │ /health                │   │
                    │  └──────────┬────────────┘   │
                    └─────────────┼────────────────┘
                                  │
                    ┌─────────────┼────────────────┐
                    │             ▼                 │
                    │  ┌───────────────────────┐   │
                    │  │  vLLM Engine          │   │
                    │  │  PagedAttention       │   │
                    │  │  Continuous Batching  │   │
                    │  │  Tensor Parallelism   │   │
                    │  │  Prefix Caching       │   │
                    │  └───────────────────────┘   │
                    │  GPU 0 .. GPU N              │
                    └─────────────────────────────┘

Quick Start: Serving Llama 3.2

# Install
pip install openllm

# Serve a model (one command)
openllm serve llama3.2:1b

# Server is live at http://localhost:3000
# OpenAI-compatible API at /v1/chat/completions
# Chat UI at /chat

# client.py — works with any OpenAI SDK
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:3000/v1",
    api_key="not-needed",  # OpenLLM doesn't require auth locally
)

response = client.chat.completions.create(
    model="meta-llama/Llama-3.2-1B-Instruct",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain quantum computing in 3 sentences."},
    ],
    temperature=0.7,
    max_tokens=256,
    stream=True,
)

for chunk in response:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

Interactive CLI Chat

# Start an interactive terminal chat session
openllm run llama3:8b

# Or with a system prompt
openllm run llama3:8b --system-prompt "You are a code review assistant."

Model Repository Management

# List all available models
openllm model list

# Get details about a specific model
openllm model get llama3.2:1b

# Update the model catalog
openllm repo update

# Add a custom model repository
openllm repo add my-models https://github.com/myorg/openllm-models

Deploy to BentoCloud

# Log in to BentoCloud
bentoml cloud login --api-token <your-token>

# Build and push
openllm build llama3.2:1b --backend=vllm --push

# Deploy with autoscaling
openllm deploy llama3.2:1b --env HF_TOKEN

# Update to scale-to-zero (pay only when used)
bentoml deployment update llama3.2:1b --scaling-min 0 --scaling-max 3

How to Use Effectively

1. Always set HF_TOKEN for gated models. Llama, Mistral, and many other models require HuggingFace authentication. Set it as an environment variable before serving:

export HF_TOKEN=hf_your_token_here
openllm serve llama3.1:8b

2. Use the right model tag for your hardware. OpenLLM’s model repository includes multiple variants of each model with different quantization levels. A 70B model in FP16 requires 2x A100 80GB, but the same model in INT4 fits on a single A100:

# Check GPU requirements before serving
openllm model get llama3.3:70b
# Output shows: GPU: 80G x 2 (FP16), 80G x 1 (INT4)

# Serve the quantized variant
openllm serve llama3.3:70b --quantization int4

3. Configure tensor parallelism for multi-GPU setups. Large models require splitting across GPUs. OpenLLM auto-detects available GPUs but you can override:

# Explicit tensor parallelism
openllm serve deepseek:r1-671b --tensor-parallel-size 8

# Pipeline parallelism for very large models
openllm serve deepseek:r1-671b --tensor-parallel-size 4 --pipeline-parallel-size 4

4. Use the built-in chat UI for quick testing. The /chat endpoint provides a web-based interface that’s useful for manual testing and demos. It’s available automatically when the server is running — no additional setup needed.

5. Monitor with the built-in metrics endpoint. OpenLLM exposes Prometheus-compatible metrics at /metrics:

# Check server health
curl http://localhost:3000/health

# Get Prometheus metrics
curl http://localhost:3000/metrics | grep openllm

Use Cases

1. Self-hosted LLM API for internal tools. Replace OpenAI API calls in your internal applications with a self-hosted OpenLLM server. Use Llama 3.1 8B for code generation, Qwen 2.5 7B for document summarization, and Mistral 7B for classification — all behind the same OpenAI-compatible endpoint. Teams save 80-90% on API costs compared to GPT-4 for internal workloads.

2. Multi-model experimentation platform. Run multiple LLMs simultaneously on different ports or GPU sets. Use OpenLLM’s model repository to quickly switch between models for evaluation. Compare DeepSeek R1, Llama 3.3, and Qwen 2.5 on the same benchmark suite without changing your evaluation code — just change the model parameter in the OpenAI client.

3. Production LLM gateway with autoscaling. Deploy OpenLLM to BentoCloud with scale-to-zero for cost-effective production serving. The service spins down when idle and cold-starts in under 10 seconds. Use BentoCloud’s built-in observability (tokens/sec, TTFT, GPU utilization) to monitor performance and set autoscaling policies based on concurrent request count.

4. Custom fine-tuned model serving. Fine-tune a model using HuggingFace Transformers + PEFT/LoRA, package it as a BentoML Bento, and serve it through OpenLLM’s custom repository system. This gives you the same OpenAI-compatible API for your fine-tuned model as for the base models — no custom API code needed.

5. Edge deployment with Docker containers. Build a Docker image with bentoml containerize and deploy to edge locations. OpenLLM’s small model support (Gemma 3 3B, Phi-4 14B) makes it practical to run on single-GPU edge servers. The same Docker image works on-premises, in the cloud, or at the edge — no code changes required.

Cheat Sheet

Task	Command / Pattern
Install OpenLLM	pip install openllm
Serve a model	openllm serve <model>:<tag>
Interactive chat	openllm run <model>:<tag>
List available models	openllm model list
Get model details	openllm model get <model>:<tag>
Update model catalog	openllm repo update
Add custom repo	openllm repo add <name> <url>
Set tensor parallelism	--tensor-parallel-size <N>
Set quantization	--quantization fp8|int4|awq
Set max model length	--max-model-len <tokens>
Set GPU memory utilization	--gpu-memory-utilization <0.0-1.0>
Enable prefix caching	--enable-prefix-caching
Enable chunked prefill	--enable-chunked-prefill
Build a Bento	openllm build <model>:<tag>
Containerize to Docker	bentoml containerize <bento>:<ver> -t <image>
Deploy to BentoCloud	openllm deploy <model>:<tag> --env HF_TOKEN
Set scale-to-zero	bentoml deployment update <name> --scaling-min 0
Health check	curl http://localhost:3000/health
OpenAI-compatible chat	POST /v1/chat/completions
List models via API	GET /v1/models
Chat UI	http://localhost:3000/chat
Prometheus metrics	GET /metrics
Set HuggingFace token	export HF_TOKEN=<token>
Run with custom backend	--backend vllm|tensorrt-llm

Vibe Coding Projects

1. Multi-model chat playground. Build a web app that lets users switch between 3-4 OpenLLM-served models (Llama 3.1 8B, Qwen 2.5 7B, Mistral 7B, Gemma 3 3B) in real-time. Run each model on a separate OpenLLM server (different ports), and use the OpenAI Python client to route requests. Add a side-by-side comparison view that sends the same prompt to all models and displays responses in parallel. Deploy with Docker Compose orchestrating the four servers plus a React frontend.

2. RAG pipeline with OpenLLM backend. Build a retrieval-augmented generation system using ChromaDB for vector storage and OpenLLM for generation. Use sentence-transformers for embeddings and OpenLLM’s OpenAI-compatible API for the LLM. The key insight: because OpenLLM exposes a standard /v1/chat/completions endpoint, you can use LangChain or LlamaIndex directly — no custom integration code. Deploy the full stack (ChromaDB + OpenLLM + FastAPI orchestrator) with a single docker-compose up.

3. Fine-tuned model A/B testing platform. Fine-tune two variants of a small model (e.g., Phi-4 14B) on different datasets using PEFT/LoRA. Package each as a BentoML Bento and serve through OpenLLM’s custom repository. Build a FastAPI proxy that splits traffic 50/50 between the two variants and logs response quality scores. Use the logged data to determine which fine-tuning approach performs better before deploying to production.

Problems Solved Efficiently

Problem	OpenLLM Solution	Why It Works
LLM serving setup complexity	One-command openllm serve	Pre-configured model repository with optimal serving parameters
OpenAI API compatibility	/v1/chat/completions + /v1/models	Drop-in replacement for OpenAI SDK, LangChain, LlamaIndex
GPU memory fragmentation	vLLM PagedAttention	Manages KV cache like virtual memory pages, reducing waste from 60% to under 4%
Low throughput under concurrency	Continuous batching	Interleaves prefill and decode phases across requests
Multi-GPU model parallelism	Declarative --tensor-parallel-size	Automatic model sharding across GPUs
Model versioning and switching	Model repository system	Curated catalog with versioned model entries
Production deployment	Docker + Kubernetes + BentoCloud	Same Bento artifact deploys anywhere
Cost management	Scale-to-zero on BentoCloud	Pay only for active inference, no idle GPU cost
Observability	Built-in Prometheus metrics	Tokens/sec, TTFT, GPU utilization, request counts
Custom model serving	Custom model repositories	Package fine-tuned models as Bentos with same API

Architectural Tradeoffs

Gained:

• One-command serving. openllm serve llama3.2:1b is the fastest path from zero to a running LLM API. No Dockerfiles, no CUDA configuration, no model download scripts.
• OpenAI API compatibility. The /v1/chat/completions endpoint means any tool that works with OpenAI (LangChain, LlamaIndex, Continue.dev, Cursor) works with OpenLLM by changing the base_url.
• vLLM performance out of the box. PagedAttention, continuous batching, and tensor parallelism are pre-configured for each model. You get production-grade inference without tuning.
• Model repository abstraction. The curated catalog eliminates the “which model variant, which quantization, which backend” decision tree. Each model entry encodes the optimal configuration.
• Multi-target deployment. The same model serves locally, in Docker, on Kubernetes, or on BentoCloud. No code changes between environments.

Sacrificed:

• No built-in fine-tuning. OpenLLM is a serving platform, not a training framework. You must fine-tune models externally (HuggingFace Transformers, PEFT/LoRA) and then package them for serving. Earlier versions had a planned LLM.tuning() API, but it was deprioritized in favor of serving features.
• vLLM dependency. OpenLLM’s performance is tied to vLLM. If vLLM doesn’t support a model or a feature (e.g., certain quantization formats, model architectures), OpenLLM can’t serve it. You can’t swap in a different inference engine without forking.
• BentoML ecosystem lock-in. Custom models must be packaged as BentoML Bentos. You can’t point OpenLLM at an arbitrary HuggingFace model ID and have it work — the model must be in the repository or packaged as a Bento.
• Limited to LLM serving. Unlike BentoML (which serves any ML model type), OpenLLM is focused exclusively on large language models. You can’t serve a vision model, audio model, or embedding model through OpenLLM’s CLI — use BentoML directly for those.
• Smaller model catalog than Ollama. Ollama supports hundreds of community-contributed models. OpenLLM’s curated catalog is smaller (tens of models) but each entry is production-tested with optimal configuration.

The honest tradeoff: OpenLLM trades model breadth for production readiness. If you want to experiment with any model on any hardware, Ollama gives you more flexibility. If you need a production-grade LLM API with OpenAI compatibility, autoscaling, and observability, OpenLLM gets you there faster. The inflection point is around 10 concurrent users — below that, Ollama’s simplicity wins; above that, OpenLLM’s vLLM-backed performance and deployment options become necessary.

Course-Style Deep Dive

Under the Hood: The Model Repository System

OpenLLM’s model repository is a Git-based catalog hosted at github.com/bentoml/openllm-models. Each model entry is a YAML file that specifies:

# Example: llama3.2:1b model entry
name: llama3.2
tag: 1b
model_id: meta-llama/Llama-3.2-1B-Instruct
backend: vllm
requirements:
  gpu_memory: 12G
  gpu_count: 1
config:
  dtype: bfloat16
  max_model_len: 8192
  gpu_memory_utilization: 0.90
  tensor_parallel_size: 1
  enable_prefix_caching: true
  enable_chunked_prefill: true

When you run openllm serve llama3.2:1b, the CLI:

1. Fetches the model entry from the repository (or local cache).
2. Resolves the HuggingFace model ID (meta-llama/Llama-3.2-1B-Instruct).
3. Downloads model weights if not cached (uses HuggingFace snapshot_download).
4. Constructs a vLLM AsyncLLMEngine with the parameters from the config.
5. Wraps the engine in a BentoML service with OpenAI-compatible API endpoints.
6. Starts the Starlette ASGI server on port 3000.

The repository is versioned — openllm repo update pulls the latest model entries, giving you access to new models and updated configurations without upgrading OpenLLM itself.

Under the Hood: The vLLM Integration

OpenLLM uses vLLM as its default inference backend. The integration works through BentoML’s Runner abstraction:

# Simplified view of OpenLLM's vLLM integration
import vllm
from bentoml import Runner

class VLLMRunner(Runner):
    def __init__(self, model_id: str, config: dict):
        super().__init__(
            name="llm_runner",
            resources={"gpu": config.get("tensor_parallel_size", 1)},
        )
        self.model_id = model_id
        self.config = config

    async def async_init(self):
        self.engine = vllm.AsyncLLMEngine.from_engine_args(
            vllm.AsyncEngineArgs(
                model=self.model_id,
                tensor_parallel_size=self.config.get("tensor_parallel_size", 1),
                dtype=self.config.get("dtype", "bfloat16"),
                max_model_len=self.config.get("max_model_len", 4096),
                gpu_memory_utilization=self.config.get("gpu_memory_utilization", 0.90),
                enable_prefix_caching=self.config.get("enable_prefix_caching", False),
                enable_chunked_prefill=self.config.get("enable_chunked_prefill", False),
            )
        )

    async def generate(self, prompt: str, sampling_params: dict) -> str:
        params = vllm.SamplingParams(**sampling_params)
        result = await self.engine.generate(prompt, params)
        return result.outputs[0].text

The Runner abstraction gives OpenLLM process-level isolation — the vLLM engine runs in a separate process from the HTTP server. If the engine crashes (OOM, CUDA error), the HTTP server stays up and the Runner is restarted automatically by Circus.

Under the Hood: The OpenAI-Compatible API Layer

The API layer maps OpenAI’s schema to vLLM’s generation parameters:

OpenAI Parameter	vLLM Mapping
model	Validates against served model ID
messages	Tokenizes and formats with chat template
temperature	SamplingParams(temperature=...)
max_tokens	SamplingParams(max_tokens=...)
stream	Enables streaming via AsyncGenerator
stop	SamplingParams(stop=...)
top_p	SamplingParams(top_p=...)
frequency_penalty	SamplingParams(frequency_penalty=...)
presence_penalty	SamplingParams(presence_penalty=...)

The chat template is loaded from the HuggingFace tokenizer config (tokenizer.chat_template). If the model doesn’t have one, OpenLLM falls back to a default template. This is why the same model ID works across OpenAI SDK, LangChain, and LlamaIndex — the API contract is identical.

Advanced Pattern: Custom Model Repository with Fine-Tuned Models

# 1. Fine-tune a model externally
# (Using HuggingFace Transformers + PEFT/LoRA)
# Save to: ./my-fine-tuned-model/

# 2. Package as a BentoML Bento
cat > bentofile.yaml << 'EOF'
service: "service.py:MyLLM"
name: "my-fine-tuned-llm"
models:
  - "my-fine-tuned-model"
python:
  packages:
    - torch
    - transformers
    - vllm
EOF

# 3. Create the service file
cat > service.py << 'PYEOF'
from __future__ import annotations
import bentoml
import vllm

@bentoml.service(
    resources={"gpu": 1},
    traffic={"timeout": 120},
)
class MyLLM:
    def __init__(self):
        self.engine = vllm.AsyncLLMEngine.from_engine_args(
            vllm.AsyncEngineArgs(
                model="./my-fine-tuned-model",
                dtype="bfloat16",
                max_model_len=4096,
                gpu_memory_utilization=0.90,
            )
        )

    @bentoml.api
    async def generate(self, prompt: str, max_tokens: int = 256) -> str:
        params = vllm.SamplingParams(max_tokens=max_tokens)
        result = await self.engine.generate(prompt, params)
        return result.outputs[0].text
PYEOF

# 4. Build the Bento
bentoml build

# 5. Create a custom repository
mkdir -p my-repo/bentos
cp ~/bentoml/bentos/my-fine-tuned-llm/* my-repo/bentos/
cd my-repo && git init && git add . && git commit -m "Initial"

# 6. Register with OpenLLM
openllm repo add my-models /path/to/my-repo

# 7. Serve
openllm serve my-fine-tuned-llm:latest

Production Pattern: Kubernetes Deployment with GPU Autoscaling

# k8s-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: openllm-llama
  namespace: ai-infra
spec:
  replicas: 2
  selector:
    matchLabels:
      app: openllm-llama
  template:
    metadata:
      labels:
        app: openllm-llama
    spec:
      containers:
      - name: openllm
        image: my-registry/openllm-llama:latest
        ports:
        - containerPort: 3000
        env:
        - name: HF_TOKEN
          valueFrom:
            secretKeyRef:
              name: hf-secret
              key: token
        resources:
          requests:
            cpu: "4"
            memory: "16Gi"
            nvidia.com/gpu: 1
          limits:
            cpu: "8"
            memory: "32Gi"
            nvidia.com/gpu: 1
        livenessProbe:
          httpGet:
            path: /health
            port: 3000
          initialDelaySeconds: 120
          periodSeconds: 15
        readinessProbe:
          httpGet:
            path: /health
            port: 3000
          initialDelaySeconds: 60
          periodSeconds: 10
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: openllm-hpa
  namespace: ai-infra
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: openllm-llama
  minReplicas: 1
  maxReplicas: 5
  metrics:
  - type: Pods
    pods:
      metric:
        name: openllm_requests_inflight
      target:
        type: AverageValue
        averageValue: 8
---
apiVersion: v1
kind: Service
metadata:
  name: openllm-service
  namespace: ai-infra
spec:
  type: LoadBalancer
  selector:
    app: openllm-llama
  ports:
  - port: 80
    targetPort: 3000

Production Pattern: Observability Stack

# prometheus-config.yaml
scrape_configs:
  - job_name: 'openllm'
    scrape_interval: 10s
    static_configs:
      - targets: ['openllm-service.ai-infra:3000']
    metrics_path: /metrics

Key metrics to monitor:

Metric	Type	What It Tells You
vllm:num_requests_running	Gauge	Current concurrent requests
vllm:num_requests_waiting	Gauge	Queue depth (backpressure indicator)
vllm:gpu_cache_usage_perc	Gauge	KV cache utilization (OOM risk)
vllm:time_to_first_token_seconds	Histogram	P50/P95/P99 TTFT
vllm:time_per_output_token_seconds	Histogram	Decode speed
vllm:num_preemptions_total	Counter	Request preemption rate (overload signal)
openllm_request_duration_seconds	Histogram	End-to-end request latency

The Results

Metric	Before (FastAPI + raw Transformers)	After (OpenLLM)	Improvement
Time to first LLM API	2-3 days	2 minutes	1,000x faster
Concurrent users supported	1-2	256+	100x+ improvement
GPU memory efficiency	40-60%	95%+	2x better
Throughput (Llama 8B, 8 concurrent)	45 tok/s	285 tok/s	6x improvement
Throughput (Llama 8B, 128 concurrent)	N/A (OOM)	4,200 tok/s	Infinite improvement
Model switching time	15-30 min (rebuild + redeploy)	< 1 second	1,000x faster
Lines of serving code	300-500	0 (CLI)	Eliminated
Infrastructure complexity	Docker + CUDA + custom API	pip install openllm	10x reduction
Deployment to production	1-2 weeks	1-2 hours	50x faster
Cost per 100M tokens (self-hosted)	~$2,500 (A10G)	~$780 (A10G, continuous batching)	3x cheaper

What to Watch Out For

1. GPU memory planning is critical with continuous batching. vLLM’s PagedAttention reduces fragmentation, but continuous batching means the KV cache grows with the number of concurrent requests. A single long sequence (32K tokens) with 8 concurrent requests can consume 40GB+ of GPU memory for the KV cache alone. Set --gpu-memory-utilization 0.85 to leave headroom for cache growth.

Lesson learned: “We deployed Llama 3 70B on 2x A100 80GB with --gpu-memory-utilization 0.95. Under load with 32K context windows, the KV cache grew past the reserved headroom and vLLM started preempting requests. Throughput dropped from 1,800 tok/s to 400 tok/s as requests were constantly being evicted and re-queued. Dropping to 0.85 fixed it — we lost 5% of model memory but gained 3x stable throughput.” — ML Infrastructure Engineer at a legal AI startup

2. The model repository is curated, not exhaustive. OpenLLM supports tens of models, not hundreds. If you need a niche model (e.g., StarCoder2, CodeGemma, or a specific community fine-tune), you may need to package it as a custom Bento. Check the model list with openllm model list before committing to OpenLLM for your use case.

3. Cold start latency on BentoCloud. Scale-to-zero is cost-effective but adds cold start time. The first request after an idle period triggers model loading, which can take 30-60 seconds for a 70B model. For latency-sensitive applications, set --scaling-min 1 to keep one replica warm.

Lesson learned: “We set --scaling-min 0 on BentoCloud to save costs. Our CI/CD pipeline ran integration tests every hour, and the first test always timed out waiting for the model to load. We added a warm-up endpoint that sends a dummy request every 5 minutes, but that defeated the purpose of scale-to-zero. The fix was to set --scaling-min 1 and accept the small baseline cost.” — DevOps Engineer at a SaaS company

4. vLLM version matters. OpenLLM pins a specific vLLM version. If a new vLLM release adds a feature you need (e.g., a new quantization format, a new model architecture), you may need to wait for an OpenLLM release that updates the dependency. Check the pyproject.toml for the pinned version before planning your deployment.

5. The openllm run chat mode is single-user. The interactive CLI chat (openllm run llama3:8b) is designed for development and testing, not production. It starts a single-session chat that blocks the terminal. For multi-user access, use openllm serve and connect via the OpenAI SDK or the built-in chat UI.

6. Custom model repositories require public URLs. OpenLLM’s openllm repo add command currently supports only public repositories. If your fine-tuned model is in a private Git repository, you’ll need to either make it public or use BentoML’s direct Bento packaging workflow instead of the repository system.

Lesson learned: “We spent a day trying to add a private GitHub repo as an OpenLLM model repository. The command succeeded but the server couldn’t clone the repo at runtime because it didn’t have SSH keys. We ended up packaging the model as a Bento directly and using bentoml serve instead of openllm serve. The workflow was slightly different but the result was the same — OpenAI-compatible API with our fine-tuned model.” — ML Engineer at a healthcare AI company

---

Next in the Open-Source AI Tools Mastery series: XInference