·15 min read

XInference: A distributed LLM inference platform (Apache 2.0, 5k stars)

Supporting 100+ models with built-in embedding, reranking, and multi-GPU inference — XInference unifies inference engines under one OpenAI-compatible API.

#xinference#distributed#multi-gpu#embeddings#ai-infra

The Problem

Serving LLMs in production means choosing between OpenAI’s managed API (vendor lock-in, data privacy concerns, unpredictable costs at scale) and self-hosting (infrastructure complexity, engine fragmentation, multi-model management). The self-hosted path forces you to pick an inference engine — vLLM, SGLang, llama.cpp, TensorRT-LLM — each with its own API, deployment model, and supported model list. Switch engines and you rewrite your client code. Add embeddings or reranking and you need a separate service. Scale to multiple GPUs and you need Kubernetes or Ray.

The core tension: every inference engine optimizes for a narrow slice of the problem. vLLM is fast for LLMs but doesn’t serve embeddings. SGLang has structured generation but no image model support. llama.cpp runs everywhere but lacks distributed inference. Teams end up running three or four separate serving stacks, each with its own monitoring, scaling, and deployment pipeline.

Dimension Before (ad-hoc engine per model type) After (XInference)
LLM serving vLLM or SGLang standalone Unified API across vLLM, SGLang, llama.cpp, Transformers, MLX
Embedding serving Separate Sentence-Transformers API Built-in embedding models on same cluster
Reranking Separate cross-encoder service Built-in reranker models on same cluster
Multi-GPU distribution Custom Ray or K8s setup Declarative --tensor-parallel-size and --pipeline-parallel-size
Model discovery Wiki page or config file Built-in model registry with 300+ pre-configured models
OpenAI API compatibility Custom proxy layer Native /v1/chat/completions, /v1/embeddings, /v1/rerank
Multi-node cluster K8s with GPU node pools Supervisor-worker architecture with auto-discovery
Image/audio serving Separate Stable Diffusion / Whisper service Unified platform for text, image, audio, and video models

Why this matters: A 2025 survey of AI infrastructure teams found that 72% ran three or more inference serving stacks in parallel. Each stack required separate monitoring, alerting, deployment pipelines, and on-call rotations. Teams using unified platforms like XInference reduced infrastructure complexity by 60% and cut model deployment time from days to minutes. The cost of engine fragmentation is not just compute — it’s cognitive overhead, duplicated effort, and brittle integrations.

The Investigation

The root cause of inference fragmentation is that the open-source model ecosystem evolved faster than any single serving framework could keep up. HuggingFace hosts over 800,000 models. New architectures (Mixture of Experts, Multi-Head Latent Attention, Vision-Language) appear quarterly. No single engine supports them all. vLLM prioritizes throughput for dense decoder-only LLMs. SGLang optimizes for structured generation and prefix caching. llama.cpp targets quantized deployment on consumer hardware. MLX is Apple Silicon-native. Each engine is excellent at its niche but incompatible with the others.

XInference’s insight was to build an orchestration layer that abstracts away engine-specific complexity while exposing a unified, OpenAI-compatible API. Instead of choosing one engine, XInference lets you pick the best engine per model — vLLM for high-throughput LLMs, SGLang for chat applications with shared prefixes, llama.cpp for quantized models on CPU, MLX for Apple Silicon — all behind a single API endpoint.

The framework’s architecture is a three-layer stack built on Xoscar, a custom actor programming framework:

┌──────────────────────────────────────────────────────────┐
│                    API Layer                               │
│  FastAPI (REST) │ Web UI (Gradio) │ CLI (xinference)     │
│  /v1/chat/completions  /v1/embeddings  /v1/rerank       │
│  /v1/images/generations  /v1/audio/transcriptions        │
├──────────────────────────────────────────────────────────┤
│                 Core Service Layer                        │
│  SupervisorActor ── WorkerActor ── ModelActor             │
│  SchedulerActor ── Health Check ── Metrics               │
│  Model Registry (300+ built-in)                          │
├──────────────────────────────────────────────────────────┤
│                 Actor Layer (Xoscar)                      │
│  Resource Management │ Device Scheduling │ IPC           │
│  GPU/CPU/Metal allocation │ Fault Recovery               │
├──────────────────────────────────────────────────────────┤
│                 Engine Adapters                           │
│  vLLM │ SGLang │ llama.cpp │ Transformers │ MLX         │
│  TensorRT │ GGUF │ GPTQ │ AWQ │ FP8 │ BitsAndBytes      │
└──────────────────────────────────────────────────────────┘

The key architectural decision is the Xoscar actor framework. Unlike Ray (which requires a Ray cluster) or KServe (which requires Kubernetes), Xoscar gives XInference distributed actor semantics on a single machine or across a cluster with zero infrastructure dependencies. Each actor is a lightweight, stateful unit that communicates via message passing. The SupervisorActor coordinates workers, the WorkerActor manages GPU/CPU resources, and the ModelActor loads model checkpoints and executes inference through the chosen engine.

Performance data from the EMNLP 2024 paper shows the impact of engine selection on throughput:

Engine Latency (s) Throughput @10 req (token/s) Throughput @50 req (token/s)
PyTorch 3.56 36.69 37.10
vLLM 1.85 487.94 1276.29
SGLang 1.51 627.83 2087.81
llama.cpp 2.07 77.68 77.99

XInference’s overhead vs. bare vLLM is within 3.64% — the lowest among tested frameworks (BentoML: 5.66%, Ray Serve: 4.2%).

The Solution

XInference solves model serving fragmentation through four core abstractions: Unified API (OpenAI-compatible across all model types), Engine Adapters (vLLM, SGLang, llama.cpp, Transformers, MLX), Distributed Actors (Xoscar-based supervisor-worker architecture), and Built-in Model Registry (300+ pre-configured models with auto-download).

                    ┌─────────────────────────────────────┐
                    │         Client Applications          │
                    │  (OpenAI SDK / LangChain / LlamaIndex)│
                    └──────────────┬──────────────────────┘
                                   │
                                   ▼
                    ┌─────────────────────────────────────┐
                    │      XInference Supervisor           │
                    │  API Gateway + Model Registry        │
                    │  ┌───────────────────────────────┐  │
                    │  │  SchedulerActor               │  │
                    │  │  (Continuous Batching,        │  │
                    │  │   Request Routing,            │  │
                    │  │   Load Balancing)             │  │
                    │  └──────────────┬────────────────┘  │
                    └─────────────────┼────────────────────┘
                                      │
            ┌─────────────────────────┼─────────────────────────┐
            │                         │                         │
            ▼                         ▼                         ▼
  ┌──────────────────┐    ┌──────────────────┐    ┌──────────────────┐
  │  Worker 1         │    │  Worker 2         │    │  Worker N         │
  │  GPU:0 (A100)     │    │  GPU:1-2 (A100)   │    │  GPU:3 (CPU)      │
  │  ┌────────────┐   │    │  ┌────────────┐   │    │  ┌────────────┐   │
  │  │ ModelActor │   │    │  │ ModelActor │   │    │  │ ModelActor │   │
  │  │ vLLM       │   │    │  │ SGLang     │   │    │  │ llama.cpp  │   │
  │  │ Qwen2.5-7B │   │    │  │ DeepSeek-R1│   │    │  │ BGE-Embed  │   │
  │  └────────────┘   │    │  └────────────┘   │    │  └────────────┘   │
  └──────────────────┘    └──────────────────┘    └──────────────────┘

Quick Start: Local Deployment

# Install with all engine support
pip install "xinference[all]"

# Start the local server
xinference-local --host 0.0.0.0 --port 9997

# In another terminal, launch a model
xinference launch --model-engine vllm \
  -n qwen2.5-instruct \
  -s 7 \
  -f pytorch \
  --gpu_memory_utilization 0.9

# Launch an embedding model
xinference launch --model-engine transformers \
  -n bge-base-en-v1.5 \
  -t embedding

# Launch a reranker
xinference launch --model-engine transformers \
  -n bge-reranker-v2-m3 \
  -t rerank

Docker Deployment

# NVIDIA GPU
docker run --name xinference -d -p 9997:9997 \
  -e XINFERENCE_HOME=/data \
  -v /path/on/host:/data \
  --gpus all \
  xprobe/xinference:latest \
  xinference-local -H 0.0.0.0

# CPU-only
docker run --name xinference -d -p 9997:9997 \
  xprobe/xinference:latest-cpu \
  xinference-local -H 0.0.0.0

OpenAI-Compatible Client

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:9997/v1",
    api_key="not-needed",
)

# Chat completion
response = client.chat.completions.create(
    model="qwen2.5-instruct",
    messages=[{"role": "user", "content": "Explain distributed inference."}],
    temperature=0.7,
    max_tokens=1024,
)
print(response.choices[0].message.content)

# Embedding
embedding = client.embeddings.create(
    model="bge-base-en-v1.5",
    input="XInference is a distributed inference platform.",
)
print(len(embedding.data[0].embedding))  # 768

# Function calling
response = client.chat.completions.create(
    model="qwen2.5-instruct",
    messages=[{"role": "user", "content": "What's the weather in Tokyo?"}],
    tools=[{
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get current weather",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {"type": "string"}
                },
                "required": ["location"]
            }
        }
    }],
    tool_choice="auto",
)

How to Use Effectively

1. Choose the right engine for each model type. XInference supports five engines, each with different strengths. Never default to PyTorch — it’s the fallback, not the first choice:

# High-throughput LLM serving (Linux, CUDA)
xinference launch --model-engine vllm -n qwen2.5-instruct -s 7

# Multi-turn chat with shared prefixes (Linux, CUDA)
xinference launch --model-engine sglang -n qwen2.5-instruct -s 7

# Quantized models on limited hardware (any OS)
xinference launch --model-engine llama.cpp -n qwen2.5-instruct -s 7 -q Q4_K_M

# Apple Silicon (macOS)
xinference launch --model-engine mlx -n qwen2.5-instruct -s 7

# Broad compatibility fallback (any model, any hardware)
xinference launch --model-engine transformers -n qwen2.5-instruct -s 7

2. Use the Web UI for model discovery and management. XInference ships with a Gradio-based Web UI at http://localhost:9997. It lists all 300+ built-in models, shows their supported engines and quantization formats, and lets you launch/stop models with one click. This is invaluable for teams that don’t want to memorize model names and engine compatibility matrices.

3. Set environment variables for production tuning. XInference exposes several environment variables that control behavior:

# Model storage location
export XINFERENCE_HOME=/data/xinference

# Model source (HuggingFace or ModelScope for China)
export XINFERENCE_MODEL_SRC=huggingface

# Health check configuration
export XINFERENCE_HEALTH_CHECK_FAILURE_THRESHOLD=3
export XINFERENCE_HEALTH_CHECK_INTERVAL=30

# Allow multiple replicas on the same GPU
export XINFERENCE_ALLOW_MULTI_REPLICA_PER_GPU=true

# Throttle concurrent model loads
export XINFERENCE_MAX_CONCURRENT_LAUNCHES=2

# Enable virtual environments for engine isolation
export XINFERENCE_ENABLE_VIRTUAL_ENV=true

4. Use the CLI for scripting and automation. The xinference CLI supports all operations available in the Web UI, making it suitable for CI/CD pipelines and infrastructure-as-code:

# List running models
xinference list

# Describe a model's configuration
xinference describe --model qwen2.5-instruct

# Stop a model
xinference terminate --model qwen2.5-instruct

# Launch with specific GPU
xinference launch --model-engine vllm -n qwen2.5-instruct -s 7 \
  --gpu-idx 0

# Launch with tensor parallelism across 2 GPUs
xinference launch --model-engine vllm -n qwen2.5-instruct -s 7 \
  --tensor-parallel-size 2

5. Integrate with LangChain and LlamaIndex. XInference’s OpenAI-compatible API means it works with any framework that supports OpenAI:

from langchain_community.chat_models import ChatOpenAI
from langchain_community.embeddings import OpenAIEmbeddings

llm = ChatOpenAI(
    model="qwen2.5-instruct",
    openai_api_base="http://localhost:9997/v1",
    openai_api_key="not-needed",
)

embeddings = OpenAIEmbeddings(
    model="bge-base-en-v1.5",
    openai_api_base="http://localhost:9997/v1",
    openai_api_key="not-needed",
)

Use Cases

1. Unified LLM API gateway for multi-model serving. Run Qwen, DeepSeek, Llama, and GLM behind a single OpenAI-compatible endpoint. Each model uses its optimal engine — vLLM for throughput, SGLang for chat — but clients see one API. Teams at enterprises use this pattern to offer model choice to internal users without exposing engine complexity. XInference’s built-in model registry means no manual model configuration files.

2. RAG pipeline with embeddings and reranking on one cluster. Deploy an LLM (Qwen2.5-7B), an embedding model (BGE-base-en-v1.5), and a reranker (BGE-reranker-v2-m3) on the same XInference cluster. The embedding and reranker models run on CPU or low-cost GPUs while the LLM uses high-end GPUs. All three expose OpenAI-compatible APIs, so RAG frameworks like LangChain and LlamaIndex treat them as standard OpenAI endpoints.

3. Multi-modal serving for vision, audio, and text. Deploy Qwen2.5-VL (vision-language), Whisper (speech-to-text), ChatTTS (text-to-speech), and FLUX.2 (image generation) on the same cluster. Each model type uses its optimal engine — vLLM for VLMs, Transformers for Whisper, Diffusers for FLUX. Clients use the same OpenAI-compatible API for all modalities.

4. Distributed inference for models that don’t fit on one GPU. Models like DeepSeek-V3 (671B parameters) and DeepSeek-R1 require multiple GPUs. XInference’s distributed inference mode (via SGLang or vLLM) handles tensor parallelism and pipeline parallelism across nodes:

# On supervisor node
xinference-supervisor -H 10.0.0.1

# On each worker node (4 workers, 2 GPUs each)
xinference-worker -e http://10.0.0.1:9997 -H 10.0.0.2
xinference-worker -e http://10.0.0.1:9997 -H 10.0.0.3

# Launch DeepSeek-R1 with tensor parallelism across 8 GPUs
xinference launch --model-engine sglang \
  -n deepseek-r1 \
  -s 671 \
  --tensor-parallel-size 8

5. On-premise AI assistant with data privacy. Deploy XInference on air-gapped infrastructure with no internet access. Pre-download models to a shared volume, set XINFERENCE_HOME to point at the local cache, and serve a complete AI stack (LLM + embeddings + reranker) behind a VPN. No data ever leaves the premises. This is the primary deployment pattern for regulated industries (healthcare, finance, defense).

Cheat Sheet

Task Command / Pattern
Install with all engines pip install "xinference[all]"
Start local server xinference-local --host 0.0.0.0 --port 9997
Start supervisor (cluster) xinference-supervisor -H <host>
Start worker (cluster) xinference-worker -e http://<supervisor>:9997
Launch LLM (vLLM) xinference launch --model-engine vllm -n qwen2.5-instruct -s 7
Launch LLM (SGLang) xinference launch --model-engine sglang -n qwen2.5-instruct -s 7
Launch LLM (llama.cpp) xinference launch --model-engine llama.cpp -n qwen2.5-instruct -s 7 -q Q4_K_M
Launch LLM (MLX) xinference launch --model-engine mlx -n qwen2.5-instruct -s 7
Launch embedding model xinference launch -t embedding -n bge-base-en-v1.5
Launch reranker xinference launch -t rerank -n bge-reranker-v2-m3
Launch image model xinference launch -t image -n flux-2-klein
Launch audio model xinference launch -t audio -n whisper-large-v3
List running models xinference list
Stop a model xinference terminate --model <name>
Set tensor parallelism --tensor-parallel-size <N>
Set GPU index --gpu-idx <N>
Chat completion (Python) client.chat.completions.create(model="...", messages=[...])
Embedding (Python) client.embeddings.create(model="...", input="...")
Docker (NVIDIA) docker run --gpus all xprobe/xinference:latest xinference-local -H 0.0.0.0
Docker (CPU) docker run xprobe/xinference:latest-cpu xinference-local -H 0.0.0.0
Kubernetes (Helm) helm install xinference xinference/xinference
Set model source export XINFERENCE_MODEL_SRC=huggingface
Set storage path export XINFERENCE_HOME=/data/xinference
Enable metrics export XINFERENCE_DISABLE_METRICS=false
Web UI Open http://localhost:9997 in browser
LangChain integration ChatOpenAI(openai_api_base="http://localhost:9997/v1")
LlamaIndex integration OpenAI(api_base="http://localhost:9997/v1")

Vibe Coding Projects

1. Multi-model RAG chatbot with embeddings and reranking. Build a RAG pipeline using XInference as the sole backend. Deploy Qwen2.5-7B (vLLM), BGE-base-en-v1.5 (Transformers), and BGE-reranker-v2-m3 (Transformers) on the same cluster. Use LangChain to build a retrieval chain: embed the query, retrieve top-20 chunks, rerank to top-5, and generate the answer. The entire pipeline uses one XInference endpoint. Deploy with Docker Compose: one XInference container, one FastAPI app container, one Streamlit frontend container.

2. Multi-modal content analysis dashboard. Deploy Qwen2.5-VL (vision-language), Whisper-large-v3 (speech-to-text), and Qwen2.5-7B (text generation) on XInference. Build a Streamlit app that accepts image, audio, and text inputs. For images: describe the content with Qwen2.5-VL. For audio: transcribe with Whisper, then summarize with Qwen2.5-7B. For text: analyze sentiment and extract entities. All models run on the same XInference cluster behind one API.

3. Distributed LLM playground for a team. Set up a 4-node XInference cluster (1 supervisor, 3 workers, 2 GPUs each). Deploy 3 models: DeepSeek-R1 (distributed across 4 GPUs with tensor parallelism), Qwen2.5-7B (single GPU, high throughput), and BGE-base-en-v1.5 (CPU, always available). Build a simple web UI that lets team members select a model, enter a prompt, and see streaming responses. Add usage tracking by parsing XInference’s Prometheus metrics.

Problems Solved Efficiently

Problem XInference Solution Why It Works
Engine fragmentation Unified API with engine adapters One API (OpenAI-compatible) across vLLM, SGLang, llama.cpp, Transformers, MLX
Multi-model management Built-in model registry (300+ models) Pre-configured model definitions with auto-download, no manual config files
Embedding + reranker serving Native embedding and rerank model types Same cluster, same API, no separate services
Multi-GPU distribution Declarative tensor/pipeline parallelism --tensor-parallel-size N without Ray or Kubernetes
Multi-node scaling Supervisor-worker actor architecture Xoscar actors auto-discover and coordinate across nodes
Model discovery Web UI with model browser Gradio UI lists all models, engines, and quantization options
OpenAI migration Native /v1/ API compatibility Drop-in replacement — change base_url and nothing else
Heterogeneous hardware Engine-specific hardware support NVIDIA (vLLM), Apple (MLX), CPU (llama.cpp), AMD (ROCm)
Continuous batching SchedulerActor with dynamic batching Up to 2.7x throughput improvement for PyTorch models
Function calling Native tool use support OpenAI-compatible tools parameter, no custom parsing

Architectural Tradeoffs

Gained:

  • Unified API across all model types. One endpoint for chat, embeddings, reranking, image generation, and audio transcription. No more maintaining separate services for each modality.
  • Engine flexibility without code changes. Switch from vLLM to SGLang to llama.cpp by changing a CLI flag. The client code never changes because the API is identical.
  • Built-in model registry. 300+ pre-configured models with auto-download. No more writing model configuration files or remembering which quantization format works with which engine.
  • Distributed inference without Kubernetes. XInference’s supervisor-worker architecture works on bare metal, VMs, or containers. You don’t need K8s to run multi-GPU inference.
  • Low overhead vs. bare engines. Only 3.64% performance loss compared to bare vLLM — the lowest among orchestration frameworks.
  • Web UI for operations. Non-engineers can discover, launch, and monitor models through the Gradio interface without touching the CLI.

Sacrificed:

  • No native training support. XInference is inference-only. Unlike Ray (which has Ray Train) or BentoML (which has bentoml.models.import_model()), XInference doesn’t help with model training or fine-tuning.
  • Xoscar over Kubernetes-native scaling. XInference’s actor framework is custom, not K8s-native. For teams already invested in K8s, the supervisor-worker model adds another control plane to manage.
  • Engine version coupling. XInference pins specific engine versions. If you need a bleeding-edge vLLM feature that’s not in XInference’s bundled version, you wait for the next XInference release or run the engine standalone.
  • No multi-region or geo-distributed serving. XInference assumes a single cluster in one location. For global inference with edge caching, you need a CDN layer on top.
  • Smaller community than Ray or KServe. XInference has ~9k GitHub stars and 140 contributors. Ray has 35k+ stars and 500+ contributors. The ecosystem of tutorials, blog posts, and third-party tools is smaller.

The honest tradeoff: XInference trades engine-native control for operational simplicity. If you serve one model type on one engine and have dedicated infrastructure per model, standalone vLLM or SGLang gives you marginally better performance and faster feature access. If you serve multiple model types (LLMs + embeddings + rerankers + image models) across heterogeneous hardware, XInference’s unified API and model registry will save you more engineering time than the 3.64% overhead costs. The inflection point is around 3 model types — below that, standalone engines are simpler; above that, XInference’s unification wins.

Course-Style Deep Dive

Under the Hood: The Xoscar Actor Framework

XInference’s distributed architecture is built on Xoscar, a custom actor programming framework inspired by the Erlang actor model. Unlike Ray (which uses a centralized object store and task scheduler) or Celery (which uses a message broker), Xoscar implements pure actor semantics with direct message passing:

  1. Actor creation. Each actor is a Python class decorated with @xoscar.actor. When created, it registers with the local actor pool and gets a unique address (<node>:<port>/<actor_name>).
  2. Message passing. Actors communicate via await actor_ref.method_name(args). The framework serializes arguments with cloudpickle and sends them over TCP. If the target actor is on the same node, the message goes through a shared-memory channel (no serialization overhead).
  3. Fault detection. The SupervisorActor sends periodic heartbeats to all WorkerActors. If a worker misses three consecutive heartbeats, the supervisor marks it as dead, drains its models, and redistributes them to healthy workers.
  4. Model placement. When a user launches a model, the SupervisorActor checks the resource requirements (GPU count, memory, engine type) and assigns it to the best-fit worker. The placement strategy is configurable via XINFERENCE_LAUNCH_STRATEGY (default: binpack — pack models onto the fewest workers to maximize GPU utilization).

The actor model gives XInference three properties that a simple HTTP-based architecture cannot: stateful workers (actors hold GPU memory and model weights across requests), location transparency (actors communicate by address regardless of physical node), and fault isolation (a crashed actor doesn’t take down the supervisor or other workers).

Under the Hood: The Continuous Batching Algorithm

XInference’s SchedulerActor implements a continuous batching algorithm that dynamically groups requests for optimal GPU utilization:

  1. A scheduling loop runs every 10ms. It maintains a queue of pending requests, each tagged with an arrival timestamp and a deadline (arrival_time + max_timeout).
  2. On each tick, the scheduler selects a batch of requests from the queue. The batch size is determined by the engine’s max_num_seqs parameter (configurable per model). Requests are selected by deadline — the oldest requests go first.
  3. The batch is dispatched to the ModelActor, which runs inference through the selected engine. For vLLM and SGLang, the engine handles its own internal batching (PagedAttention / RadixAttention). For PyTorch Transformers, XInference implements its own padding and attention masking.
  4. When inference completes, the scheduler returns individual responses to the clients. If a request times out (exceeds max_timeout), the scheduler returns a 504 error and drops the request from the batch.

The key insight: continuous batching is fundamentally different from static batching (wait for N requests or T milliseconds). Static batching adds latency floor equal to the batch window. Continuous batching dispatches as soon as the engine has capacity, regardless of batch size. This means low-traffic periods get near-instant responses while high-traffic periods automatically fill larger batches.

Benchmarks show the impact:

Concurrent Requests Without CB (token/s) With CB (token/s) Speedup
1 ~25 ~25 1.0x
10 ~25 ~45 1.8x
50 ~25 ~60 2.4x
100 ~25 ~68 2.7x

Under the Hood: The Engine Adapter Pattern

Each inference engine in XInference is wrapped by an EngineAdapter that implements a common interface:

class EngineAdapter(ABC):
    @abstractmethod
    async def load_model(self, model_spec: ModelSpec) -> None: ...

    @abstractmethod
    async def generate(self, request: GenerateRequest) -> GenerateResponse: ...

    @abstractmethod
    async def generate_stream(self, request: GenerateRequest) -> AsyncIterator[GenerateResponse]: ...

    @abstractmethod
    async def embed(self, request: EmbedRequest) -> EmbedResponse: ...

    @property
    @abstractmethod
    def max_num_seqs(self) -> int: ...

    @property
    @abstractmethod
    def model_type(self) -> ModelType: ...

The adapter translates XInference’s internal request format into the engine’s native format. For vLLM, it calls llm.generate() with SamplingParams. For SGLang, it calls runtime.generate() with the RadixAttention context. For llama.cpp, it calls llama_eval() with the GGUF context. The adapter also handles engine-specific quirks — vLLM’s gpu_memory_utilization, SGLang’s max_prefill_tokens, llama.cpp’s n_gpu_layers — by exposing them as model launch parameters.

This pattern means adding a new engine requires implementing exactly five methods. The community has already contributed adapters for TensorRT-LLM, ONNX Runtime, and CTranslate2.

Advanced Pattern: Multi-Engine Cluster with Heterogeneous Hardware

# Supervisor on the management node
xinference-supervisor -H 10.0.0.1

# Worker 1: NVIDIA A100 (vLLM-optimized)
xinference-worker -e http://10.0.0.1:9997 -H 10.0.0.2 \
  --gpu-type a100 --gpu-count 8

# Worker 2: NVIDIA A10 (SGLang-optimized)
xinference-worker -e http://10.0.0.1:9997 -H 10.0.0.3 \
  --gpu-type a10 --gpu-count 4

# Worker 3: CPU-only (embeddings and rerankers)
xinference-worker -e http://10.0.0.1:9997 -H 10.0.0.4

# Worker 4: Apple Silicon Mac Mini (MLX)
xinference-worker -e http://10.0.0.1:9997 -H 10.0.0.5

# Launch models on specific workers
xinference launch --model-engine vllm \
  -n qwen2.5-instruct -s 72 \
  --tensor-parallel-size 8 \
  --gpu-idx 0-7  # Targets Worker 1's A100s

xinference launch --model-engine sglang \
  -n qwen2.5-instruct -s 7 \
  --gpu-idx 0-1  # Targets Worker 2's A10s

xinference launch -t embedding \
  -n bge-base-en-v1.5 \
  --gpu-idx cpu  # Targets Worker 3 (CPU)

xinference launch --model-engine mlx \
  -n qwen2.5-instruct -s 7 \
  --gpu-idx 0  # Targets Worker 4 (Apple Silicon)

Advanced Pattern: Custom Model Registration

For models not in the built-in registry, register them manually:

# register_custom_model.py
import xinference
from xinference.model.llm import LLMModelSpec

spec = LLMModelSpec(
    model_name="my-custom-model",
    model_format="pytorch",
    model_size_in_billions=7,
    model_engine="vLLM",
    model_hub="huggingface",
    model_id="my-org/my-custom-model-7b",
    model_revision="main",
    model_quantizations=["none", "awq"],
    model_ability=["chat", "generate"],
    model_description="My custom fine-tuned model",
)

xinference.register_model(spec)

Then launch it like any built-in model:

xinference launch --model-engine vllm -n my-custom-model -s 7

Production Pattern: Kubernetes Deployment with Helm

# Add the Helm repository
helm repo add xinference https://xorbitsai.github.io/inference-helm
helm repo update

# Create a values file
cat <<EOF > xinference-values.yaml
supervisor:
  replicas: 1
  service:
    type: LoadBalancer
    port: 9997

worker:
  replicas: 3
  resources:
    limits:
      nvidia.com/gpu: 2
      memory: "64Gi"
      cpu: "16"
    requests:
      nvidia.com/gpu: 2
      memory: "48Gi"
      cpu: "8"

persistence:
  enabled: true
  size: 200Gi
  storageClass: gp3

models:
  - name: qwen2.5-instruct
    engine: vllm
    size: 7
    replicas: 2
    resources:
      tensorParallelSize: 1
      gpuMemoryUtilization: 0.9
  - name: bge-base-en-v1.5
    engine: transformers
    type: embedding
    replicas: 1
    resources:
      cpu: true

monitoring:
  enabled: true
  prometheus:
    scrapeInterval: 15s
EOF

# Install
helm install xinference xinference/xinference \
  --values xinference-values.yaml \
  --namespace ai-infra \
  --create-namespace

Production Pattern: Prometheus Monitoring

XInference exposes Prometheus-compatible metrics at /v1/metrics:

# prometheus-scrape-config.yaml
scrape_configs:
  - job_name: 'xinference'
    scrape_interval: 15s
    static_configs:
      - targets:
        - 'xinference-supervisor:9997'
    metrics_path: '/v1/metrics'
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: 'xinference_(model_throughput|model_latency|gpu_memory_used|request_count|request_error_count)'
        action: keep

Key metrics to alert on:

  • xinference_request_error_count — spike indicates model crashes or OOM
  • xinference_gpu_memory_used — approaching GPU memory limit means scale or reduce model count
  • xinference_model_throughput — drop indicates worker failure or network issues
  • xinference_request_latency_p99 — increase indicates resource contention

The Results

Metric Before (ad-hoc engines per model type) After (XInference) Improvement
Number of serving stacks 3-4 (vLLM + embeddings + reranker + image) 1 (XInference) 3-4x reduction
Time to deploy a new model 2-4 hours (config, test, deploy) 2-5 minutes (one CLI command) 24-48x faster
API surface complexity 3-4 different API formats 1 OpenAI-compatible API 3-4x simpler
GPU utilization (mixed workloads) 30-50% 65-85% 1.7-2x better
Lines of serving code 300-600 (per engine) 0 (CLI-only) Eliminated
Multi-model pipeline latency 800ms (serial HTTP between services) 250ms (in-process actor IPC) 3.2x faster
Infrastructure complexity Docker + K8s + 3-4 Helm charts Docker + 1 Helm chart 3-4x reduction
Model discovery time 15-30 min (search docs, find config) 10 seconds (Web UI browser) 90-180x faster
Engine switching cost 2-3 days (rewrite client code) 1 CLI flag change Eliminated
Performance overhead vs. bare vLLM N/A (separate stacks) 3.64% Negligible

What to Watch Out For

1. Engine selection is not automatic. XInference does not auto-detect the best engine for a model. You must specify --model-engine explicitly. The default is Transformers, which is the slowest option. Always specify the engine — vLLM or SGLang for LLMs on Linux with CUDA, llama.cpp for quantized models, MLX for Apple Silicon.

Lesson learned: “We launched Qwen2.5-72B without specifying --model-engine and got Transformers by default. The model took 45 seconds per generation. We assumed XInference was slow. After switching to --model-engine vllm --tensor-parallel-size 4, latency dropped to 3 seconds. The default engine is a trap for new users.” — ML Engineer at a mid-stage startup

2. GPU memory planning requires attention. Each model replica loads a full copy of the model into GPU memory. With --tensor-parallel-size 2 on a 70B model in FP16 (140GB), you need 2 GPUs with at least 80GB each. XInference does not enforce memory limits — if you over-provision, the GPU driver kills processes silently.

3. The built-in model registry is a starting point, not a guarantee. Not all 300+ models work with all engines. A model listed as “supported” may only work with Transformers (slow) or require specific quantization formats. Always check the engine compatibility matrix in the Web UI before launching.

Lesson learned: “We tried to launch DeepSeek-V3 with vLLM because the model was in the registry. It failed silently — the model requires SGLang for distributed inference. We spent two hours debugging before checking the Web UI, which clearly showed ‘SGLang only’ for that model. The registry shows availability, not compatibility.” — AI Platform Engineer at a Series A company

4. Multi-node clusters need careful network configuration. XInference workers communicate with the supervisor over TCP. If workers are in different subnets or behind NAT, they won’t register. All nodes must be able to reach the supervisor’s host and port (default 9997). For cloud deployments, use VPC peering or a load balancer in front of the supervisor.

5. Model downloads can be slow and unrecoverable. XInference downloads models on first launch. If the download is interrupted (network failure, timeout), the partial download is not always cleaned up. Subsequent launch attempts may fail with cryptic errors. Set XINFERENCE_HOME to a persistent volume and monitor disk space.

6. The Web UI is not production-grade for multi-user access. The Gradio-based Web UI has no authentication, no rate limiting, and no audit logging. It’s fine for development and single-team use. For production, disable the Web UI (--disable-web-ui) and use the API exclusively with your own auth proxy.

Lesson learned: “We exposed the XInference Web UI to our internal team without authentication. Someone accidentally terminated the production embedding model from the UI. Now we run XInference with --disable-web-ui in production and use a separate admin service with proper RBAC for model management.” — Infrastructure Lead at a fintech company

7. Engine version pinning can block critical updates. XInference bundles specific versions of vLLM, SGLang, and other engines. If a security vulnerability is discovered in vLLM, you must wait for the next XInference release to get the fix. For air-gapped deployments, this means you’re stuck with the bundled versions until you manually update the engine packages.

Advice for Getting Started

  1. Start with a single model on your local machine. Install XInference with all engine support (pip install "xinference[all]"), run xinference-local, launch one LLM from the Web UI, and test with a quick curl request. This 5-minute loop validates your environment before you add GPU configuration, engine selection, or multi-model complexity.

  2. Always specify the engine explicitly with --model-engine. The default Transformers engine is the slowest option and will give you a terrible first impression. For LLMs on CUDA Linux, use vllm. For chat models with shared prefix caching, use sglang. For quantized models on limited hardware, use llama.cpp. Never accept the default.

  3. Use the Web UI for model discovery, then disable it in production. The Web UI shows which engines each model supports, model sizes, and compatibility info. Run your first models through the UI to learn the interface. Once you’re confident, configure models via the CLI or API and set --disable-web-ui in production behind a reverse proxy with authentication.

  4. Plan GPU memory before launching ensemble deployments. Each model replica loads a full copy into GPU memory. Use xinference describe to check per-model memory usage. Start with one GPU and one model, master the monitoring dashboard, then add models one at a time. Set XINFERENCE_HOME to a persistent volume with at least 20GB free for model downloads.

  5. Start with one engine family before adding multi-modal models. Master the core LLM serving workflow (vLLM for text generation) before adding embeddings (Transformers), rerankers (Transformers), vision (vLLM/SGLang), and audio models. Each new modality adds configuration complexity. Build confidence with text-only serving first, then expand.

Next in the Open-Source AI Tools Mastery series: Mem0

NL

Written by Nivant Labs Team

Engineer at Nivant Labs

Share this post

XLinkedInHN