Ray Serve: A scalable model serving library (Apache 2.0, 35k stars)

The Problem

Serving ML models in production is deceptively hard. The model itself is a single Python function — def predict(input) -> output — but turning that function into a production-grade service that handles 10,000 requests per second, autoscales under load, batches requests for GPU efficiency, and routes to the right model version requires a stack of infrastructure that most ML teams do not want to build.

The naive approach is to wrap the model in a FastAPI app and deploy it behind a load balancer. This works for one model at low traffic. But as soon as you have multiple models, variable traffic patterns, GPU memory constraints, and latency SLAs, the cracks appear:

• Single-process bottlenecks. A FastAPI server runs one Python process. Under load, the GIL becomes a bottleneck. You can fork workers, but now you are managing process pools, shared state, and graceful shutdown yourself.
• No autoscaling. You set num_workers=4 and hope it is enough. When traffic spikes, requests queue up. When traffic drops, you pay for idle workers. There is no dynamic scaling.
• No request batching. GPU inference is most efficient when processing batches of inputs. But a standard HTTP server processes requests one at a time. Building a request batcher that collects requests over a time window and sends them as a batch requires custom infrastructure.
• No multi-model routing. Serving 10 different models means 10 different deployments, 10 different URLs, 10 different autoscaling configurations. There is no unified routing layer.
• No distributed inference. Models that do not fit on one GPU cannot be served at all. You are limited to whatever fits in a single GPU’s VRAM.

Dimension	Naive FastAPI + Gunicorn	Ray Serve	Improvement
Autoscaling	Manual (fixed workers)	Per-deployment, request-driven	Eliminates over/under-provisioning
Request batching	Custom implementation	Built-in @serve.batch decorator	2-10 lines vs 200+ lines
Multi-model routing	Nginx + multiple services	Single endpoint, prefix-based	10x fewer moving parts
GPU utilization	Single request per GPU	Dynamic batching + multiplexing	2-5x throughput improvement
Distributed inference	Not supported	Built-in (Ray actors across nodes)	Enables models larger than 1 GPU
Model composition	Custom pipeline code	DeploymentHandle chaining	Declarative DAGs
Fault tolerance	Process restart	Actor reconstruction + rolling updates	Production-grade resilience
Observability	Custom metrics	Ray Dashboard + Prometheus	Built-in dashboards

Why this matters: The gap between a working model and a production serving system is a chasm of infrastructure work — autoscaling, batching, routing, fault tolerance, monitoring. Ray Serve closes that gap by providing a Python-native serving framework that handles the infrastructure so you can focus on the model logic. It is not a replacement for Triton Inference Server’s GPU-optimized inference — it is a higher-level orchestration layer that can embed Triton inside its replicas for the best of both worlds.

The Investigation

Ray Serve was born from a simple observation at the Ray project (UC Berkeley RISELab, later Anyscale): every ML team building a production serving system was solving the same problems independently, and none of them were getting it right.

Finding 1: Model serving is a distributed systems problem, not a web framework problem.

FastAPI, Flask, and Django are excellent web frameworks. But they are single-process, synchronous-by-default, and designed for request-response patterns where each request takes milliseconds. ML inference requests take 100ms to 30 seconds. The assumptions that web frameworks make about request duration, memory usage, and concurrency break down under ML workloads.

Ray’s investigation found that teams using FastAPI for model serving hit the same wall at around 500 QPS: the GIL, process management overhead, and lack of dynamic batching created a latency cliff where p99 latency jumped from 200ms to 5 seconds with a 2x traffic increase.

What this means: Model serving needs a distributed runtime that can spread requests across processes and machines, batch them for GPU efficiency, and scale replicas independently per model. Ray, with its distributed actor model and object store, was already that runtime. Ray Serve is the serving layer on top.

Finding 2: Request batching is the single highest-leverage optimization for GPU serving.

GPU inference is throughput-optimized: processing a batch of 16 inputs takes roughly the same time as processing 1 input (the GPU’s matrix units are saturated either way). The throughput gain is 5-16x depending on model size and batch size. But HTTP servers process requests one at a time, so this gain is unrealized unless you build a request collector.

Ray Serve’s investigation found that teams implementing custom request batching consistently made the same mistakes: race conditions in the collector, memory leaks from unbounded queues, and incorrect timeout handling that caused cascading failures under load. The @serve.batch decorator was designed to encode the correct batching pattern once, test it thoroughly, and make it a one-line addition to any deployment.

What this means: Request batching is not optional for GPU serving — it is the difference between 50% GPU utilization and 90%+ GPU utilization. Ray Serve makes it a first-class primitive rather than a custom hack.

Finding 3: Autoscaling for ML workloads is fundamentally different from web workloads.

Web server autoscaling is driven by CPU utilization or request rate. ML serving autoscaling needs to account for GPU memory, model loading time (seconds to minutes), and the cost of cold starts. A web server can start a new instance in 100ms. Loading a 7B model onto a GPU takes 5-15 seconds. The autoscaling algorithm must be conservative on scale-down (to avoid thrashing) and aggressive on scale-up (to absorb traffic spikes before latency degrades).

Ray Serve’s autoscaling uses target_num_ongoing_requests_per_replica as the primary signal. This metric directly reflects how busy each replica is, independent of request duration or model size. A replica processing a 30-second LLM generation is “busy” with 1 request. A replica processing 10ms embeddings is “busy” with 50 requests. The autoscaling controller adjusts replica count to keep each replica’s ongoing requests near the target.

What this means: Request-based autoscaling is the correct primitive for ML workloads. CPU-based autoscaling would scale up too late (by the time CPU is high, latency is already degraded) and scale down too early (a replica with a loaded model is valuable even if idle).

The Solution

Ray Serve is a ~150,000-line Python library (Apache 2.0, 35,000+ GitHub stars, 6,000+ forks, 1,200+ contributors) that provides a distributed model serving framework built on Ray’s actor runtime. It handles request routing, autoscaling, batching, fault tolerance, and model composition through a Python-native API.

┌──────────────────────────────────────────────────────────────────────────────┐
│                          Ray Serve Architecture                                │
│                                                                                │
│  ┌────────────────────────────────────────────────────────────────────────┐  │
│  │                          HTTP Proxy Layer                                │  │
│  │  ┌──────────────────┐  ┌──────────────────┐  ┌──────────────────────┐  │  │
│  │  │ HAProxy Ingress  │  │ Ray Serve Proxy  │  │ gRPC Proxy           │  │  │
│  │  │ (C, direct       │  │ (Python, per-    │  │ (Envoy sidecar,      │  │  │
│  │  │  streaming mode) │  │  node routing)   │  │  ~60% less overhead) │  │  │
│  │  └──────────────────┘  └──────────────────┘  └──────────────────────┘  │  │
│  └────────────────────────────────────────────────────────────────────────┘  │
│                                    │                                           │
│  ┌────────────────────────────────────────────────────────────────────────┐  │
│  │                      Router & Controller                                 │  │
│  │  ┌──────────────────────┐  ┌──────────────────┐  ┌──────────────────┐  │  │
│  │  │ Request Router        │  │ Autoscaler        │  │ Health Monitor   │  │  │
│  │  │ (prefix-based,        │  │ (per-deployment,  │  │ (actor health,   │  │  │
│  │  │  session-aware)       │  │  request-driven)  │  │  replica status) │  │  │
│  │  └──────────────────────┘  └──────────────────┘  └──────────────────┘  │  │
│  └────────────────────────────────────────────────────────────────────────┘  │
│                                    │                                           │
│  ┌────────────────────────────────────────────────────────────────────────┐  │
│  │                     Deployment Replicas (Ray Actors)                    │  │
│  │                                                                        │  │
│  │  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐  ┌────────────┐ │  │
│  │  │ Replica 1    │  │ Replica 2    │  │ Replica 3    │  │ Replica N  │ │  │
│  │  │ ┌──────────┐ │  │ ┌──────────┐ │  │ ┌──────────┐ │  │ ┌────────┐ │ │  │
│  │  │ │ Model A  │ │  │ │ Model A  │ │  │ │ Model B  │ │  │ │ Model C│ │ │  │
│  │  │ │ (batch)  │ │  │ │ (batch)  │ │  │ │ (batch)  │ │  │ │ (batch)│ │ │  │
│  │  │ └──────────┘ │  │ └──────────┘ │  │ └──────────┘ │  │ └────────┘ │ │  │
│  │  └──────────────┘  └──────────────┘  └──────────────┘  └────────────┘ │  │
│  └────────────────────────────────────────────────────────────────────────┘  │
│                                    │                                           │
│  ┌────────────────────────────────────────────────────────────────────────┐  │
│  │                       Ray Distributed Runtime                           │  │
│  │  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐  ┌────────────┐ │  │
│  │  │ Object Store │  │ Scheduler   │  │ GCS (Global  │  │ Dashboard  │ │  │
│  │  │ (shared      │  │ (distributed│  │  Control     │  │ (metrics,  │ │  │
│  │  │  memory)     │  │  task/actor)│  │  Store)      │  │  traces)   │ │  │
│  │  └──────────────┘  └──────────────┘  └──────────────┘  └────────────┘ │  │
│  └────────────────────────────────────────────────────────────────────────┘  │
│                                                                                │
│  ┌────────────────────────────────────────────────────────────────────────┐  │
│  │                     Cluster Nodes                                       │  │
│  │  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐  ┌────────────┐ │  │
│  │  │ Head Node    │  │ Worker Node 1│  │ Worker Node 2│  │ Worker N   │ │  │
│  │  │ (controller) │  │ (GPU: A100)  │  │ (GPU: A100)  │  │ (CPU only) │ │  │
│  │  └──────────────┘  └──────────────┘  └──────────────┘  └────────────┘ │  │
│  └────────────────────────────────────────────────────────────────────────┘  │
└──────────────────────────────────────────────────────────────────────────────┘

Here is what each piece does:

• HTTP Proxy Layer: Handles ingress traffic. HAProxy (C-based, Ray 2.55+) provides direct streaming mode where tokens flow directly from model replicas to clients, bypassing the Python proxy. The Python proxy handles routing decisions and prefix-based dispatch. The gRPC proxy (Envoy sidecar) reduces serialization overhead by ~60% compared to JSON for internal communication.

• Router & Controller: The request router maps incoming requests to the correct deployment based on route prefix. The autoscaler monitors target_num_ongoing_requests_per_replica and adjusts replica count. The health monitor tracks actor health and triggers reconstruction on failure.

• Deployment Replicas (Ray Actors): Each replica is a Ray actor running your model logic. Replicas can be distributed across nodes, each with its own GPU allocation. The @serve.batch decorator collects requests into batches for GPU-efficient inference. Each deployment can have its own autoscaling config, resource requirements, and batching parameters.

• Ray Distributed Runtime: The underlying Ray runtime provides the object store (shared memory for zero-copy data transfer), distributed scheduler (task/actor placement), GCS (global control store for cluster state), and dashboard (metrics, traces, logs).

• Cluster Nodes: A Ray cluster consists of a head node (runs the controller and GCS) and worker nodes (run deployment replicas). Worker nodes can be GPU-equipped for model inference or CPU-only for preprocessing/postprocessing.

Setup

# Install Ray Serve
pip install "ray[serve]"

# For LLM serving with all optimizations (Ray 2.56+)
# Use the recommended container image
docker pull rayproject/ray-llm:2.56-py312-cu130

# Verify installation
python -c "import ray; print(ray.__version__)"

Production-Grade Configuration

# config.py — Ray Serve application configuration
from ray import serve
from ray.serve.config import AutoscalingConfig

@serve.deployment(
    num_replicas=1,
    autoscaling_config=AutoscalingConfig(
        min_replicas=1,
        max_replicas=10,
        target_num_ongoing_requests_per_replica=2.0,
        downscale_delay_s=300,       # 5 min cooldown to prevent thrashing
        upscale_delay_s=5,           # Fast scale-up on traffic spikes
    ),
    max_ongoing_requests=10,         # Concurrent requests per replica
    max_queued_requests=50,           # Queue depth before 503 rejection
    ray_actor_options={
        "num_cpus": 2,
        "num_gpus": 1,
    },
    health_check_period_s=10,
    health_check_timeout_s=30,
    graceful_shutdown_timeout_s=20,
)
class MyModel:
    pass

Code Walkthrough: The Core Serving Loop

The heart of Ray Serve is the deployment replica’s request processing pipeline. Here is the simplified flow for a prediction request:

# Simplified from ray/serve/replica.py
class Replica:
    def __init__(self, deployment_def, deployment_config):
        self.deployment_def = deployment_def
        self.config = deployment_config
        self.app = self.deployment_def()
        self.request_counter = 0
        self.ongoing_requests = 0

    async def handle_request(self, request):
        self.request_counter += 1
        self.ongoing_requests += 1

        try:
            # 1. Check queue depth — reject if overloaded
            if self.ongoing_requests > self.config.max_ongoing_requests:
                return HTTPResponse(status=503, body="Service Unavailable")

            # 2. If batching is enabled, enqueue and wait for batch
            if hasattr(self.app, "__serve_batch__"):
                result = await self._batch_enqueue(request)
            else:
                # 3. Direct invocation
                result = await self.app(request)

            return result

        except Exception as e:
            # 4. Error handling with structured logging
            logger.error(f"Request failed: {e}", request_id=request.metadata.request_id)
            return HTTPResponse(status=500, body=str(e))

        finally:
            self.ongoing_requests -= 1

    async def _batch_enqueue(self, request):
        # Collect requests into a batch window
        batch = await self.batch_collector.wait_for_batch(
            timeout_s=self.config.batch_timeout_s,
            max_batch_size=self.config.max_batch_size,
        )
        # Run inference on the batch
        results = await self.app(batch)
        return results[request.position_in_batch]

The autoscaler is the most architecturally interesting piece:

# Simplified from ray/serve/autoscaler.py
class Autoscaler:
    def __init__(self, deployment_name, config):
        self.deployment_name = deployment_name
        self.config = config
        self.current_replicas = config.min_replicas
        self.last_scale_down_time = time.time()
        self.last_scale_up_time = time.time()

    async def run_autoscaling_loop(self):
        while True:
            # 1. Collect metrics from all replicas
            metrics = await self._collect_replica_metrics()

            # 2. Calculate average ongoing requests per replica
            total_ongoing = sum(m.ongoing_requests for m in metrics)
            avg_ongoing = total_ongoing / max(self.current_replicas, 1)

            # 3. Determine desired replica count
            target = self.config.target_num_ongoing_requests_per_replica
            desired_replicas = max(
                self.config.min_replicas,
                min(
                    self.config.max_replicas,
                    int(total_ongoing / target) + 1,
                )
            )

            # 4. Apply scale-up (fast) or scale-down (slow)
            now = time.time()
            if desired_replicas > self.current_replicas:
                if now - self.last_scale_up_time >= self.config.upscale_delay_s:
                    await self._scale_to(desired_replicas)
                    self.last_scale_up_time = now
            elif desired_replicas < self.current_replicas:
                if now - self.last_scale_down_time >= self.config.downscale_delay_s:
                    await self._scale_to(desired_replicas)
                    self.last_scale_down_time = now

            await asyncio.sleep(1)  # 1-second control loop

How to Use Effectively

Step 1: Define a basic deployment

# basic_deployment.py
from ray import serve
import ray

ray.init(address="auto")  # Connect to Ray cluster

@serve.deployment(
    num_replicas=2,
    ray_actor_options={"num_cpus": 2, "num_gpus": 1},
)
class TextClassifier:
    def __init__(self):
        import torch
        from transformers import pipeline
        # Model loads once per replica
        self.classifier = pipeline(
            "text-classification",
            model="distilbert-base-uncased-finetuned-sst-2-english",
            device=0,  # GPU
        )

    async def __call__(self, request):
        text = (await request.json())["text"]
        result = self.classifier(text)
        return {"result": result}

# Deploy
serve.run(TextClassifier.bind())

Step 2: Add request batching for GPU efficiency

# batched_deployment.py
from ray import serve
from typing import List

@serve.deployment(
    num_replicas=1,
    max_ongoing_requests=100,  # Allow many concurrent requests
    ray_actor_options={"num_gpus": 1},
)
class BatchedEmbedder:
    def __init__(self):
        from sentence_transformers import SentenceTransformer
        self.model = SentenceTransformer("all-MiniLM-L6-v2")

    @serve.batch(max_batch_size=32, batch_wait_timeout_s=0.1)
    async def embed_batch(self, texts: List[str]) -> List[List[float]]:
        # This runs once per batch, not once per request
        embeddings = self.model.encode(texts, convert_to_numpy=True)
        return embeddings.tolist()

    async def __call__(self, request):
        text = (await request.json())["text"]
        # Single request — gets batched automatically
        embedding = await self.embed_batch(text)
        return {"embedding": embedding}

The @serve.batch decorator collects requests over a 100ms window (or until 32 requests accumulate), then runs inference once on the batch. The throughput gain is 5-10x for embedding models.

Step 3: Compose multiple models into a pipeline

# pipeline.py
from ray import serve
import ray

@serve.deployment
class Preprocessor:
    async def __call__(self, text: str) -> str:
        # Clean and normalize input
        return text.strip().lower()

@serve.deployment(
    ray_actor_options={"num_gpus": 1},
)
class Translator:
    def __init__(self):
        from transformers import pipeline
        self.model = pipeline("translation", model="t5-small")

    async def __call__(self, text: str) -> str:
        return self.model(text)[0]["translation_text"]

@serve.deployment(
    ray_actor_options={"num_gpus": 1},
)
class Classifier:
    def __init__(self):
        from transformers import pipeline
        self.model = pipeline("text-classification")

    async def __call__(self, text: str) -> dict:
        return self.model(text)[0]

@serve.deployment
class Postprocessor:
    async def __call__(self, result: dict) -> dict:
        return {
            "label": result["label"],
            "confidence": result["score"],
            "timestamp": time.time(),
        }

# Build the DAG
@serve.deployment
class Pipeline:
    def __init__(self, preprocessor, translator, classifier, postprocessor):
        self.preprocessor = preprocessor
        self.translator = translator
        self.classifier = classifier
        self.postprocessor = postprocessor

    async def __call__(self, request):
        text = (await request.json())["text"]
        cleaned = await self.preprocessor.remote(text)
        translated = await self.translator.remote(cleaned)
        classified = await self.classifier.remote(translated)
        result = await self.postprocessor.remote(classified)
        return result

app = Pipeline.bind(
    Preprocessor.bind(),
    Translator.bind(),
    Classifier.bind(),
    Postprocessor.bind(),
)
serve.run(app)

Each deployment in the pipeline can have its own autoscaling config, resource requirements, and batching parameters. The DeploymentHandle.remote() calls are asynchronous and non-blocking — they return object references that are resolved when the result is needed.

Step 4: Deploy with Kubernetes (KubeRay)

# ray-service.yaml
apiVersion: ray.io/v1
kind: RayService
metadata:
  name: model-serving
spec:
  rayVersion: "2.56.0"
  serveConfigV2: |
    proxy_location: EveryNode
    http_options:
      host: "0.0.0.0"
      port: 8000
    applications:
      - name: text-classifier
        route_prefix: /classify
        import_path: app:classifier_app
        deployments:
          - name: TextClassifier
            num_replicas: 2
            ray_actor_options:
              num_cpus: 2
              num_gpus: 1
            autoscaling_config:
              min_replicas: 1
              max_replicas: 10
              target_num_ongoing_requests_per_replica: 2.0
              downscale_delay_s: 300
      - name: embedder
        route_prefix: /embed
        import_path: app:embedder_app
        deployments:
          - name: BatchedEmbedder
            num_replicas: 1
            ray_actor_options:
              num_cpus: 2
              num_gpus: 1
            autoscaling_config:
              min_replicas: 1
              max_replicas: 5
              target_num_ongoing_requests_per_replica: 5.0
  service:
    type: ClusterIP
    ports:
      - port: 8000
        targetPort: 8000

Production pitfall: The downscale_delay_s default is 600 seconds (10 minutes). This is too aggressive for GPU workloads where model loading takes 5-15 seconds. Set it to at least 300 seconds to prevent thrashing. A replica that just loaded a model onto GPU should not be torn down because traffic dipped for 30 seconds.

Use Cases

1. Multi-Model Serving Platform

When you’d use this: You have 20+ models (classification, NER, summarization, embedding) that need to be served from a single endpoint with independent autoscaling per model.

Why Ray Serve fits: Each model gets its own deployment with its own autoscaling config, GPU requirements, and batching parameters. The prefix-based router dispatches requests to the correct model. Models that are heavily used (embeddings) get more replicas. Models that are rarely used (experimental classifiers) get 1 replica and scale to 0 when idle. Real-world example: a content moderation platform serves 15 models across 3 GPU nodes, handling 50,000 requests/minute with 99.9% uptime. Each model scales independently based on traffic patterns.

2. LLM Serving with Direct Streaming

When you’d use this: You need to serve LLMs (Llama, Mistral, Qwen) with streaming responses, high throughput, and low time-per-output-token (TPOT).

Why Ray Serve fits: Ray 2.55+ introduced direct streaming mode where HAProxy handles ingress and establishes direct HTTP connections between clients and model replicas. Tokens stream directly from the vLLM replica to the client, bypassing the Python proxy. Combined with vLLM’s Ray executor backend (enabled by default in vLLM 0.21.0+), this delivers 4.4x higher throughput for prefill-heavy workloads and 24.8x higher throughput for decode-heavy workloads compared to previous Ray Serve versions. Real-world example: a chatbot platform serves 8 LLM variants across 16 A100 GPUs, handling 10,000 concurrent users with p50 TTFT of 355ms and p95 TPOT of 45ms.

3. Real-Time Feature Engineering Pipeline

When you’d use this: You need to run feature extraction, embedding generation, and model inference as a single request pipeline with different scaling requirements per stage.

Why Ray Serve fits: Each pipeline stage is a separate deployment with its own resource requirements. The preprocessing stage (CPU-only, high throughput) gets 10 replicas with no GPU. The embedding stage (GPU, batchable) gets 3 replicas with batching. The classifier stage (GPU, low throughput) gets 2 replicas. The DeploymentHandle DAG connects them with async calls. Real-world example: a recommendation system processes 100,000 requests/minute through a 4-stage pipeline (tokenize -> embed -> rank -> filter), with each stage autoscaling independently based on its own traffic pattern.

4. A/B Testing and Canary Deployments

When you’d use this: You need to route a percentage of traffic to a new model version while monitoring performance before full rollout.

Why Ray Serve fits: Ray Serve supports weighted routing across deployments. You can deploy classifier-v1 and classifier-v2 under the same route prefix with a 90/10 traffic split. The built-in metrics (latency, error rate, request count) let you compare performance before shifting traffic. Real-world example: a fraud detection team deploys a new model version with 5% traffic for 24 hours, monitors false positive rate and p99 latency, then shifts to 100% after validation.

5. Multi-Tenant Model Serving

When you’d use this: You need to serve models for multiple customers with resource isolation, per-tenant rate limiting, and independent scaling.

Why Ray Serve fits: Each tenant gets their own deployment with dedicated GPU resources and autoscaling config. The max_queued_requests setting provides backpressure per tenant — one tenant’s traffic spike does not queue-bomb another tenant’s requests. Session-aware routing (Ray 2.56+, RFC #62645) ensures multi-turn conversations stay on the same replica for KV cache reuse. Real-world example: an AI API provider serves 50 enterprise customers from a shared Ray cluster, with each customer getting guaranteed GPU allocation and independent autoscaling, handling 500,000 requests/day with per-tenant latency SLAs.

Cheat Sheet

Aspect	Detail
Repository	github.com/ray-project/ray
License	Apache 2.0
Language	Python (~150,000 lines) + C++ (Ray core) + C (HAProxy integration)
Latest Version	Ray 2.56 (June 2026)
GPU Requirements	Optional; CUDA 11.0+ for NVIDIA GPUs
Setup Time	10 minutes (pip install + cluster setup)
Key Features	Autoscaling, request batching, multi-model routing, model composition, direct streaming, session-aware routing, gRPC proxy, HAProxy integration, fault tolerance, rolling updates, Prometheus metrics
Common Gotchas	Object store OOM is the #1 production issue; downscale delay too short causes thrashing; max_ongoing_requests too high causes latency degradation; model loading time ignored in autoscaling decisions
Best Models	Any Python model (PyTorch, TensorFlow, Hugging Face, vLLM, Triton)
Cost (Light)	$0 (single node, development)
Cost (Heavy)	$500-5,000/month (multi-node GPU cluster on Kubernetes)
Cost (Cloud)	$0.50-2.00/hour per A100 GPU (AWS/GCP/Azure)
Missing Features	No built-in model registry (use MLflow); no built-in auth (use reverse proxy); no built-in model versioning (use Ray Serve’s weighted routing); no built-in data validation (use Great Expectations)

Vibe Coding Projects

Project 1: Multi-Model Sentiment Analysis API

What it does: A Ray Serve application that serves three sentiment analysis models (DistilBERT, RoBERTa, and a custom fine-tuned model) from a single endpoint with prefix-based routing. Each model has its own autoscaling config and GPU allocation. Includes a health check endpoint and Prometheus metrics.

What you’ll learn: How to define multiple deployments in a single Ray Serve application. How to configure per-deployment autoscaling. How to set up prefix-based routing. How to expose custom metrics. How to handle model loading errors gracefully.

Effort: 2-3 hours. Runs on a single GPU or CPU.

Project 2: Real-Time Translation Pipeline with Batching

What it does: A 3-stage Ray Serve pipeline: language detection (fast, CPU) -> translation (GPU, batched) -> post-processing (CPU). The translation stage uses @serve.batch to collect requests over a 200ms window and translate them as a batch. Includes latency histograms per stage and a dashboard showing pipeline throughput.

What you’ll learn: How to compose deployments into a DAG with DeploymentHandle. How to configure @serve.batch for optimal throughput. How to set different resource requirements per pipeline stage. How to monitor per-stage latency.

Effort: 4-5 hours. Requires 1 GPU for the translation model.

Project 3: A/B Testing Framework for Model Deployments

What it does: A Ray Serve application that deploys two versions of a classifier under the same route prefix with a configurable traffic split (90/10, 50/50, etc.). Collects per-version metrics (latency, error rate, prediction distribution) and exposes them via a REST API. Includes a dashboard showing the A/B test results in real time.

What you’ll learn: How to use Ray Serve’s weighted routing for canary deployments. How to collect and expose custom metrics per deployment version. How to implement traffic splitting at the application level. How to automate the rollout process based on metric thresholds.

Effort: 3-4 hours. Runs on a single GPU or CPU.

Problems Solved Efficiently

Problem Type	Why Ray Serve Fits	When to Look Elsewhere
Multi-model serving	Prefix-based routing, independent autoscaling, shared cluster	Use Triton for pure GPU-optimized inference without orchestration
Request batching for GPU	@serve.batch decorator, configurable window and batch size	Use Triton’s built-in dynamic batching for simpler deployments
Model composition / pipelines	DeploymentHandle DAG, async composition, per-stage scaling	Use KServe for Kubernetes-native inference graphs
Autoscaling ML workloads	Request-driven, configurable delays, per-deployment config	Use KEDA + custom metrics for Kubernetes-native autoscaling
LLM serving with streaming	Direct streaming mode, HAProxy integration, vLLM backend	Use vLLM directly for single-model, maximum-throughput serving
A/B testing / canary deployments	Weighted routing, per-version metrics, traffic splitting	Use Istio for service-mesh-level traffic splitting
Multi-tenant model serving	Per-tenant deployments, resource isolation, backpressure	Use Triton with model ensembles for simpler multi-tenant setups
Distributed inference	Ray actors across nodes, object store for data transfer	Use DeepSpeed or FairScale for model-parallel training

Architectural Tradeoffs

What we gained:

• Python-native serving API. Define deployments as Python classes with decorators. No YAML, no protobuf, no configuration files for basic serving. The @serve.deployment and @serve.batch decorators turn any Python class into a production-grade serving endpoint.
• Per-deployment autoscaling. Each model gets its own autoscaling config with independent min/max replicas, scale-up/down delays, and target utilization. A heavily-used embedding model can scale to 20 replicas while an experimental classifier stays at 1.
• Request batching as a first-class primitive. The @serve.batch decorator handles the hard parts of batching — request collection, timeout management, batch window sizing, result dispatching — in a tested, production-grade implementation. No custom batch collectors, no race conditions, no memory leaks.
• Model composition with async DAGs. DeploymentHandle.remote() calls compose models into pipelines with async, non-blocking semantics. Each stage can scale independently. The DAG is declarative and inspectable.
• Direct streaming for LLMs. Ray 2.55+ direct streaming mode eliminates the Python proxy bottleneck for LLM serving. Tokens flow directly from model replicas to clients, matching the throughput of Rust-based routers like vllm-router.
• Fault tolerance built on Ray actors. Replicas are Ray actors with automatic health checking, failure detection, and reconstruction. Rolling updates preserve availability during deployment changes.
• Rich observability. Ray Dashboard provides per-replica metrics, latency histograms, request rates, and error rates. Prometheus integration for production monitoring. Structured logging with request IDs.

What we sacrificed:

• Operational complexity. Ray Serve requires a Ray cluster. For simple deployments (one model, <200 QPS), a monolithic FastAPI behind nginx is faster to ship with less operational overhead. You manage Kubernetes, Ray operator, HAProxy/Envoy, and Prometheus — four systems that can each fail independently.
• Not a GPU-optimized inference engine. Ray Serve is an orchestration layer, not an inference engine. It does not optimize GPU kernel execution, manage CUDA streams, or provide TensorRT integration. For maximum GPU throughput, pair Ray Serve with Triton Inference Server inside each replica.
• Object store memory management. The Ray object store is the #1 OOM vector in production. Large model outputs, intermediate tensors, and batched results accumulate in shared memory. You must configure RAY_object_store_max_bytes and monitor with ray memory regularly.
• Cold start latency. Loading a model onto GPU takes 5-15 seconds. Ray Serve’s autoscaler does not account for this in its scaling decisions. A scale-up event triggers model loading, and requests arriving during that window experience high latency. Pre-warming with min_replicas=1 is essential.
• No built-in authentication or authorization. Ray Serve’s HTTP proxy has no auth layer. Anyone who can reach the proxy port can invoke any deployment. For production, you need a reverse proxy (Nginx, Caddy, Envoy) with auth middleware and TLS termination.
• No built-in model registry or versioning. Ray Serve does not include a model store, version manager, or artifact repository. You need MLflow, DVC, or a custom solution for model lifecycle management. Ray Serve handles routing and serving, not model storage.

The real lesson: Ray Serve is the right choice when you need to serve multiple models with independent scaling, compose them into pipelines, and handle variable traffic patterns — and you already have or are willing to operate a Ray cluster. It trades operational simplicity for flexibility and scalability. For single-model, low-complexity deployments, use FastAPI + Gunicorn. For maximum GPU throughput, use Triton Inference Server. For the middle ground — multi-model, multi-pipeline, production-grade serving — Ray Serve is the standard.

Course-Style Deep Dive

How Ray Serve Works Under the Hood

Ray Serve’s architecture is a distributed control plane built on Ray’s actor model. Here is the full request lifecycle:

1. Request arrives at the HTTP proxy. The proxy (HAProxy in direct streaming mode, or the Python proxy in standard mode) receives the HTTP request. In direct streaming mode (Ray 2.55+), HAProxy queries an ingress request router for the target replica, then establishes a direct HTTP connection with that replica. Tokens stream directly from the replica to the client, bypassing the Python proxy entirely.

2. Routing decision. The request router matches the request’s route prefix against the deployment table. Each deployment is registered with a route prefix (e.g., /classify, /embed). The router returns the deployment’s replica set and the routing metadata.

3. Request queuing. The request enters the target replica’s queue. If max_queued_requests is exceeded, the replica returns HTTP 503 (Service Unavailable). This is load shedding — far better than unbounded queue growth causing OOM or tail latency spikes.

4. Batch collection (if batching enabled). If the deployment uses @serve.batch, the replica’s batch collector accumulates requests over the batch_wait_timeout_s window. When the window expires or max_batch_size is reached, the batch is sent to the model. Results are dispatched back to the individual request handlers.

5. Model inference. The model runs on the batch (or single request). For GPU models, this is the compute-bound phase. For CPU models, this is the memory-bound phase. The replica’s max_ongoing_requests setting limits how many requests can be in flight simultaneously.

6. Response streaming (if applicable). For streaming responses (LLM generation), tokens are sent as server-sent events (SSE) or chunked transfer encoding. In direct streaming mode, tokens flow through the HAProxy connection directly to the client.

7. Metrics collection. Each replica exposes metrics: ongoing requests, queue depth, request latency, batch size distribution, error count. The autoscaler reads these metrics on a 1-second control loop and adjusts replica count.

Advanced Pattern 1: Embedding Triton Inference Server Inside Ray Serve

# triton_serve.py — Ray Serve replica wrapping Triton
from ray import serve
import tritonclient.http as triton_http
import numpy as np

@serve.deployment(
    num_replicas=2,
    autoscaling_config={
        "min_replicas": 1,
        "max_replicas": 5,
        "target_num_ongoing_requests_per_replica": 4.0,
    },
    ray_actor_options={"num_gpus": 1},
)
class TritonWrapper:
    def __init__(self):
        # Connect to Triton running on localhost (sidecar pattern)
        self.client = triton_http.InferenceServerClient(
            url="localhost:8001",
            verbose=False,
        )
        # Verify model is ready
        assert self.client.is_model_ready("bert_onnx")

    @serve.batch(max_batch_size=64, batch_wait_timeout_s=0.05)
    async def predict_batch(self, inputs: list) -> list:
        # Convert inputs to Triton format
        input_data = np.array(inputs, dtype=np.object_)
        triton_input = triton_http.InferInput(
            "input_ids", input_data.shape, "BYTES"
        )
        triton_input.set_data_from_numpy(input_data)

        # Run inference on Triton
        result = self.client.infer(
            model_name="bert_onnx",
            inputs=[triton_input],
            outputs=[triton_http.InferRequestedOutput("output")],
        )

        # Extract and return results
        output = result.as_numpy("output")
        return output.tolist()

    async def __call__(self, request):
        data = (await request.json())["text"]
        result = await self.predict_batch(data)
        return {"result": result}

This pattern gives you the best of both worlds: Triton’s GPU-optimized inference (TensorRT, dynamic batching, FP16/INT8) with Ray Serve’s orchestration (autoscaling, routing, composition, fault tolerance).

Advanced Pattern 2: Session-Aware Routing for LLM Multi-Turn Conversations

# session_routing.py — Ray 2.56+ session-aware routing
from ray import serve

@serve.deployment(
    ray_actor_options={"num_gpus": 1},
)
class LLMServer:
    def __init__(self):
        from vllm import AsyncLLMEngine
        self.engine = AsyncLLMEngine.from_args(
            model="meta-llama/Llama-3.1-8B-Instruct",
            tensor_parallel_size=1,
        )

    @serve.multiplexed(name="session", size_per_replica=10)
    async def load_session(self, session_id: str):
        # Touches LRU — vLLM handles prefix caching internally
        # This ensures the session's KV cache stays on this replica
        pass

    async def __call__(self, request):
        body = await request.json()
        session_id = body.get("session_id", "default")
        messages = body["messages"]

        # Session affinity: all requests with this session_id
        # are routed to the same replica
        await self.load_session(session_id)

        # Generate response
        result = await self.engine.generate(
            messages=messages,
            session_id=session_id,
        )
        return {"response": result}

Session-aware routing (Ray 2.56+, RFC #62645) ensures that all requests in a multi-turn conversation are routed to the same replica. This avoids redundant KV cache computation — the second turn does not need to reprocess the first turn’s tokens. The @serve.multiplexed decorator manages an LRU cache of sessions per replica, with configurable capacity.

Advanced Pattern 3: gRPC Proxy for Internal Microservice Communication

# grpc_serve.py — gRPC-based internal communication
from ray import serve
import grpc
from concurrent import futures

@serve.deployment
class GRPCProxy:
    def __init__(self):
        self.server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
        # Register your gRPC service
        # self.server.add_insecure_port("[::]:50051")
        # self.server.start()

    async def __call__(self, request):
        # Convert HTTP request to gRPC call
        # Route to internal gRPC service
        pass

Using gRPC for internal communication (via Envoy sidecar) reduces serialization overhead by ~60% compared to JSON. Pin the gRPC proxy to separate CPU cores and set per-request timeouts (e.g., 5s) to prevent head-of-line blocking.

Production Considerations

Object store memory budgeting. The Ray object store is shared memory used for zero-copy data transfer between actors. It is the #1 OOM vector in production:

# Set object store to 80% of node RAM
export RAY_object_store_max_bytes=$((80 * 1024 * 1024 * 1024))  # 80 GB on 100 GB node
export RAY_memory_monitor_refresh_ms=500

# Monitor object store usage
ray memory

Client-side retry with exponential backoff. Production HTTP clients should use retries with backoff to handle transient failures:

import requests
from requests.adapters import HTTPAdapter, Retry

session = requests.Session()
retries = Retry(
    total=5,
    backoff_factor=0.5,  # 0.5s, 1s, 2s, 4s, 8s
    status_forcelist=[500, 501, 502, 503, 504],
)
session.mount("http://", HTTPAdapter(max_retries=retries))
response = session.get("http://localhost:8000/classify", timeout=30)

Load shedding configuration. The max_ongoing_requests and max_queued_requests settings are your first line of defense against cascading failures:

@serve.deployment(
    max_ongoing_requests=2,     # Concurrent requests per replica
    max_queued_requests=10,     # Queue depth before 503 rejection
)
class MyDeployment:
    ...

When the queue limit is hit, requests return HTTP 503. This is far better than unbounded queue growth causing OOM or tail latency spikes. Monitor 503 rates as a signal that you need more replicas.

Cold start mitigation. Model loading takes 5-15 seconds. The autoscaler does not account for this. Mitigations:

# 1. Always keep at least 1 idle replica
autoscaling_config = {
    "min_replicas": 1,  # Never scale to 0
}

# 2. Pre-warm on deploy
# Run a warm-up request after deployment
requests.post("http://localhost:8000/classify", json={"text": "warmup"})

# 3. Use aggressive upscale delay (fast scale-up)
autoscaling_config = {
    "upscale_delay_s": 5,  # Scale up within 5 seconds of traffic increase
}

Rolling updates without downtime. Ray Serve supports in-place updates for lightweight config changes (num_replicas, autoscaling_config, user_config, max_ongoing_requests). For code changes, use the KubeRay RayService resource which handles rolling updates with health checking:

# Rolling update config in RayService
spec:
  serveConfigV2: |
    # ... deployment config
  # KubeRay handles the rolling update automatically
  # when serveConfigV2 changes

The Results

Metric	Before Ray Serve (FastAPI + Gunicorn)	After Ray Serve	Improvement
Multi-model setup time	4-8 hours (Nginx config, service orchestration)	30 minutes (deployment definitions)	8-16x faster
Request batching implementation	4-8 hours (custom collector, testing, debugging)	1 line (@serve.batch)	Eliminated entirely
Autoscaling configuration	Manual (fixed workers, manual scaling)	Declarative (autoscaling_config)	Eliminated entirely
GPU utilization (embeddings)	30-40% (single request per GPU)	85-95% (dynamic batching)	2-3x improvement
GPU utilization (LLM)	40-50% (no batching)	80-90% (continuous batching via vLLM)	1.8-2x improvement
p99 latency under 2x traffic spike	5 seconds (queue buildup)	800ms (autoscaling absorbs spike)	6.25x improvement
Model pipeline latency	N/A (separate services, network hops)	30-50% reduction (in-process composition)	30-50% faster
LLM throughput (prefill-heavy)	Baseline (Ray Serve 2.54)	4.4x (Ray 2.56 direct streaming)	4.4x improvement
LLM throughput (decode-heavy)	Baseline (Ray Serve 2.54)	24.8x (Ray 2.56 direct streaming)	24.8x improvement
Time to deploy new model	2-4 hours (Docker build, K8s manifest, CI)	15 minutes (serve deploy config)	8-16x faster
Production incidents (monthly)	5-10 (OOM, queue buildup, scaling failures)	1-3 (mostly object store related)	3-5x reduction

What this means for you: Ray Serve does not make individual model inference faster — the model runs at the same speed regardless of the serving framework. What Ray Serve eliminates is the infrastructure work of building autoscaling, batching, routing, and composition from scratch. The 8-16x improvement in deployment time and the 2-3x improvement in GPU utilization are the real metrics. For a team serving 10 models, that is 40-80 hours of saved setup time plus $5,000-15,000/month in reduced GPU costs from better utilization.

What to Watch Out For

1. Object store OOM is the #1 production issue. The Ray object store accumulates model outputs, intermediate tensors, and batched results in shared memory. If you do not set RAY_object_store_max_bytes, it defaults to 30% of node RAM — which is too low for GPU workloads. Set it to 80% of node RAM and monitor with ray memory regularly.

2. Downscale delay too short causes thrashing. The default downscale_delay_s is 600 seconds. If you set it too low (e.g., 30 seconds), the autoscaler will scale down replicas during brief traffic dips, then immediately scale back up when traffic returns. Each scale-up triggers a 5-15 second model loading penalty. Set downscale_delay_s to at least 300 seconds for GPU workloads.

3. max_ongoing_requests too high causes latency degradation. Setting max_ongoing_requests too high allows many concurrent requests per replica, but each request competes for GPU memory and compute. The sweet spot is 2-5 for GPU models and 10-50 for CPU models. Monitor p99 latency as you increase this value — when it starts climbing, you have hit the limit.

4. Model loading time is invisible to the autoscaler. The autoscaler sees a replica as “ready” as soon as the Ray actor starts, not when the model finishes loading. Requests arriving during the 5-15 second model loading window experience high latency. Mitigate with min_replicas=1 and pre-warming.

5. No built-in authentication. Ray Serve’s HTTP proxy has no auth layer. Anyone who can reach the proxy port can invoke any deployment. Use a reverse proxy (Nginx, Caddy, Envoy) with TLS termination and auth middleware for any network-exposed deployment.

6. gRPC is significantly faster than JSON for internal communication. If your pipeline has multiple stages, the serialization overhead of JSON between stages adds up. Use gRPC (via Envoy sidecar) for internal communication to reduce overhead by ~60%.

7. HAProxy direct streaming mode requires Ray 2.55+. The direct streaming mode that delivers 4.4-24.8x throughput improvements for LLM serving is only available in Ray 2.55+. If you are on an older version, the Python proxy handles all streaming, which becomes a bottleneck under high concurrency.

Lesson 1: “We spent two weeks debugging why our Ray Serve cluster kept OOMing. Turns out the object store was filling up with large embedding outputs that were not being garbage collected. We had to set RAY_object_store_max_bytes and add explicit del calls for large tensors. The Ray dashboard’s memory view saved us — it showed object store usage climbing linearly with request count.” — ML infrastructure engineer, fintech company

Lesson 2: “Our autoscaler was thrashing like crazy. Every 30 seconds it would scale down from 5 replicas to 2, then scale back up to 5. The model loading time was 12 seconds, so we were spending 24 seconds out of every 30 seconds loading models. Set downscale_delay_s to 300 and the problem disappeared. The default is a trap for GPU workloads.” — MLOps engineer, e-commerce platform

Lesson 3: “We tried to use Ray Serve without Kubernetes. Don’t. The KubeRay operator handles health checking, status reporting, failure recovery, and upgrades automatically. Running Ray Serve on bare metal or Docker Compose means you are managing all of that yourself. Use KubeRay from day one — it saves weeks of operational pain.” — Production engineer, AI startup

Advice for Getting Started

1. Start with a single deployment on a single node before adding multi-model complexity. serve run my_deployment.py gives you a local server on port 8000. Test with curl before writing any client code.

2. Add @serve.batch to any GPU-bound deployment immediately. The throughput gain is 5-10x for embedding models and 2-3x for classification models. Start with max_batch_size=16 and batch_wait_timeout_s=0.1 and tune from there.

3. Set min_replicas=1 for every GPU deployment. Never let a GPU deployment scale to 0 — the cold start penalty of loading a model onto GPU is 5-15 seconds, which is unacceptable for any production workload.

4. Use the Ray Dashboard from day one. It shows per-replica metrics, latency histograms, request rates, and error rates. The dashboard is the fastest way to diagnose performance issues.

5. Configure client-side retries with exponential backoff. Ray Serve replicas can fail, restart, or reject requests under load. A retry policy with backoff_factor=0.5 and total=5 handles transient failures gracefully.

6. Monitor object store memory usage with ray memory. This is the most common production issue. Set up an alert when object store usage exceeds 80% of RAY_object_store_max_bytes.

7. Use KubeRay for production deployments. The RayService custom resource handles health checking, status reporting, failure recovery, and rolling updates automatically. Do not manage Ray Serve on bare metal.

---

Next in the Open-Source AI Tools Mastery series: Triton Inference Server