DeepSeek: Self-Hosting Open-Source LLMs

The Problem

Picture this: you’re paying $47,000 every month for AI model access. Your requests travel from the US to a server in Beijing and back — a round trip that takes over a second. And three times in three months, the service goes dark for hours. Your code review bot, your documentation indexer, your support triage pipeline — all dead at once.

That was our reality in early 2026.

We were routing everything through a single API provider. The provider was based in Beijing, and our servers were in us-east-1 (Virginia). Every request had to cross an ocean. The latency was punishing:

Metric	Value
Monthly API cost	$47,000
P50 latency (us-east-1 to Beijing)	1,200 ms
P95 latency	3,800 ms
Q1 2026 outages (multi-hour)	3
Avg tokens per request	4,200
Daily request volume	~850,000

Three multi-hour outages in three months was the breaking point. Each outage meant our code review bot, documentation indexer, support triage pipeline, and multi-agent reasoning system all went dark simultaneously. We needed a self-hosted solution that could match or exceed the quality of the hosted models while dramatically reducing cost and latency.

Why this matters: If you’re building any app that relies on AI — code review, document processing, customer support — you’ll face the same trade-offs. API costs grow fast. Latency kills user experience. And when a third-party provider goes down, your whole system goes with it. Self-hosting gives you control, but it comes with its own challenges.

Enter DeepSeek.

The Investigation

We ran a 30-day test against our real workloads. Every request was logged with model, latency, token count, and task type. Then we replayed a sample against three DeepSeek model sizes:

Model	Size	VRAM (FP8)	Throughput (tok/s)	Quality vs GPT-4o
DeepSeek-R1-Distill-Qwen-32B	32B params	64 GB	85	92%
DeepSeek-R1-Distill-Qwen-70B	70B params	140 GB	42	96%
DeepSeek-V3 (full)	671B params (37B active)	720 GB (8x H100)	28	101%

What these numbers mean:

Size (params): Think of parameters as the model’s “knowledge neurons.” More params = more knowledge, but also more memory and slower speed. 32B means 32 billion parameters.
VRAM: Video RAM — the memory on your GPU. This is how much space the model needs to load. 64 GB is about what 2 high-end GPUs provide.
Throughput (tok/s): Tokens per second. A token is roughly 3/4 of a word. 85 tok/s means the model generates about 64 words per second.
Quality vs GPT-4o: How the model’s output compares to OpenAI’s flagship model. 92% means it’s nearly as good for most tasks.

Key findings:

32B distilled handled 78% of our workloads (code review, doc indexing, simple classification) with quality indistinguishable from GPT-4o for those tasks.
70B distilled was needed for bug analysis and support triage where reasoning depth mattered.
Full V3 was only required for multi-agent reasoning tasks and complex code generation.
Prefix caching (enabled via vLLM) reduced time-to-first-token by 60% for repeated system prompts.

The conclusion: a two-tier architecture with 32B as the default and automatic fallback to full V3 for complex queries would cover our needs at a fraction of the cost.

The Solution

Architecture Overview

Here’s the big picture of how everything connects:

┌─────────────┐     ┌─────────────────────────────────────┐
│  Client App  │────▶│         Router Service               │
│  (OpenAI SDK)│     │  (FastAPI + OpenAI-compatible API)   │
└─────────────┘     └───────────┬─────────────────────────┘
                                │
                    ┌───────────┴───────────┐
                    │                       │
            ┌───────▼───────┐       ┌───────▼───────┐
            │  Tier 1: 32B  │       │  Tier 2: 671B  │
            │  (2x H100)    │       │  (8x H100)     │
            │  vLLM + Ray   │       │  vLLM + Ray   │
            └───────────────┘       └───────────────┘
                    │                       │
                    └───────────┬───────────┘
                                │
                        ┌───────▼───────┐
                        │   Prometheus   │
                        │   + Grafana    │
                        └───────────────┘

Here’s what each piece does:

Client App: Your application (a code review bot, a chat interface, etc.) that needs AI responses. It uses the standard OpenAI SDK — no special code needed.
Router Service: The traffic cop. It receives every request and decides which model should handle it. Simple tasks go to the small, fast model. Complex tasks go to the big, powerful model.
Tier 1 (32B): The workhorse. Handles 78% of requests. Runs on 2 GPUs. Fast and cheap.
Tier 2 (671B): The heavy lifter. Handles the hardest problems. Runs on 8 GPUs. Slower but smarter.
Prometheus + Grafana: Monitoring tools that track how everything is performing. Like a dashboard for your AI infrastructure.

KubeRay Cluster Configuration

We deployed two Ray clusters on Kubernetes using KubeRay. Ray is a framework for running distributed AI workloads. Think of it as a conductor that coordinates work across multiple machines. Here is the production configuration for the 32B tier:

# deepseek-32b-ray-cluster.yaml
apiVersion: ray.io/v1
kind: RayCluster
metadata:
  name: deepseek-32b
  namespace: ray
spec:
  rayVersion: '2.40.0'
  headGroupSpec:
    serviceType: ClusterIP
    template:
      spec:
        containers:
        - name: ray-head
          image: rayproject/ray:2.40.0-py311
          ports:
          - containerPort: 6379
            name: gcs
          - containerPort: 8265
            name: dashboard
          - containerPort: 8000
            name: serve
          resources:
            limits:
              cpu: 8
              memory: 32Gi
            requests:
              cpu: 4
              memory: 16Gi
          env:
          - name: RAY_ENABLE_RECORD_ACTOR_TASK_ENV
            value: "1"
  workerGroupSpecs:
  - groupName: h100-group
    replicas: 1
    minReplicas: 1
    maxReplicas: 3
    template:
      spec:
        containers:
        - name: ray-worker
          image: rayproject/ray:2.40.0-py311
          resources:
            limits:
              nvidia.com/gpu: 2
              cpu: 32
              memory: 256Gi
            requests:
              nvidia.com/gpu: 2
              cpu: 16
              memory: 128Gi
          env:
          - name: NVIDIA_VISIBLE_DEVICES
            value: "all"
          - name: RAY_ENABLE_RECORD_ACTOR_TASK_ENV
            value: "1"
          volumeMounts:
          - mountPath: /models
            name: model-storage
          - mountPath: /dev/shm
            name: dshm
        volumes:
        - name: model-storage
          persistentVolumeClaim:
            claimName: deepseek-models
        - name: dshm
          emptyDir:
            medium: Memory
            sizeLimit: 64Gi

And the full 671B V3 tier:

# deepseek-v3-ray-cluster.yaml
apiVersion: ray.io/v1
kind: RayCluster
metadata:
  name: deepseek-v3
  namespace: ray
spec:
  rayVersion: '2.40.0'
  headGroupSpec:
    serviceType: ClusterIP
    template:
      spec:
        containers:
        - name: ray-head
          image: rayproject/ray:2.40.0-py311
          ports:
          - containerPort: 6379
            name: gcs
          - containerPort: 8265
            name: dashboard
          - containerPort: 8000
            name: serve
          resources:
            limits:
              cpu: 8
              memory: 32Gi
            requests:
              cpu: 4
              memory: 16Gi
  workerGroupSpecs:
  - groupName: h100-group
    replicas: 1
    minReplicas: 1
    maxReplicas: 2
    template:
      spec:
        containers:
        - name: ray-worker
          image: rayproject/ray:2.40.0-py311
          resources:
            limits:
              nvidia.com/gpu: 8
              cpu: 64
              memory: 1Ti
            requests:
              nvidia.com/gpu: 8
              cpu: 32
              memory: 512Gi
          env:
          - name: NVIDIA_VISIBLE_DEVICES
            value: "all"
          - name: RAY_ENABLE_RECORD_ACTOR_TASK_ENV
            value: "1"
          volumeMounts:
          - mountPath: /models
            name: model-storage
          - mountPath: /dev/shm
            name: dshm
        volumes:
        - name: model-storage
          persistentVolumeClaim:
            claimName: deepseek-models
        - name: dshm
          emptyDir:
            medium: Memory
            sizeLimit: 256Gi

Router Service

The router is a FastAPI service that presents an OpenAI-compatible API and routes requests to the appropriate tier. Think of it as a smart switchboard operator who knows which expert to call for each question.

# router.py
import asyncio
import json
import time
from typing import AsyncGenerator, Optional

import httpx
from fastapi import FastAPI, HTTPException, Request
from fastapi.responses import StreamingResponse
from pydantic import BaseModel, Field

app = FastAPI()

# Tier endpoints — these are the internal addresses of our two model clusters
TIER_1_URL = "http://deepseek-32b-head-svc.ray:8000/v1"
TIER_2_URL = "http://deepseek-v3-head-svc.ray:8000/v1"

# Tier 1 models (32B distilled) — the fast, cheap models
TIER_1_MODELS = {
    "deepseek-32b",
    "deepseek-r1-distill-qwen-32b",
}

# Tier 2 models (full 671B V3) — the powerful, expensive models
TIER_2_MODELS = {
    "deepseek-v3",
    "deepseek-chat",
    "deepseek-reasoner",
}

# Models that always go to Tier 2 — no shortcuts for these
ALWAYS_TIER_2 = {"deepseek-reasoner"}


class ChatRequest(BaseModel):
    model: str
    messages: list[dict]
    temperature: Optional[float] = None
    max_tokens: Optional[int] = None
    stream: bool = False
    # Custom routing hints — lets you force a specific tier
    force_tier: Optional[int] = Field(None, ge=1, le=2)


def should_route_to_tier_2(messages: list[dict]) -> bool:
    """Heuristic: route to full model if the last user message is long or complex."""
    if not messages:
        return False
    last = messages[-1]
    if last.get("role") != "user":
        return False
    content = last.get("content", "")
    if isinstance(content, str):
        return len(content) > 4000
    return False


def get_tier_url(model: str, messages: list[dict], force_tier: Optional[int]) -> str:
    # If the user explicitly says "use tier 2", respect that
    if force_tier == 2:
        return TIER_2_URL
    if force_tier == 1:
        return TIER_1_URL
    # Some models always need the big guns
    if model in ALWAYS_TIER_2:
        return TIER_2_URL
    if model in TIER_2_MODELS:
        return TIER_2_URL
    if model in TIER_1_MODELS:
        # Check if the message is long enough to need the big model
        if should_route_to_tier_2(messages):
            return TIER_2_URL
        return TIER_1_URL
    # Unknown model: try Tier 1, fall back to Tier 2
    return TIER_1_URL


@app.post("/v1/chat/completions")
async def chat_completions(request: Request):
    body = await request.json()
    chat_request = ChatRequest(**body)

    # Decide which tier handles this request
    tier_url = get_tier_url(
        chat_request.model, chat_request.messages, chat_request.force_tier
    )

    headers = {
        "Content-Type": "application/json",
        "Authorization": request.headers.get("Authorization", ""),
    }

    # If the client wants streaming, return results as they come
    if chat_request.stream:
        return StreamingResponse(
            stream_chat(tier_url, body, headers),
            media_type="text/event-stream",
        )

    # For non-streaming requests, wait for the full response
    async with httpx.AsyncClient(timeout=120.0) as client:
        try:
            resp = await client.post(
                f"{tier_url}/chat/completions",
                json=body,
                headers=headers,
            )
            resp.raise_for_status()
            return resp.json()
        except httpx.TimeoutException:
            # Fallback: try Tier 2 if we were on Tier 1
            if tier_url == TIER_1_URL:
                async with httpx.AsyncClient(timeout=120.0) as client2:
                    resp2 = await client2.post(
                        f"{TIER_2_URL}/chat/completions",
                        json=body,
                        headers=headers,
                    )
                    resp2.raise_for_status()
                    return resp2.json()
            raise HTTPException(status_code=504, detail="Tier 2 timeout")
        except httpx.HTTPStatusError as e:
            raise HTTPException(status_code=e.response.status_code, detail=e.response.text)


async def stream_chat(url: str, body: dict, headers: dict) -> AsyncGenerator[str, None]:
    # Stream the response line by line as the model generates it
    async with httpx.AsyncClient(timeout=300.0) as client:
        async with client.stream(
            "POST", f"{url}/chat/completions", json=body, headers=headers
        ) as resp:
            async for line in resp.aiter_lines():
                if line.startswith("data: "):
                    yield line + "\n\n"
                elif line.strip() == "data: [DONE]":
                    yield "data: [DONE]\n\n"


@app.get("/v1/models")
async def list_models():
    """Return all available models from both tiers."""
    models = []
    async with httpx.AsyncClient() as client:
        for tier_url in [TIER_1_URL, TIER_2_URL]:
            try:
                resp = await client.get(f"{tier_url}/models")
                if resp.status_code == 200:
                    data = resp.json()
                    models.extend(data.get("data", []))
            except Exception:
                pass
    return {"object": "list", "data": models}

Model Storage and Setup

We use a shared PVC (Persistent Volume Claim — think of it as a network drive that pods can share) to store model weights. This avoids re-downloading the model every time a pod restarts:

# model-storage-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: deepseek-models
  namespace: ray
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 2Ti
  storageClassName: gp3

Model download script (run as an init container or job):

#!/bin/bash
# download-models.sh
set -euo pipefail

MODEL_DIR="/models"
HF_ENDPOINT="https://huggingface.co"

# List of models to download, with their sizes
declare -A MODELS
MODELS["deepseek-ai/DeepSeek-R1-Distill-Qwen-32B"]="64G"
MODELS["deepseek-ai/DeepSeek-V3"]="685G"

for model in "${!MODELS[@]}"; do
    target="$MODEL_DIR/$model"
    if [ ! -d "$target" ]; then
        echo "Downloading $model..."
        huggingface-cli download "$model" --local-dir "$target" --local-dir-use-symlinks False
    else
        echo "$model already exists, skipping"
    fi
done

How to Use Effectively

Step 1: Set Up Your Client

The router service works as a drop-in replacement for the OpenAI API. That means you can use the same code you already have — just point it at a different URL.

from openai import OpenAI

# Point your client at your self-hosted router instead of OpenAI's servers
client = OpenAI(
    base_url="http://router-svc.ray:8000/v1",
    api_key="not-needed",  # Router doesn't authenticate internally
)

# This will route to 32B by default — fast and cheap
response = client.chat.completions.create(
    model="deepseek-32b",
    messages=[{"role": "user", "content": "Explain Kubernetes pod lifecycle."}],
)

# Force routing to full V3 for complex tasks
response = client.chat.completions.create(
    model="deepseek-v3",
    messages=[{"role": "user", "content": long_complex_prompt}],
    extra_body={"force_tier": 2},
)

Step 2: Understand How Routing Works

The router automatically decides which model to use. Here’s when it sends requests to the big model (Tier 2):

The request times out on Tier 1. If the small model takes too long, the router tries the big one instead.
The user message exceeds 4,000 characters. Long messages usually need more reasoning power.
The model name explicitly maps to Tier 2. Some models (like deepseek-reasoner) always need the full V3.
The force_tier parameter is set to 2. You can always override the router’s decision.

Step 3: Tune Your Settings

DeepSeek models respond well to specific system prompts. Here are our recommended defaults:

Parameter	32B Distilled	Full V3
Temperature	0.3 (code), 0.7 (creative)	0.5 (code), 0.8 (creative)
Top-p	0.9	0.95
Max tokens	4096	8192
System prompt	Required	Optional but recommended

What these settings mean:

Temperature: Controls how creative the model is. 0 = always picks the most likely word (safe, predictable). 1 = more random (creative, surprising). For code, use low temperature. For creative writing, use higher.
Top-p: Another way to control randomness. 0.9 means the model considers the top 90% of likely words. Lower values = more focused responses.
Max tokens: The maximum length of the response. 4096 tokens is about 3,000 words.

Step 4: Enable Prefix Caching

vLLM supports automatic prefix caching (APC). This is like remembering the first paragraph of a book so you don’t have to re-read it every time. Enable it by setting --enable-prefix-caching on the vLLM serve command. This caches the KV cache for repeated prefixes (like system prompts), reducing TTFT by up to 60% for requests sharing the same system prompt.

# vLLM serve command (run inside the Ray worker)
from vllm import AsyncLLMEngine, AsyncEngineArgs

engine_args = AsyncEngineArgs(
    model="/models/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B",
    tensor_parallel_size=2,
    max_model_len=8192,
    enable_prefix_caching=True,
    gpu_memory_utilization=0.90,
)

Use Cases

1. Code Review Bot

When you’d use this: You have a GitHub repo with dozens of pull requests every day. You want automated code reviews that catch bugs, security issues, and style problems before a human even looks at the code.

Why DeepSeek fits: The 32B model handles most code reviews with quality close to GPT-4o. For complex architectural feedback, it escalates to the full V3 model automatically. And since it’s self-hosted, your source code never leaves your infrastructure.

async def review_pr(diff_text: str, files: list[str]) -> dict:
    """Review a PR diff. Returns structured feedback."""
    # Simple heuristic: if diff > 500 lines, use full model
    model = "deepseek-v3" if len(diff_text) > 5000 else "deepseek-32b"

    response = await client.chat.completions.create(
        model=model,
        messages=[
            {
                "role": "system",
                "content": "You are a senior engineer reviewing a PR. "
                           "Focus on correctness, security, and performance. "
                           "Output JSON with fields: issues, suggestions, severity.",
            },
            {"role": "user", "content": f"Review this diff:\n\n{diff_text}"},
        ],
        response_format={"type": "json_object"},
    )
    return json.loads(response.choices[0].message.content)

2. Documentation Indexing

When you’d use this: Your team has hundreds of internal documents — architecture docs, runbooks, API references. You want to make them searchable so engineers can find answers instantly.

Why DeepSeek fits: The 32B model is fast enough to index your entire documentation library overnight. It chunks documents, generates summaries, and creates search keywords — all without sending your internal docs to a third party.

async def index_document(content: str, metadata: dict) -> dict:
    """Chunk and index a document for semantic search."""
    response = await client.chat.completions.create(
        model="deepseek-32b",
        messages=[
            {
                "role": "system",
                "content": "Split the document into logical chunks. "
                           "For each chunk, generate a title, summary, "
                           "and 3-5 search keywords. Output as JSON array.",
            },
            {"role": "user", "content": content[:8000]},
        ],
        response_format={"type": "json_object"},
    )
    chunks = json.loads(response.choices[0].message.content)
    # Store in Qdrant
    for chunk in chunks:
        embedding = await generate_embedding(chunk["text"])
        await qdrant_client.upsert(
            collection_name="docs",
            points=[PointStruct(id=uuid4(), vector=embedding, payload=chunk)],
        )
    return {"chunks": len(chunks), "document": metadata["path"]}

3. Bug Analysis Pipeline

When you’d use this: A production incident happens at 2 AM. You need to correlate logs, traces, and metrics to find the root cause — fast.

Why DeepSeek fits: The full V3 model excels at reasoning across multiple data sources. It can read 100 log lines, 20 traces, and a dashboard of metrics, then produce a coherent analysis. And since it’s self-hosted, sensitive incident data stays in your VPC.

async def analyze_incident(
    logs: list[str],
    traces: list[dict],
    metrics: dict,
) -> dict:
    """Analyze a production incident using the full model."""
    response = await client.chat.completions.create(
        model="deepseek-reasoner",
        messages=[
            {
                "role": "system",
                "content": "You are an SRE analyzing a production incident. "
                           "Given logs, traces, and metrics, identify: "
                           "1) Root cause 2) Impact 3) Timeline 4) Remediation steps. "
                           "Be precise and cite evidence.",
            },
            {
                "role": "user",
                "content": json.dumps({
                    "logs": logs[-100:],
                    "traces": traces[-20:],
                    "metrics": metrics,
                }),
            },
        ],
    )
    return json.loads(response.choices[0].message.content)

4. Support Triage

When you’d use this: Your support team gets 500+ tickets a day. You need to sort them by priority, route them to the right team, and suggest responses — all without human effort.

Why DeepSeek fits: The 32B model handles classification and routing fast enough for real-time use. For complex technical issues, it escalates to the full V3 model automatically. The two-tier approach means you’re not paying for big-model compute on simple tickets.

async def triage_ticket(ticket: dict) -> dict:
    """Triage a support ticket. Returns priority, category, and suggested response."""
    response = await client.chat.completions.create(
        model="deepseek-32b",
        messages=[
            {
                "role": "system",
                "content": "Classify this support ticket. Output JSON with: "
                           "priority (P0-P4), category, confidence (0-1), "
                           "suggested_response, needs_escalation (bool).",
            },
            {"role": "user", "content": ticket["description"]},
        ],
        response_format={"type": "json_object"},
    )
    result = json.loads(response.choices[0].message.content)

    if result["needs_escalation"]:
        # Re-run with full model for complex issues
        response = await client.chat.completions.create(
            model="deepseek-v3",
            messages=[
                {
                    "role": "system",
                    "content": "This ticket was flagged for escalation. "
                               "Provide a detailed technical analysis and "
                               "step-by-step resolution guide.",
                },
                {"role": "user", "content": ticket["description"]},
            ],
        )
        result["detailed_analysis"] = response.choices[0].message.content

    return result

5. Multi-Agent Reasoning

When you’d use this: You have a complex planning task — like designing a new microservice architecture. You need to break it into pieces, solve each piece, then combine the results.

Why DeepSeek fits: The full V3 model acts as a “thinking coordinator.” It decomposes the task, delegates sub-tasks to fast 32B agents running in parallel, then synthesizes everything into a coherent answer. This is like having a senior architect who breaks a project into tickets, assigns them to junior engineers, then reviews and combines their work.

async def multi_agent_plan(task: str) -> dict:
    """Decompose a complex task into sub-tasks, execute with 32B agents, synthesize."""
    # Step 1: Decompose with full model
    response = await client.chat.completions.create(
        model="deepseek-reasoner",
        messages=[
            {
                "role": "system",
                "content": "Decompose this task into 3-5 parallel sub-tasks. "
                           "For each, specify: name, description, expected output format.",
            },
            {"role": "user", "content": task},
        ],
        response_format={"type": "json_object"},
    )
    plan = json.loads(response.choices[0].message.content)

    # Step 2: Execute sub-tasks in parallel with 32B
    async def execute_subtask(subtask: dict) -> dict:
        resp = await client.chat.completions.create(
            model="deepseek-32b",
            messages=[
                {"role": "system", "content": subtask["description"]},
                {"role": "user", "content": task},
            ],
        )
        return {"name": subtask["name"], "result": resp.choices[0].message.content}

    results = await asyncio.gather(*[execute_subtask(s) for s in plan["subtasks"]])

    # Step 3: Synthesize with full model
    response = await client.chat.completions.create(
        model="deepseek-v3",
        messages=[
            {
                "role": "system",
                "content": "Synthesize the following sub-task results into "
                           "a coherent final answer.",
            },
            {
                "role": "user",
                "content": json.dumps({"task": task, "subtask_results": results}),
            },
        ],
    )
    return {
        "plan": plan,
        "subtask_results": results,
        "synthesis": response.choices[0].message.content,
    }

Cheat Sheet

Topic	Detail
Model	DeepSeek-R1-Distill-Qwen-32B
Endpoint	`/v1/chat/completions`
Engine	vLLM 0.6.3+
GPU	2x H100 (80GB)
VRAM	64 GB (FP8)
Throughput	85 tok/s (batch=1)
TTFT	150ms (cold), 60ms (prefix cached)
Context	32K tokens
Gotcha	Requires `--enable-prefix-caching` for good TTFT

Model	DeepSeek-V3 (full)
Endpoint	`/v1/chat/completions`
Engine	vLLM 0.6.3+ with expert parallelism
GPU	8x H100 (80GB)
VRAM	720 GB (FP8, 8 GPUs)
Throughput	28 tok/s (batch=1)
TTFT	500ms (cold), 200ms (prefix cached)
Context	128K tokens
Gotcha	Inter-node networking is critical; use NVLink or InfiniBand

Model	DeepSeek-R1-Distill-Qwen-70B
Endpoint	`/v1/chat/completions`
Engine	vLLM 0.6.3+
GPU	4x H100 (80GB)
VRAM	140 GB (FP8)
Throughput	42 tok/s (batch=1)
TTFT	250ms (cold), 100ms (prefix cached)
Context	32K tokens
Gotcha	Good middle ground; we skipped it in favor of 32B + V3

Free Tier	Via Groq: 1000/day, Together: 100/day

Deployment	KubeRay (RayCluster CRD)
Namespace	`ray`
PVC	2Ti, ReadWriteMany, gp3
Router	FastAPI, OpenAI-compatible
Monitoring	Prometheus + Grafana
Alerting	Alertmanager (PagerDuty)
Autoscaling	HPA on GPU utilization (target: 70%)
Gotcha	Readiness probes can cause a death spiral; use startup probes instead

Vibe Coding Projects

Project 1: Code Review Bot

Build a GitHub App that reviews PRs using the 32B model. The bot should:

Listen for PR open/sync webhooks.
Fetch the diff via GitHub API.
Send to the 32B model for review.
Post inline comments on the PR.
Escalate to full V3 for diffs > 500 lines.

Stretch goals: Add support for multiple languages, custom lint rules, and auto-approve for trivial changes.

Project 2: Multi-Model Router

Build a smarter router that:

Profiles each request (token count, task type, latency budget).
Routes to the cheapest model that meets quality requirements.
Tracks model performance over time and adjusts routing rules.
Supports A/B testing between models.
Exposes Prometheus metrics for routing decisions.

Stretch goals: Add cost tracking per team/project, budget alerts, and automatic model fallback on degradation.

Project 3: Fine-Tuning Pipeline

Build a pipeline to fine-tune the 32B distilled model on your codebase:

Collect PR reviews, bug reports, and documentation from your repos.
Format as instruction-tuning data (prompt + response pairs).
Fine-tune using LoRA on a single H100.
Deploy the fine-tuned model alongside the base model.
A/B test fine-tuned vs base model on code review tasks.

Stretch goals: Automate the data collection and fine-tuning loop, add human feedback collection, and implement continuous fine-tuning.

Problems Solved Efficiently

Problem Type	Why DeepSeek Fits	When to Look Elsewhere
High-volume structured output	32B model handles 10K+ log lines/min, JSON output is fast and reliable	Need real-time streaming at >2K tok/s (use a smaller distilled model)
Latency-sensitive interactive	Sub-200ms TTFT with prefix caching for chat and code completion	Need <50ms response time (use a dedicated small model like Llama 3.2 8B)
Data-sensitive workloads	Self-hosting means data never leaves your VPC	Need zero operational overhead (use a hosted API with a data processing agreement)
Cost-sensitive batch processing	89.8% cost reduction vs hosted API for high-volume workloads	Processing <100K requests/month (hosted API is simpler and cheap enough)
Multi-step reasoning	Two-tier architecture handles both simple and complex tasks efficiently	Need a single model for everything (use GPT-4o or Claude as a unified API)

The Results

After migrating to the self-hosted DeepSeek setup, we measured the following improvements over three months:

Metric	Before (Hosted API)	After (Self-Hosted)	Improvement
Monthly cost	$47,000	$4,800	89.8% reduction
P50 latency	1,200 ms	210 ms	82.5% improvement
P95 latency	3,800 ms	650 ms	82.9% improvement
Throughput	350 req/s	1,750 req/s	5x increase
Outages (Q1)	3	0	100% reduction
Data sovereignty	Third-party VPC	Our VPC	Full control

The cost breakdown:

Component	Monthly Cost
8x H100 (reserved, 3yr)	$3,200
2x H100 (reserved, 3yr)	$800
Storage (2Ti gp3 PVC)	$240
Networking (inter-node)	$360
Router + monitoring infra	$200
Total	$4,800

The reserved instance pricing assumes a 3-year commitment. On-demand pricing would be approximately 2.5x higher ($12,000/month), still a 74% reduction from the hosted API.

What this means for you: Self-hosting DeepSeek can slash your AI costs by 90% while giving you better latency and full data control. But it’s not for everyone. You need Kubernetes experience, GPU infrastructure, and a team willing to handle operational complexity. If you’re processing 100K+ requests per day and have the engineering team to support it, the math works. If you’re just getting started, a hosted API is simpler and still affordable.

What to Watch Out For

1. Model Freshness Lag

DeepSeek releases new model versions periodically. When a new version drops, there is a lag between the hosted API switching to it and you updating your self-hosted deployment.

Beginner-friendly advice: Subscribe to the DeepSeek release feed. Run a canary deployment that tests new versions against your benchmark suite. Automate the model download and deployment pipeline so updates are push-button, not manual.

2. Asymmetric GPU Utilization

The 32B tier runs at ~60% utilization during peak hours and ~15% during off-peak. The full V3 tier is even more asymmetric: ~40% peak, ~5% off-peak. You’re paying for GPUs that sit idle most of the time.

Beginner-friendly advice: Use HPA-based autoscaling that scales down to 0 replicas during off-peak hours. Consider spot instance preemptible workers for the 32B tier. Set up a batch queue that fills off-peak capacity with non-urgent inference tasks.

3. Operational Complexity

Self-hosting adds significant operational overhead compared to an API call. You need:

Kubernetes cluster management (we use EKS).
GPU node management (we use Karpenter for auto-scaling).
Model storage and versioning.
Monitoring and alerting for model quality degradation.
A team member on-call for inference infrastructure.

Beginner-friendly advice: Start with a managed Kubernetes service (EKS, GKE, AKS). Use KubeRay to simplify Ray cluster management. Don’t try to build everything at once — get the 32B model running first, then add the full V3 tier.

4. Readiness Probe Death Spiral

We discovered this the hard way. When vLLM is loading a model (which takes 2-5 minutes for the 32B model and 15-20 minutes for the full V3), the readiness probe fails. Kubernetes then restarts the pod, which triggers another model load, creating an infinite restart loop.

Fix: Use startup probes instead of readiness probes for the initial model load:

startupProbe:
  httpGet:
    path: /health
    port: 8000
  initialDelaySeconds: 30
  periodSeconds: 10
  failureThreshold: 60  # Allow up to 10 minutes for model load
readinessProbe:
  httpGet:
    path: /health
    port: 8000
  periodSeconds: 5
  failureThreshold: 3

5. Shared Memory Exhaustion

vLLM uses shared memory for inter-process communication. The default /dev/shm size (64MB in most Kubernetes configurations) is far too small. We had to mount an emptyDir with medium: Memory and set sizeLimit to at least 64GB for the 32B tier and 256GB for the full V3 tier.

Beginner-friendly advice: Always check your shared memory settings before deploying. This is one of the most common gotchas with vLLM on Kubernetes.

6. Inter-Node Networking

The full V3 model requires 8 H100 GPUs. If your cluster doesn’t have a single node with 8 GPUs (many don’t), you need to split across nodes. This requires high-bandwidth inter-node networking (InfiniBand or at minimum 100 Gbps EFA). We initially tried 25 Gbps EFA and saw 3x worse throughput due to all-reduce communication overhead.

Beginner-friendly advice: If possible, use a single node with 8 GPUs. If you must split across nodes, invest in high-bandwidth networking. This is not an area to cut costs.

Course-Style Deep Dive

Multi-Head Latent Attention (MLA)

Think of attention in AI models like a spotlight at a concert. Traditional attention shines a bright spotlight on every performer at once — it’s powerful but uses a lot of energy. MLA is like having a dimmer spotlight that only brightens when it needs to.

DeepSeek-V3’s MLA is the key innovation that makes the 671B model practical. Traditional multi-head attention (MHA) stores a full KV cache for each attention head, which for a 671B model with 128K context would require terabytes of memory.

MLA compresses the KV cache by projecting keys and values into a low-dimensional latent space:

Traditional MHA:
  K, V ∈ R^(n_heads × d_head × seq_len)
  Storage: O(n_heads × d_head × seq_len)

MLA:
  k_latent = W_k_proj @ x  (compress to latent dim d_latent)
  v_latent = W_v_proj @ x  (compress to latent dim d_latent)
  Storage: O(d_latent × seq_len) where d_latent << n_heads × d_head

For DeepSeek-V3:

n_heads = 128 (attention heads)
d_head = 128 (head dimension)
d_latent = 64 (latent dimension, shared across heads)

KV cache reduction: 128 × 128 / 64 = 256x reduction per layer. With 67 layers, the total reduction is approximately 256x, but the actual savings are about 98.6% because the latent space is shared across layers for the value cache.

Practical impact: Without MLA, a 671B model with 128K context would need ~2.4 TB of KV cache per request. With MLA, it needs ~34 GB. This is what makes serving the model on 8x H100s feasible.

Analogy: Imagine you’re writing a book and need to remember every word you’ve written. Without MLA, you’d keep a full copy of every page. With MLA, you keep a summary — a few keywords per page. You lose some detail, but you save a massive amount of memory.

DeepSeekMoE Architecture

DeepSeek-V3 uses a Mixture of Experts (MoE) architecture. Think of it like a hospital with 256 specialist doctors. For any given patient (token), only 8 doctors are called in — the ones best suited for that case.

Total parameters: 671B
Active parameters per token: 37B (8 experts × 4.6B per expert)
Experts: 256 (shared + routed)
Top-k routing: 8 (6 routed + 2 shared)

The shared experts are always active and handle common patterns. The routed experts specialize in different domains. The router network selects which experts to activate based on the input token:

# Simplified MoE routing
def moe_forward(x: torch.Tensor, router: nn.Linear, experts: list[nn.Module]):
    # x: (batch, seq_len, d_model)
    # router: projects to expert logits
    logits = router(x)  # (batch, seq_len, n_experts)
    weights, indices = torch.topk(logits, k=8, dim=-1)
    weights = torch.softmax(weights, dim=-1)

    output = torch.zeros_like(x)
    for i, expert in enumerate(experts):
        mask = (indices == i).any(dim=-1)
        if mask.any():
            output[mask] += weights[mask] * expert(x[mask])

    return output

Load balancing: DeepSeek-V3 uses an auxiliary loss to ensure tokens are distributed evenly across experts. Without this, a few experts would handle most tokens while others remain idle — like a hospital where two doctors do all the work while 254 sit around.

Multi-Token Prediction (MTP)

DeepSeek-V3 introduces MTP, where the model predicts multiple future tokens simultaneously rather than one at a time:

Traditional: P(t_{n+1} | t_1, ..., t_n)
MTP: P(t_{n+1}, t_{n+2}, ..., t_{n+k} | t_1, ..., t_n)

MTP uses a shared trunk with k independent prediction heads. During training, all k heads are trained jointly. During inference, only the first head is used (the others are discarded), but the training signal from MTP produces better representations that improve single-token prediction quality.

Why it works: MTP forces the model to learn longer-range dependencies during training. The model can’t just predict the next token based on local patterns; it must understand the broader context to predict multiple tokens ahead.

Analogy: It’s like learning to play chess by predicting not just your next move, but your next three moves. Even if you only use the first move in actual play, the practice of thinking ahead makes your single-move decisions better.

Prefix Caching (APC)

vLLM’s automatic prefix caching stores the KV cache for common prefixes (like system prompts) and reuses them across requests:

Request 1: [system_prompt] + [user_message_1]
Request 2: [system_prompt] + [user_message_2]

Without APC:
  Request 1: Compute KV for [system_prompt] + [user_message_1]
  Request 2: Compute KV for [system_prompt] + [user_message_2] (redundant)

With APC:
  Request 1: Compute KV for [system_prompt] + [user_message_1]
  Request 2: Reuse KV for [system_prompt], compute only [user_message_2]

The cache is a hash table keyed by token IDs. When a new request arrives, vLLM computes the longest matching prefix and reuses its KV cache. This is especially effective for:

Shared system prompts across requests.
Few-shot examples that are the same across requests.
Conversation history in multi-turn chats.

Analogy: It’s like having a whiteboard where you keep the instructions written at the top. Instead of erasing and rewriting them for every new student, you just add the new question below.

Expert Parallelism

For the full V3 model, we use expert parallelism (EP) in addition to tensor parallelism (TP). EP distributes experts across GPUs, so each GPU only loads a subset of experts:

Without EP (TP=8):
  Each GPU: 671B / 8 = 84B parameters
  Memory: ~168 GB per GPU (FP8) — doesn't fit on 80GB H100

With EP (TP=4, EP=2):
  Each GPU: (active 37B / 4 TP) + (experts / 2 EP) = ~65 GB per GPU
  Fits on 80GB H100 with room for KV cache

The trade-off: EP adds communication overhead because tokens may need to be sent to the GPU hosting the required expert. DeepSeek-V3 mitigates this with a custom all-to-all communication kernel optimized for NVLink.

Analogy: Imagine a library where books are spread across multiple rooms. Without EP, every room has a copy of every book (lots of duplication, lots of space). With EP, each room only has the books it needs (less space, but you might need to walk to another room to find a specific book).

Prometheus Alerting

We monitor the following metrics and alert on anomalies:

# prometheus-alerts.yaml
groups:
- name: deepseek-inference
  rules:
  - alert: HighP95Latency
    expr: histogram_quantile(0.95, rate(vllm:request_latency_seconds_bucket[5m])) > 1.0
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "P95 latency > 1s on {{ $labels.model }}"

  - alert: HighErrorRate
    expr: rate(vllm:request_errors_total[5m]) / rate(vllm:request_count_total[5m]) > 0.01
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "Error rate > 1% on {{ $labels.model }}"

  - alert: GPUMemoryPressure
    expr: nvidia_gpu_memory_used_bytes / nvidia_gpu_memory_total_bytes > 0.95
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: "GPU memory > 95% on {{ $labels.gpu }}"

  - alert: ModelLoadingTimeout
    expr: time() - vllm:model_load_timestamp_seconds > 600
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: "Model {{ $labels.model }} taking > 10 minutes to load"

  - alert: KVCacheEvictionRate
    expr: rate(vllm:kv_cache_evictions_total[5m]) > 100
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "High KV cache eviction rate on {{ $labels.model }}"

Autoscaling with HPA

We use Kubernetes HPA with custom metrics from Prometheus:

# hpa-32b.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: deepseek-32b-hpa
  namespace: ray
spec:
  scaleTargetRef:
    apiVersion: ray.io/v1
    kind: RayCluster
    name: deepseek-32b
  minReplicas: 1
  maxReplicas: 3
  metrics:
  - type: Pods
    pods:
      metric:
        name: nvidia_gpu_utilization
      target:
        type: AverageValue
        averageValue: 70
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Percent
        value: 50
        periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 60
      policies:
      - type: Pods
        value: 1
        periodSeconds: 30

Integration Patterns

Qdrant for vector search: We use Qdrant for semantic search over our documentation. The 32B model generates embeddings that are stored in Qdrant and retrieved at query time.

Argo Workflows for batch inference: For non-urgent batch jobs (nightly documentation indexing, weekly code analysis), we use Argo Workflows to submit batch inference requests to the 32B tier during off-peak hours.

Open WebUI for internal chat: Our team uses Open WebUI (configured to point at our router service) for internal Q&A. The router handles model selection transparently.

# Open WebUI config (docker-compose override)
services:
  open-webui:
    environment:
      - OPENAI_API_BASE_URL=http://router-svc.ray:8000/v1
      - OPENAI_API_KEY=not-needed
      - DEFAULT_MODEL=deepseek-32b
      - MODELS_FILTER=deepseek-32b,deepseek-v3,deepseek-reasoner

Conclusion

Self-hosting DeepSeek on Kubernetes with KubeRay has been transformative for our inference infrastructure. The 89.8% cost reduction alone justified the migration, but the latency improvements, data sovereignty, and elimination of third-party outages have been equally valuable.

The key takeaways:

Start with the distilled model. The 32B variant handles 78% of our workloads with quality indistinguishable from GPT-4o.
Invest in the router. Smart routing between tiers is the single biggest lever for cost optimization.
Prefix caching is non-negotiable. Without it, TTFT is unacceptable for interactive workloads.
Plan for operational complexity. Self-hosting is cheaper but requires dedicated infrastructure engineering.
Monitor everything. Model quality degradation, GPU memory pressure, and KV cache eviction rates are the metrics that matter.

The full deployment configuration, router service, and monitoring setup are available in our infrastructure repository. We welcome contributions and feedback from the community.