·15 min read

Dhi: A self-hostable, MIT-licensed VS Code extension with FIM ghost-text autocomplete for 6 languages and RAG-based context retrieval

A self-hostable VS Code extension with FIM ghost-text autocomplete for 6 languages and RAG-based context retrieval — fully free with no API keys required.

#dhi#vscode-extension#fim#autocomplete#self-hosted

The Problem: Every AI Code Completion Tool Asks for Your Credit Card

You open VS Code, install an AI extension, and the first thing it asks for is an API key. Or a GitHub login. Or a $20/month subscription. Cursor charges $20/month. GitHub Copilot charges $10-19/month. Even Continue.dev, which is open source, requires you to bring your own API key and pay per token to a cloud provider.

The problem is structural. Every managed AI code completion service runs inference on someone else’s GPU cluster. That cluster costs money — electricity, cooling, hardware depreciation. Those costs get passed to you as a monthly subscription or per-token pricing. If you work on multiple machines, you pay multiple times. If you have an air-gapped environment (defense, finance, healthcare), you cannot use any of them at all.

The alternative — running a model locally — has historically been terrible. The setup involves installing Python, CUDA (NVIDIA’s parallel computing platform), a model server, and wiring it all together by hand. The first time you try, you spend an afternoon debugging environment variables and CUDA version mismatches before you see a single completion.

The metrics that matter:

Dimension Managed Services (Copilot, Cursor) DIY Local Setup Dhi
Monthly cost $10-20/user $0 (electricity only) $0 (or ~$3/mo for shared GPU)
Setup time 2 minutes 2-4 hours 5 minutes
API key required Yes No No
Air-gapped capable No Yes Yes
Open source No Yes Yes (MIT)
FIM autocomplete Yes Manual build Yes
RAG context retrieval Proprietary Manual build Yes
Multi-language support 10+ languages Varies 6 languages
Maintenance burden Zero High Low (Docker)

Why this matters: The barrier to AI-assisted coding is not model quality — it’s infrastructure. Dhi collapses the setup from a multi-hour engineering project to a single docker compose up command, while keeping your data on your machine and your wallet in your pocket.

The Investigation: Why Local Code Completion Has Been So Hard

The root cause of the local AI coding problem is not that models are bad — StarCoder2-3B scores 46% on HumanEval (a benchmark measuring functional correctness of generated Python code), and Qwen2.5-Coder-32B hits 90%. The problem is that the tooling around these models is fragmented and poorly documented.

Finding 1: FIM (Fill-in-the-Middle) is not the same as chat completion.

Most developers assume you can take any code model and ask it to “complete this function.” That is not how FIM works. FIM requires a specific prompt format where the model sees a prefix (everything before the cursor) and a suffix (everything after the cursor), then generates the bridging content. Each model family uses different special tokens:

  • StarCoder2: <fim_prefix>{prefix}<fim_suffix>{suffix}<fim_middle>
  • DeepSeek-Coder: <｜fim▁begin｜>#<｜fim▁begin｜># prefix\n{prefix}\n## suffix\n{suffix}\n## middle\n
  • Qwen2.5-Coder: <|fim_prefix|>{prefix}<|fim_suffix|>{suffix}<|fim_middle|>

What this means: You cannot just point a generic LLM (Large Language Model) at your code and expect autocomplete. You need model-specific prompt formatting, correct stop tokens, and proper context window management. Dhi handles all of this in server/inference/fim.py, with a model registry in models/registry.yaml that maps each model tag to its FIM template.

Finding 2: Context is everything, but naive context is worse than no context.

Early local autocomplete tools simply fed the current file to the model. This misses cross-file references — function calls defined in other modules, imported types, shared constants. The result is completions that look plausible but reference symbols that do not exist.

Dhi’s solution is a RAG (Retrieval-Augmented Generation) pipeline that indexes the entire workspace. When you trigger a completion, it retrieves the top-3 most relevant chunks from across your codebase and injects them into the FIM prompt. The retrieval uses a hybrid of dense vector search (ChromaDB + nomic-embed-text, a 768-dimensional embedding model) and sparse keyword search (BM25), fused via Reciprocal Rank Fusion (RRF):

RRF score = sum(1 / (k + rank_i)) for each result in each ranking

What this means: A completion for a function that calls calculate_total() in another file will actually know the signature and behavior of calculate_total(). The RRF fusion ensures that both semantic similarity (vector search) and exact keyword matches (BM25) contribute to the final ranking.

Finding 3: Chunking by line count destroys semantic meaning.

Most naive RAG systems split files every N lines. This produces chunks that start in the middle of one function and end in the middle of another. The embedding for such a chunk is noisy and retrieval quality suffers.

Dhi uses Tree-sitter, a parser that produces a concrete syntax tree (CST) for 40+ languages in under 5ms per file. The chunker splits on semantic boundaries — function declarations, class definitions, method bodies. Each chunk carries metadata: file path, line range, and symbol name. This metadata is stored alongside the embedding in Chroma and used during retrieval to deduplicate and rank results.

Finding 4: Latency expectations are different for local vs. cloud.

Cloud services like Copilot deliver completions in 200-500ms. Local models on CPU take 2-15 seconds. This is not a failure — it is a hardware constraint. The key insight is that 2-5 second completions are still useful if they are correct, because the alternative (writing the code yourself) takes longer.

Dhi’s model tier system lets you choose your latency/quality tradeoff:

Tier Model VRAM CPU Latency GPU Latency HumanEval
CPU (recommended start) starcoder2:1b 0 GB 2-5 s — 27%
Default starcoder2:3b 6 GB 8-15 s < 1 s 46%
Quality deepseek-coder-v2:16b 12 GB — 1-2 s 73%
Max qwen2.5-coder:32b 24 GB — 2-4 s 90%

What this means: If you have a GPU, you get sub-second completions with quality comparable to cloud services. If you are on CPU, you get 2-5 second completions with the 1B model — slower, but functional and free. The 1B model’s 27% HumanEval is low, but for boilerplate code, simple function bodies, and repetitive patterns, it is surprisingly effective.

The Solution: A Two-Container Stack That Replaces Your Copilot Subscription

Dhi’s architecture is deliberately minimal. Two containers, one VS Code extension, zero API keys.

┌─────────────────────────────────────────────────────────────┐
│  VS Code Extension (TypeScript)                             │
│  ┌──────────────────┐  ┌──────────────┐  ┌──────────────┐  │
│  │  FIM Provider     │  │  Chat Panel  │  │  Agent View  │  │
│  │  (ghost text)     │  │  (Post 3)    │  │  (Post 4)    │  │
│  └────────┬─────────┘  └──────┬───────┘  └──────┬───────┘  │
│           └───────────────────┼──────────────────┘           │
│                        DhiClient                            │
│                   (single HTTP client)                       │
└──────────────────────────────┼──────────────────────────────┘
                               │ HTTP (localhost:8000)
┌──────────────────────────────▼──────────────────────────────┐
│  FastAPI Server (Python) — Container 1                      │
│  ┌────────────────────┐  ┌────────────────────┐            │
│  │  POST /complete    │  │  POST /index       │            │
│  │  POST /search      │  │  GET /health       │            │
│  └────────┬───────────┘  └────────┬───────────┘            │
│           │                       │                         │
│  ┌────────▼───────────┐  ┌───────▼────────────┐            │
│  │  inference/fim.py  │  │  rag/chunker.py    │            │
│  │  (FIM prompt fmt)  │  │  rag/store.py      │            │
│  └────────┬───────────┘  └───────┬────────────┘            │
└───────────┼──────────────────────┼────────────────────────┘
            │                      │
┌───────────▼──────────┐  ┌───────▼────────────┐
│  Ollama — Container 2│  │  ChromaDB          │
│  ┌─────────────────┐ │  │  (vector store)    │
│  │ starcoder2:3b   │ │  │  (Docker volume)   │
│  │ nomic-embed-text│ │  └────────────────────┘
│  └─────────────────┘ │
└───────────────────────┘

Here is what each piece does:

  • VS Code Extension (TypeScript): Contains the FIM Provider that implements VS Code’s InlineCompletionItemProvider interface. It listens for keystrokes, debounces them (default 150ms), snapshots the prefix and suffix around the cursor, and sends them to the server. All HTTP calls go through a single DhiClient class — no provider calls fetch() directly.

  • FastAPI Server (Python): Exposes three endpoints. POST /complete accepts {file_path, prefix, suffix, language} and returns a completion. POST /index accepts a directory path and indexes all supported files. GET /health returns {"status":"ok"} for health checks.

  • Ollama: Serves the FIM model (default starcoder2:3b) and the embedding model (nomic-embed-text). The FIM model generates completions. The embedding model converts code chunks into 768-dimensional vectors for the RAG pipeline.

  • ChromaDB: The vector store. Stores embeddings and their metadata (file path, line range, symbol name). Persisted via a Docker volume so the index survives container restarts.

  • Layer rules enforced by ruff: ChunkStore is the only module that imports chromadb. Service functions receive all dependencies as arguments — no module-level singletons except in main.py. This keeps the dependency graph clean and testable.

Production-Grade Code Walkthrough: The FIM Completion Endpoint

The core of Dhi is the /complete endpoint. Here is the server-side implementation pattern:

# server/inference/fim.py
from typing import Optional
import httpx
from pydantic import BaseModel

class CompletionRequest(BaseModel):
    file_path: str
    prefix: str
    suffix: str
    language: str

class CompletionResponse(BaseModel):
    text: str
    model: str
    latency_ms: float

class FIMService:
    """Handles FIM prompt construction and Ollama inference."""

    def __init__(self, ollama_base_url: str, model: str, max_tokens: int):
        self.ollama_base_url = ollama_base_url
        self.model = model
        self.max_tokens = max_tokens
        self._fim_template = self._load_fim_template(model)

    def _load_fim_template(self, model: str) -> dict:
        """Load FIM template from the model registry.

        Each model family uses different special tokens for FIM.
        The registry maps model tags to their template strings.
        """
        # In production, this reads from models/registry.yaml
        templates = {
            "starcoder2": {
                "prefix": "<fim_prefix>",
                "suffix": "<fim_suffix>",
                "middle": "<fim_middle>",
                "stop": ["<fim_prefix>", "<fim_suffix>", "<fim_middle>",
                         "<|endoftext|>", "<file_sep>"],
            },
            "deepseek-coder": {
                "prefix": "<｜fim▁begin｜># # prefix\n",
                "suffix": "\n# # suffix\n",
                "middle": "\n# # middle\n",
                "stop": ["\n# # prefix", "\n# # suffix", "\n# # middle"],
            },
        }
        for key, template in templates.items():
            if key in model:
                return template
        raise ValueError(f"No FIM template for model: {model}")

    def build_prompt(self, prefix: str, suffix: str,
                     context_chunks: Optional[list[str]] = None) -> str:
        """Build the FIM prompt with optional RAG context injection.

        When context chunks are available, they are injected before the
        FIM prefix-suffix pair, separated by <file_sep> tokens for
        StarCoder2 models.
        """
        t = self._fim_template
        if context_chunks:
            context_block = "<file_sep>\n".join(context_chunks)
            return f"{context_block}\n<file_sep>\n{t['prefix']}{prefix}{t['suffix']}{suffix}{t['middle']}"
        return f"{t['prefix']}{prefix}{t['suffix']}{suffix}{t['middle']}"

    async def complete(self, request: CompletionRequest,
                       context_chunks: Optional[list[str]] = None) -> CompletionResponse:
        """Send the FIM prompt to Ollama and return the completion."""
        import time
        prompt = self.build_prompt(request.prefix, request.suffix, context_chunks)
        start = time.monotonic()

        async with httpx.AsyncClient(timeout=120.0) as client:
            response = await client.post(
                f"{self.ollama_base_url}/api/generate",
                json={
                    "model": self.model,
                    "prompt": prompt,
                    "options": {
                        "num_predict": self.max_tokens,
                        "stop": self._fim_template["stop"],
                        "temperature": 0.2,
                        "top_p": 0.95,
                    },
                    "stream": False,
                },
            )
            response.raise_for_status()
            result = response.json()

        elapsed_ms = (time.monotonic() - start) * 1000
        return CompletionResponse(
            text=result["response"].strip(),
            model=self.model,
            latency_ms=elapsed_ms,
        )

And the VS Code extension side that triggers completions:

// extension/src/fimProvider.ts
import * as vscode from 'vscode';
import { DhiClient } from './dhiClient';

export class DhiFIMProvider implements vscode.InlineCompletionItemProvider {
    private client: DhiClient;
    private debounceTimer: ReturnType<typeof setTimeout> | null = null;
    private pendingRequest: AbortController | null = null;

    constructor(client: DhiClient) {
        this.client = client;
    }

    async provideInlineCompletionItems(
        document: vscode.TextDocument,
        position: vscode.Position,
        context: vscode.InlineCompletionContext,
        token: vscode.CancellationToken
    ): Promise<vscode.InlineCompletionItem[] | undefined> {
        // Only trigger on explicit typing, not on paste or undo
        if (context.triggerKind !== vscode.InlineCompletionTriggerKind.Automatic) {
            return undefined;
        }

        // Check language support
        const supportedLanguages = ['python', 'typescript', 'javascript',
                                    'go', 'rust', 'java'];
        if (!supportedLanguages.includes(document.languageId)) {
            return undefined;
        }

        // Extract prefix (text before cursor) and suffix (text after cursor)
        const prefix = document.getText(
            new vscode.Range(
                new vscode.Position(Math.max(0, position.line - 20), 0),
                position
            )
        );
        const suffix = document.getText(
            new vscode.Range(
                position,
                new vscode.Position(
                    Math.min(document.lineCount - 1, position.line + 10),
                    document.lineAt(Math.min(
                        document.lineCount - 1, position.line + 10
                    )).text.length
                )
            )
        );

        // Cancel any in-flight request
        if (this.pendingRequest) {
            this.pendingRequest.abort();
        }
        this.pendingRequest = new AbortController();

        try {
            const response = await this.client.complete({
                filePath: document.uri.fsPath,
                prefix: prefix.slice(-256),   // MAX_PREFIX_CHARS
                suffix: suffix.slice(0, 128),  // MAX_SUFFIX_CHARS
                language: document.languageId,
            }, this.pendingRequest.signal);

            if (!response?.text) return undefined;

            return [
                new vscode.InlineCompletionItem(
                    response.text,
                    new vscode.Range(position, position)
                )
            ];
        } catch (error) {
            if (error instanceof DOMException &&
                error.name === 'AbortError') {
                return undefined; // User kept typing, ignore
            }
            console.error('[Dhi] Completion error:', error);
            return undefined;
        }
    }
}

Setup Instructions

# Step 1: Clone the repository
git clone https://github.com/sochaty/dhi
cd dhi

# Step 2: Run the bootstrap script (detects GPU, pulls models, starts containers)
./scripts/bootstrap.sh

# On Windows (PowerShell), use docker compose directly:
# docker compose up -d

# Step 3: Verify the server is running
curl http://localhost:8000/health
# Expected: {"status":"ok"}

# Step 4: Install the VS Code extension
# Download dhi-0.1.0.vsix from GitHub Releases
# VS Code → Extensions → ⋯ → Install from VSIX...

# Step 5: (Optional) Index your workspace for RAG context
# Ctrl+Shift+P → Dhi: Index Workspace

# Step 6: Start coding
# Open a .py, .ts, .js, .go, .rs, or .java file and start typing

For CPU-only machines, create a .env file before starting the stack:

# .env — switch to the 1B model for CPU
FIM_MODEL=starcoder2:1b
EMBED_MODEL=nomic-embed-text
FIM_MODEL_MAX_TOKENS=10

How to Use Effectively

Step 1: Choose Your Model Tier Based on Hardware

Run docker compose logs ollama | head -5 after startup to confirm the model loaded. If completions take longer than 10 seconds, switch to starcoder2:1b by setting FIM_MODEL=starcoder2:1b in .env and running docker compose up -d again.

The 1B model produces shorter, simpler completions but runs on any machine. The 3B model is the default and balances quality with resource usage. The 16B and 32B models require a GPU with sufficient VRAM (12 GB and 24 GB respectively).

Step 2: Index Your Workspace for Cross-File Context

Run Ctrl+Shift+P → Dhi: Index Workspace. This reads every supported file in your project, parses it with Tree-sitter into semantic chunks (functions, classes, methods), embeds each chunk with nomic-embed-text, and stores the vectors in Chroma.

The index is persisted in a Docker volume. You only need to re-index when you add or significantly restructure files. The index command is idempotent — running it twice does not duplicate entries.

Step 3: Configure the Debounce and Server URL

Open VS Code settings (Ctrl+,) and search for dhi:

{
    "dhi.serverUrl": "http://localhost:8000",
    "dhi.completionEnabled": true,
    "dhi.completionDebounceMs": 150
}

The debounce controls how long the extension waits after your last keystroke before requesting a completion. 150ms is the default. Increase to 300ms on slower machines to reduce the number of requests. Decrease to 50ms if you want more aggressive completions and have a fast GPU.

Step 4: Accept and Dismiss Completions

When ghost text appears, press Tab to accept the full completion. Press Escape to dismiss it. If you keep typing, the ghost text disappears automatically and a new completion is requested after the debounce period.

Multi-line completions are supported. The model generates up to FIM_MODEL_MAX_TOKENS tokens (default 10), which typically produces 1-5 lines of code depending on the language and context.

Step 5: Monitor Performance via the Output Channel

Open View → Output and select Dhi from the dropdown. This shows every completion request, its latency, and any errors. Use this to diagnose slow completions, connection issues, or malformed requests.

[Dhi] Request: POST /complete (file: routes/auth.py, lang: python)
[Dhi] Response: 842ms, 12 tokens, model: starcoder2:3b
[Dhi] Completion: "    return authenticate_user(email, password)"

Use Cases

1. Solo Developer on a Budget

When you would use this: You are a freelance developer or indie hacker who cannot justify a $20/month Copilot subscription. You work on a laptop with no GPU and want AI assistance without ongoing costs.

Why Dhi fits: Zero monthly cost. The 1B model runs on CPU with 2-5 second latency. For boilerplate code (getters, setters, CRUD operations, API route handlers), the completions are accurate enough to save time. The RAG index gives you cross-file context even on a single laptop.

2. Air-Gapped Development Environment

When you would use this: You work in defense, finance, healthcare, or any environment where code cannot leave the machine. Cloud-based AI tools are prohibited by policy.

Why Dhi fits: Everything runs locally. No API keys, no external network calls, no data leaves your machine. The Docker stack can be pre-loaded with model weights on an air-gapped build machine and distributed as a tarball. The MIT license means no legal review for open-source usage.

3. Team Standardization Across Hardware

When you would use this: Your team of 20 developers has a mix of MacBooks, Linux workstations, and Windows machines. Some have GPUs, most do not. You want everyone using the same AI tool without managing 20 API keys.

Why Dhi fits: The Docker stack is identical across platforms. Developers with GPUs get sub-second completions with the 3B model. Developers without GPUs use the 1B model. Everyone uses the same VS Code extension and the same server configuration. The shared GPU pool option (~$3/month) is planned for teams that want a middle ground.

4. Learning a New Language or Framework

When you would use this: You are learning Rust, Go, or Java and want AI assistance to see idiomatic patterns as you type. You need completions that respect the language’s conventions, not just generic code.

Why Dhi fits: The Tree-sitter chunker understands the AST of each supported language. The FIM model (StarCoder2) was trained on permissively licensed code across all six languages. Completions reflect the idioms of each language — Rust’s ownership patterns, Go’s error handling, Java’s class structure.

5. Prototyping and Rapid Iteration

When you would use this: You are building a proof of concept and writing a lot of boilerplate — database models, API endpoints, test fixtures. You want to type the signature and have the body filled in.

Why Dhi fits: The FIM model excels at completing function bodies given a descriptive name and parameter types. Combined with RAG context from your existing codebase, completions match your project’s existing patterns. A function called create_user in a FastAPI app will generate the correct ORM (Object-Relational Mapping) calls based on your model definitions.

Cheat Sheet

Aspect Detail
Repository github.com/sochaty/dhi
License MIT
Primary language Python (72.8%), TypeScript (19.0%)
GPU requirements None (CPU mode with 1B model); 6 GB VRAM for default 3B model
Setup time 5 minutes (clone + bootstrap)
Key features FIM ghost-text autocomplete, RAG context retrieval, Tree-sitter chunking, BM25 hybrid search
Supported languages Python, TypeScript/TSX, JavaScript, Go, Rust, Java
Default model starcoder2:3b (1.7 GB download)
Embedding model nomic-embed-text (768-dim vectors)
Vector store ChromaDB (Docker volume persisted)
Search modes Hybrid (vector + BM25 + RRF), vector-only, BM25-only
VS Code settings dhi.serverUrl, dhi.completionEnabled, dhi.completionDebounceMs
Server env vars FIM_MODEL, EMBED_MODEL, FIM_MODEL_MAX_TOKENS, OLLAMA_TIMEOUT, MAX_PREFIX_CHARS, MAX_SUFFIX_CHARS
Common gotcha 422 error on /complete means missing file_path, prefix, suffix, or language in the request body
Common gotcha Chroma errors on restart require docker compose down -v to clear the persisted volume
Common gotcha First model download is ~1.7 GB and takes 5-15 minutes depending on bandwidth
Common gotcha Docker Desktop needs at least 6 GB RAM allocated in settings
Status Active development — FIM and RAG are stable; chat and agent features are in progress

Vibe Coding Projects

Project 1: Build a Personal Code Assistant Dashboard

What it does: Set up Dhi on your primary development machine, index your three most-used repositories, and configure different model tiers for each. Create a simple shell script that reports completion latency, cache hit rate, and model used for the last 100 completions.

What you will learn: Docker Compose networking, VS Code extension configuration, model tier selection based on project complexity, and basic observability for AI inference.

Effort: 2-3 hours. Most of the time is waiting for model downloads and indexing.

Project 2: Extend Dhi with a Custom Language

What it does: Fork the Dhi repository and add support for a seventh language (e.g., C#, Ruby, or PHP). This requires adding a Tree-sitter grammar to the chunker, adding the language to the supported languages list in the FIM provider, and testing that completions work for that language.

What you will learn: Tree-sitter grammar configuration, VS Code language detection, FIM prompt engineering for different syntax styles, and the Dhi contribution workflow (ruff checks, pytest, eslint).

Effort: 4-6 hours. The Tree-sitter grammar integration is straightforward; the main effort is testing completions across different code patterns in the new language.

Project 3: Deploy Dhi as a Team Inference Server

What it does: Set up Dhi’s FastAPI server on a shared machine with a GPU (e.g., a workstation or cloud VM with a single A100). Configure it so that multiple developers on your team can point their VS Code extensions at the shared server. Add a simple rate limiter and request queue so that 5 developers can share one GPU without timeouts.

What you will learn: Multi-user inference serving, request queuing, GPU memory management across concurrent requests, and the difference between serving FIM (low latency, small batches) and serving chat (high throughput, large batches).

Effort: 8-12 hours. The core challenge is managing Ollama’s single-request-at-a-time limitation and implementing a queue that does not add more than 500ms overhead per request.

Problems Solved Efficiently

Problem Type Why Dhi Fits When to Look Elsewhere
Writing boilerplate code (getters, setters, CRUD, API routes) FIM models excel at completing predictable patterns from function signatures For complex algorithmic code with no existing patterns in the codebase
Learning idiomatic patterns in a new language Tree-sitter chunking preserves language-specific AST structure; StarCoder2 was trained on permissively licensed code in all 6 languages For languages not in the supported 6 (C#, Ruby, PHP, Swift, Kotlin)
Cross-file refactoring with consistent patterns RAG context retrieval surfaces relevant types and functions from across the workspace For multi-file agent editing (planned in Post 4, not yet available)
Air-gapped or compliance-constrained environments Zero network calls, no API keys, everything runs in local Docker containers For teams that need a managed SLA (service level agreement) with guaranteed uptime
Cost-sensitive individual developers Free self-hosted option with no per-token pricing For developers who want cloud-quality latency on CPU hardware (2-5s vs 200ms)
Prototyping and rapid iteration Fast completions for common patterns reduce keystrokes by 30-50% for boilerplate For production code that requires 100% correctness — always review AI-generated code

Architectural Tradeoffs

What we gained

  • Complete data sovereignty. Every inference runs on your hardware. No code snippets are sent to a third-party API. For regulated industries, this is the difference between “approved” and “blocked by legal.”

  • Zero recurring cost. After the initial setup (clone, bootstrap, download models), there are no monthly bills. The electricity cost of running a 3B model on a GPU is negligible compared to a $20/month subscription.

  • Deterministic behavior. The same prefix and suffix with the same model and temperature produce the same completion. Cloud services can change their underlying model without notice, altering completions overnight.

  • Offline capability. Once the models are downloaded, Dhi works without internet access. This is critical for developers who work on planes, trains, or in areas with unreliable connectivity.

What we sacrificed

  • Latency. Cloud services deliver completions in 200-500ms. Dhi on CPU takes 2-15 seconds. Even on GPU, the 3B model takes ~1 second versus Copilot’s ~300ms. This is a hardware constraint, not a software one.

  • Model quality ceiling. The best model Dhi supports (Qwen2.5-Coder-32B, 90% HumanEval) requires 24 GB VRAM. Cloud services can run 70B+ parameter models that achieve higher scores. You are trading model quality for cost and privacy.

  • No multi-file editing (yet). The agent-based multi-file editing feature is planned for Post 4 but not yet implemented. Cursor’s agent mode and Copilot’s Workspace are ahead here.

  • Maintenance burden. You are responsible for keeping the Docker stack running, updating models, and troubleshooting issues. Cloud services abstract all of this away. The bootstrap script handles the initial setup, but you own the ongoing operations.

The real lesson: Dhi is not trying to beat Copilot on latency or model quality. It is solving a different problem: making AI code completion available to everyone, everywhere, regardless of budget, internet access, or corporate policy. If you can afford $20/month and have no compliance constraints, use Copilot. If you cannot or will not, Dhi is your answer.

Course-Style Deep Dive

How FIM Works Under the Hood

FIM is fundamentally different from standard language model generation. In standard generation, the model sees text from left to right and predicts the next token. In FIM, the model sees a prefix (text before the cursor) and a suffix (text after the cursor), and must generate the tokens that bridge them.

The training procedure for FIM models is called “causal masking with span corruption.” During training, a random span of code is replaced with a sentinel token (e.g., <fim_middle>). The model learns to predict the masked span given the surrounding context. At inference time, the cursor position acts as the mask boundary.

StarCoder2’s FIM template uses three special tokens:

<fim_prefix>{prefix}<fim_suffix>{suffix}<fim_middle>

The model generates tokens after <fim_middle> until it produces a stop token (<fim_prefix>, <fim_suffix>, <fim_middle>, or <|endoftext|>). The generated text is the completion.

Dhi’s implementation in server/inference/fim.py handles the model-specific template selection through a registry:

# models/registry.yaml (conceptual structure)
models:
  starcoder2:3b:
    family: starcoder2
    fim_template:
      prefix: "<fim_prefix>"
      suffix: "<fim_suffix>"
      middle: "<fim_middle>"
    stop_tokens:
      - "<fim_prefix>"
      - "<fim_suffix>"
      - "<fim_middle>"
      - "<|endoftext|>"
      - "<file_sep>"
    context_window: 8192
    recommended_max_tokens: 10

Advanced Pattern 1: Hybrid Search with Reciprocal Rank Fusion

Dhi’s RAG pipeline combines three retrieval strategies into a single ranked result set. Here is the implementation pattern:

# server/rag/search.py
import math
from typing import List, Tuple
import chromadb
from chromadb.utils import embedding_functions

class HybridSearch:
    """Combines vector search, BM25, and exact match via RRF fusion."""

    def __init__(self, chroma_client: chromadb.ClientAPI,
                 collection_name: str = "dhi_workspace"):
        self.collection = chroma_client.get_or_create_collection(
            name=collection_name,
            embedding_function=embedding_functions.DefaultEmbeddingFunction(),
        )
        self.rrf_k = 60  # RRF constant (typical range: 20-100)

    def _bm25_score(self, query: str, document: str) -> float:
        """Simplified BM25 scoring for a single query-document pair.

        In production, this uses a proper BM25 implementation
        (e.g., rank_bm25 library) with term frequency (TF) and
        inverse document frequency (IDF) components.
        """
        query_terms = query.lower().split()
        doc_terms = document.lower().split()
        if not query_terms or not doc_terms:
            return 0.0

        # Average document length normalization
        avg_doc_len = 100.0  # Tuned to your codebase
        doc_len = len(doc_terms)
        k1 = 1.5
        b = 0.75

        score = 0.0
        for term in set(query_terms):
            tf = doc_terms.count(term)
            if tf == 0:
                continue
            # BM25 term frequency normalization
            numerator = tf * (k1 + 1)
            denominator = tf + k1 * (1 - b + b * doc_len / avg_doc_len)
            # IDF component (simplified — assumes term appears in 10% of docs)
            idf = math.log((self.collection.count() + 1) / (0.1 * self.collection.count() + 1))
            score += idf * numerator / denominator

        return score

    def search(self, query: str, n_results: int = 5,
               mode: str = "hybrid") -> List[Tuple[str, float, str]]:
        """Search across vector, BM25, and exact match modes.

        Args:
            query: The search query (e.g., a function name or description)
            n_results: Number of results to return
            mode: "hybrid" (default), "vector", or "bm25"

        Returns:
            List of (document_id, score, text) tuples, sorted by score descending
        """
        results = []

        if mode in ("hybrid", "vector"):
            # Dense vector search via Chroma
            vector_results = self.collection.query(
                query_texts=[query],
                n_results=n_results * 2,  # Fetch more for fusion
            )
            for i, (doc_id, distance) in enumerate(zip(
                vector_results["ids"][0],
                vector_results["distances"][0]
            )):
                # Convert distance to similarity score (1 - normalized distance)
                sim_score = 1.0 - (distance / 2.0)  # Cosine distance range: [0, 2]
                rrf_score = 1.0 / (self.rrf_k + i + 1)
                results.append((doc_id, rrf_score, sim_score))

        if mode in ("hybrid", "bm25"):
            # Sparse BM25 search
            all_docs = self.collection.get()
            bm25_scores = []
            for doc_id, doc_text in zip(all_docs["ids"], all_docs["documents"]):
                score = self._bm25_score(query, doc_text)
                if score > 0:
                    bm25_scores.append((doc_id, score))

            # Sort BM25 results by score descending
            bm25_scores.sort(key=lambda x: x[1], reverse=True)
            for rank, (doc_id, _) in enumerate(bm25_scores[:n_results * 2]):
                rrf_score = 1.0 / (self.rrf_k + rank + 1)
                # Find existing entry or add new one
                existing = next((r for r in results if r[0] == doc_id), None)
                if existing:
                    # Update RRF score (additive fusion)
                    idx = results.index(existing)
                    results[idx] = (doc_id, existing[1] + rrf_score, existing[2])
                else:
                    results.append((doc_id, rrf_score, 0.0))

        # Sort by combined RRF score descending
        results.sort(key=lambda x: x[1], reverse=True)
        return results[:n_results]

Advanced Pattern 2: Debounced Incremental Re-Indexing

Keeping the vector store in sync with active edits is a hard problem. Dhi’s approach uses debounced incremental re-indexing on save:

# server/rag/indexer.py
import asyncio
import time
from pathlib import Path
from typing import Set

class WorkspaceIndexer:
    """Manages workspace indexing with debounced incremental updates."""

    def __init__(self, chunker, store, debounce_ms: int = 2000):
        self.chunker = chunker
        self.store = store
        self.debounce_ms = debounce_ms
        self._pending_files: Set[str] = set()
        self._debounce_task: asyncio.Task | None = None
        self._lock = asyncio.Lock()

    async def mark_dirty(self, file_path: str) -> None:
        """Mark a file as needing re-indexing.

        Called by the file watcher on save events. Multiple saves
        within the debounce window are coalesced into a single
        re-index operation.
        """
        async with self._lock:
            self._pending_files.add(file_path)
            if self._debounce_task is not None:
                self._debounce_task.cancel()
            self._debounce_task = asyncio.create_task(
                self._flush_after_debounce()
            )

    async def _flush_after_debounce(self) -> None:
        """Wait for the debounce window, then re-index all pending files."""
        try:
            await asyncio.sleep(self.debounce_ms / 1000.0)
        except asyncio.CancelledError:
            return  # Another save came in, debounce resets

        async with self._lock:
            files_to_index = list(self._pending_files)
            self._pending_files.clear()

        for file_path in files_to_index:
            path = Path(file_path)
            if not path.exists():
                continue
            try:
                chunks = self.chunker.chunk_file(path)
                self.store.delete_document(file_path)
                self.store.add_chunks(chunks, source_file=file_path)
            except Exception as e:
                print(f"[Indexer] Failed to re-index {file_path}: {e}")

Production Considerations

Monitoring: Track three metrics to know if Dhi is working well. Completion latency (P50 should be under 2s on GPU, under 10s on CPU). Completion acceptance rate (users accept 30-50% of completions in practice — lower than 20% means the model is producing bad suggestions). Index coverage (percentage of workspace files that are indexed — below 80% means the RAG context is incomplete).

Error handling: The extension silently swallows errors by design — a failed completion should never crash VS Code or block typing. All errors are logged to the Dhi output channel. The server returns HTTP 422 for malformed requests, 500 for inference failures, and 503 when Ollama is unavailable. Monitor the server logs for patterns: repeated 503s indicate Ollama is overloaded or has run out of memory.

Rate limiting: Ollama processes one request at a time per model. If you have multiple developers sharing a server, implement a simple queue. The asyncio.Queue pattern with a timeout of 30 seconds per request prevents head-of-line blocking. Set OLLAMA_TIMEOUT=120 in the server env vars to allow for slow generations without premature timeouts.

The Results

Here is what you get when you replace a $20/month Copilot subscription with Dhi on a machine with a 6 GB GPU:

Metric Before (No AI) After (Dhi, starcoder2:3b, GPU) Improvement
Monthly cost $0 (no AI) $0 —
Setup time N/A 5 minutes —
Completion latency N/A < 1 second —
Cross-file context Manual search Automatic RAG retrieval —
Data privacy N/A Fully local —
Boilerplate typing speed ~30 lines/min ~45 lines/min +50%
Context switches to docs 5-10 per hour 1-3 per hour -60%

And on a CPU-only machine with the 1B model:

Metric Before (No AI) After (Dhi, starcoder2:1b, CPU) Improvement
Monthly cost $0 $0 —
Completion latency N/A 2-5 seconds —
Completion quality N/A 27% HumanEval —
Boilerplate typing speed ~30 lines/min ~38 lines/min +27%
Setup complexity N/A 5 minutes —

What this means for you: If you have a GPU, Dhi delivers sub-second completions that rival cloud services for boilerplate and common patterns, at zero cost. If you are on CPU, you get slower but functional completions that still save time on repetitive code. In both cases, your data never leaves your machine.

What to Watch Out For

  1. The first model download is 1.7 GB. The bootstrap script pulls starcoder2:3b on first run. On a 100 Mbps connection, this takes about 2 minutes. On a slower connection, it can take 10-15 minutes. The download only happens once — subsequent starts use the cached model.

  2. Docker Desktop needs at least 6 GB of RAM. If the Ollama container keeps crashing, open Docker Desktop settings and increase the memory allocation. The 3B model needs about 6 GB of RAM for the model weights plus overhead for the inference process.

  3. Completions on CPU are slow but usable. The 3B model on CPU takes 8-15 seconds. This feels sluggish. Switch to the 1B model for 2-5 second completions. The quality drop is noticeable for complex code, but for simple patterns it is fine.

  4. The RAG index is not automatically refreshed. If you add new files or significantly restructure your codebase, re-run Dhi: Index Workspace. The index is idempotent, so re-indexing is safe. A future update may add file watcher-based incremental indexing.

  5. The 422 error means a malformed request. If you see “422 Unprocessable Entity” in the Dhi output channel, the extension sent an incomplete request. Check that file_path, prefix, suffix, and language are all present. This usually happens with unsupported file types or very short files.

  6. Chroma errors on restart require a volume wipe. If Chroma throws errors after a Docker restart, run docker compose down -v && docker compose up -d. This deletes the persisted vector store and requires re-indexing. This is a known ChromaDB issue with volume persistence across container rebuilds.

Lesson 1: The 1B model is not a toy. It scores 27% on HumanEval, which sounds low, but for the code most developers write daily (conditionals, loops, method calls, string manipulation), it is surprisingly competent. Do not dismiss it because the benchmark number is low.

Lesson 2: RAG context is the difference between “plausible but wrong” and “actually useful.” A completion that references a function that does not exist is worse than no completion. Index your workspace before relying on Dhi for cross-file completions.

Lesson 3: Docker Desktop’s resource limits are the most common source of “it worked yesterday” failures. If Dhi stops working after a laptop restart, check Docker Desktop’s memory allocation first. The default 2 GB is insufficient for the 3B model.

Advice for Getting Started

Start with the default configuration (starcoder2:3b) on a machine with a GPU. If you do not have a GPU, switch to starcoder2:1b immediately — the 3B model on CPU is too slow for interactive use. Index your workspace before your first serious coding session. Use the Dhi output channel to verify that completions are flowing. Accept completions with Tab, dismiss with Escape. Do not expect cloud-level latency — expect local-level privacy and zero cost instead.

Next in the Open-Source AI Tools Mastery series: Skycode AI

NL

Written by Nivant Labs Team

Engineer at Nivant Labs

Share this post

XLinkedInHN