·15 min read

Tabby: A self-hosted AI coding assistant with no database, no cloud dependency, and no API keys

A self-hosted AI coding assistant with no database, no cloud, and no API keys — Tabby provides code completion for 15+ languages with a consumer-grade GPU.

#tabby#self-hosted#code-completion#gpu#privacy

The Problem

You are a team lead at a fintech company. Your developers want AI code completion. Your compliance officer says no code can leave the corporate network. GitHub Copilot sends every keystroke to Microsoft’s cloud. Cursor is a proprietary VS Code fork with its own backend. Even Continue.dev, while open-source, is a per-developer client — it does not give you a centralized server where you can enforce access policies, audit usage, or share a single GPU across the team.

The real problem is not just privacy. It is operational: how do you give 10, 50, or 200 developers AI-assisted coding without managing 200 separate Ollama instances, without 200 separate model downloads, without 200 separate config files, and without sending a single line of code to an external API?

Dimension GitHub Copilot Continue.dev + Ollama Tabby (self-hosted)
Architecture Cloud SaaS Per-device client Centralized server
Code leaves network Yes Configurable (local models = no) No
Team management Enterprise tier ($39/dev/mo) None Built-in admin dashboard
Usage analytics None None Per-user acceptance rates, API volume
SSO / LDAP Enterprise only None GitHub, GitLab, LDAP
GPU sharing N/A (cloud) Per-device GPU Single GPU serves entire team
Offline capable No Yes Yes
External database No No No (SQLite)
Setup time 2 minutes 15 minutes per developer 10 minutes once (Docker)
Cost for 10 devs/year $2,280-4,680 $0 + per-device GPU cost $0 + one shared GPU (~$500)

Why this matters: Tabby is the only self-hosted AI coding assistant that gives you a centralized server with team management, usage analytics, and repository-level RAG indexing — all running on a single consumer GPU, with no external database, no cloud dependency, and no API keys. It is the operational answer to the compliance question.

The Investigation

The root cause of the problem is architectural: every AI coding tool before Tabby was either a cloud service (Copilot, Cursor) or a per-device client (Continue.dev). Neither model works for teams that need centralized control, shared infrastructure, and air-gapped deployment.

Finding 1: Centralized server architecture is the only way to share a GPU across a team.

A single RTX 3090 running StarCoder2-3B can serve 2-5 developers concurrently with sub-200ms latency. If every developer runs their own Ollama instance, you need 5 GPUs to serve the same team. Tabby’s server model means one GPU, one model download, one config file, and every developer connects to it.

What this means: the GPU cost per developer drops from $500-800 (per-device) to $50-100 (shared) for a 10-person team. The break-even point is 3 developers.

Finding 2: Fill-in-the-Middle (FIM) completion requires a different model architecture than chat.

Code completion is not chat. It uses FIM: the model receives code before the cursor (prefix) and code after the cursor (suffix), and generates the code that belongs in the middle. This requires models trained with FIM sentinel tokens — special markers like <PRE>, <SUF>, <MID> for CodeLlama or <|fim_prefix|>, <|fim_suffix|>, <|fim_middle|> for Qwen2.5-Coder.

Tabby runs three independent model roles: completion (FIM), chat (instruction-tuned), and embedding (semantic retrieval). Each can be a different model on a different backend. This is not a nice-to-have — it is essential because a 3B FIM model that produces 80 tok/s for autocomplete is useless for architectural discussions, and a 7B chat model that produces 35 tok/s is too slow for inline completions.

What this means: you run a small, fast model for autocomplete (StarCoder2-3B at 80-120 tok/s) and a larger model for chat (Qwen2.5-Coder-7B at 35-55 tok/s). The embedding model runs only during indexing, not during inference.

Finding 3: Repository-level context is the difference between guessing and knowing.

Without RAG (Retrieval-Augmented Generation), a 3B model can only guess function signatures based on the open file. With RAG, it retrieves relevant snippets from across the entire repository — type definitions, function signatures, method declarations — and injects them as line comments into the prompt.

Tabby uses Tree-sitter to parse source code and extract symbol definitions. These are compiled into a BM25 token reverse index stored in Tantivy (a Rust search library). When a completion request arrives, the prefix and suffix are tokenized, a BM25 search runs against the index, and the top-ranked snippets are formatted as line comments and prepended to the prompt.

What this means: a 3B model with RAG context produces better completions than a 7B model without it. The index is rebuilt incrementally — only changed files are re-parsed.

Finding 4: The prompt template must match the model’s training tokens exactly.

The single most common cause of garbage completions in Tabby is a mismatched prompt_template. Each model family was trained with specific FIM sentinel tokens. Using CodeLlama’s <PRE> tokens on a Qwen2.5-Coder model produces nonsensical output because the model never saw those tokens during training.

Model Family FIM Template
CodeLlama <PRE>{prefix}<SUF>{suffix}<MID>
StarCoder2 <fim_prefix>{prefix}<fim_suffix>{suffix}<fim_middle>
Qwen2.5-Coder `<
DeepSeek-Coder {prefix}<｜fim▁hole｜> — the model was trained with 咒 as the FIM trigger token. Tabby’s local registry handles this automatically; you only set prompt_template manually when using remote HTTP backends.

The Solution

Tabby is a single Rust binary (built on the Axum web framework) that exposes REST API endpoints for code completion, chat, and repository indexing. It uses SQLite for configuration and user data — no PostgreSQL, no Redis, no external database. The entire server runs in one process.

┌─────────────────────────────────────────────────────────────┐
│                    TABBY SERVER (Rust + Axum)                 │
│                                                               │
│  ┌─────────────────────┐   ┌─────────────────────────────┐   │
│  │   HTTP Layer         │   │   Service Layer              │   │
│  │                      │   │                              │   │
│  │  POST /v1/completions│   │  CompletionService           │   │
│  │  POST /v1/chat/...   │──▶│    ├─ PromptBuilder          │   │
│  │  GET  /v1/health     │   │    ├─ CodeSearchService      │   │
│  │  GET  /v1beta/models │   │    └─ RAG pipeline           │   │
│  │  POST /v1/events     │   │                              │   │
│  └─────────────────────┘   │  ChatService                  │   │
│                            │    ├─ ChatState (per-session) │   │
│                            │    └─ SSE streaming            │   │
│                            │                              │   │
│                            │  IndexService                 │   │
│                            │    ├─ Scheduler (cron)         │   │
│                            │    └─ Repo sync + parse        │   │
│                            └─────────────────────────────┘   │
│                                                               │
│  ┌─────────────────────┐   ┌─────────────────────────────┐   │
│  │   Inference Layer    │   │   Data Layer                │   │
│  │   (tabby-inference)  │   │                              │   │
│  │                      │   │  SQLite (config, users,      │   │
│  │  llama.cpp bindings  │   │         sessions, events)    │   │
│  │    ├─ CUDA           │   │                              │   │
│  │    ├─ Metal          │   │  Tantivy (BM25 search index) │   │
│  │    ├─ ROCm           │   │                              │   │
│  │    └─ CPU fallback   │   │  Tree-sitter (code parsing)  │   │
│  │                      │   │                              │   │
│  │  HTTP API bindings   │   │  GGUF cache (model weights)  │   │
│  │    ├─ OpenAI-compat  │   └─────────────────────────────┘   │
│  │    ├─ Ollama         │                                      │
│  │    ├─ vLLM           │                                      │
│  │    └─ Mistral API    │                                      │
│  └─────────────────────┘                                      │
└─────────────────────────────────────────────────────────────┘

Here is what each piece does:

  • HTTP Layer (Axum): Exposes REST endpoints. The /v1/completions endpoint accepts prefix/suffix pairs and returns FIM completions. The /v1/chat/completions endpoint is OpenAI-compatible and streams responses via Server-Sent Events (SSE). The /v1/health endpoint is used by load balancers and orchestrators.

  • CompletionService: Receives the prefix and suffix from the IDE, calls CodeSearchService to retrieve relevant repository snippets, assembles the prompt using the model-specific FIM template, and runs a single inference call. The response is returned as ghost text that the IDE renders inline.

  • ChatService: Manages per-session conversation state. Supports @-mention syntax for referencing files, repositories, and documentation sources. Streams responses via SSE for real-time token delivery in the IDE chat panel.

  • IndexService: Runs a background scheduler that syncs configured Git repositories, parses each file with Tree-sitter, extracts symbol definitions, and builds a BM25 token reverse index in Tantivy. The index persists across server restarts and is updated incrementally.

  • Inference Layer (tabby-inference crate): A trait-based abstraction over model backends. The CodeGeneration trait handles FIM completion. The ChatCompletionStream trait handles streaming chat. The Embedding trait handles text embeddings for semantic search. Each trait can be backed by a local llama.cpp subprocess or a remote HTTP API.

  • Data Layer: SQLite stores configuration, user accounts, API tokens, and telemetry events. Tantivy stores the BM25 search index. Tree-sitter grammar files are bundled with the binary. GGUF model weights are cached in ~/.tabby/models/.

Production-Grade Setup

The recommended deployment is Docker with a single command:

# Pull the latest stable release
docker pull tabbyml/tabby:v0.32.0

# Run with NVIDIA GPU
docker run -it \
  --gpus all \
  -p 8080:8080 \
  -v $HOME/.tabby:/data \
  tabbyml/tabby:v0.32.0 \
  serve \
    --model StarCoder2-3B \
    --chat-model Qwen2.5-Coder-7B-Instruct \
    --device cuda

# Run with Apple Silicon (Metal)
docker run -it \
  -p 8080:8080 \
  -v $HOME/.tabby:/data \
  tabbyml/tabby:v0.32.0 \
  serve \
    --model StarCoder2-3B \
    --chat-model Qwen2.5-Coder-7B-Instruct \
    --device metal

The --model flag sets the completion model. The --chat-model flag sets a separate (larger) model for the chat panel and Answer Engine. If you omit --chat-model, Tabby runs in completion-only mode and uses less VRAM.

For production, use Docker Compose with resource reservations:

# docker-compose.yml
services:
  tabby:
    image: tabbyml/tabby:v0.32.0
    container_name: tabby
    ports:
      - "8080:8080"
    volumes:
      - tabby_data:/data
    command: >
      serve
      --model StarCoder2-3B
      --chat-model Qwen2.5-Coder-7B-Instruct
      --device cuda
      --parallelism 4
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/v1/health"]
      interval: 30s
      timeout: 10s
      retries: 3

volumes:
  tabby_data:

The --parallelism 4 flag limits concurrent completion requests. Each request consumes VRAM for the inference context. On an RTX 3090 (24 GB), parallelism of 4 with StarCoder2-3B uses approximately 8 GB of VRAM, leaving room for the chat model.

Configuration File

Tabby reads ~/.tabby/config.toml for advanced settings. The file is not created by default — you must create it manually:

# ~/.tabby/config.toml

# Model configuration: three independent roles
[model.completion.local]
model_id = "StarCoder2-3B"

[model.chat.local]
model_id = "Qwen2.5-Coder-7B-Instruct"

[model.embedding.local]
model_id = "Nomic-Embed-Text"

# Completion tuning
[completion]
max_input_length = 1536
max_decoding_tokens = 64

# Answer Engine system prompt
[answer]
system_prompt = """You are Tabby, an AI coding assistant specialized in helping developers understand and work with their codebase. You have access to the full repository index and can retrieve relevant code snippets. Answer questions concisely with code examples."""

# Additional language support (example: Swift)
[[additional_languages]]
languages = ["swift"]
exts = ["swift"]
line_comment = "//"
top_level_keywords = [
    "import", "let", "var", "func", "class", "struct",
    "enum", "protocol", "extension", "guard", "defer",
    "async", "await", "throws", "rethrows",
]

Remote Backend Configuration

If you want to offload chat to a cloud API while keeping completion local:

# Local completion (fast, private)
[model.completion.local]
model_id = "StarCoder2-3B"

# Remote chat (higher quality, uses API)
[model.chat.http]
kind = "openai/chat"
model_name = "gpt-4o-mini"
api_endpoint = "https://api.openai.com/v1"
api_key = "sk-..."  # Set via environment variable in production

# Remote embedding
[model.embedding.http]
kind = "openai/embedding"
model_name = "text-embedding-3-small"
api_endpoint = "https://api.openai.com/v1"
api_key = "sk-..."

Production pitfall: Never hardcode API keys in config.toml. Use environment variables or a secrets manager. Tabby reads TABBY_WEBSERVER_JWT_TOKEN_SECRET from the environment for JWT signing — follow the same pattern for API keys.

How to Use Effectively

Step 1: Install the IDE Extension

Tabby provides extensions for VS Code, JetBrains IDEs, Vim/Neovim, and Emacs. The VS Code extension is the most mature.

# VS Code: install from the marketplace
code --install-extension TabbyML.vscode-tabby

# Or search "Tabby" in the VS Code extensions panel

Configure the extension to point at your Tabby server:

// VS Code settings.json
{
  "tabby.apiEndpoint": "http://your-server:8080",
  "tabby.completionMode": "automatic",
  "tabby.inlineCompletion": true,
  "tabby.telemetry": false
}

For JetBrains, install from the marketplace (Settings > Plugins > Marketplace > search “Tabby”) and set the server URL in Tools > Tabby > Settings.

Step 2: Index Your Repository

Tabby needs to index your repository before it can provide context-aware completions. Run the scheduler to build the index:

# One-time index build
docker run -it \
  --entrypoint /opt/tabby/bin/tabby-cpu \
  -v $HOME/.tabby:/data \
  tabbyml/tabby:v0.32.0 \
  scheduler --now

# Or trigger from the admin dashboard at http://localhost:8080

The scheduler clones configured Git repositories, parses each file with Tree-sitter, extracts symbol definitions, and builds the BM25 index. For a 100,000-line monorepo, the initial index build takes 2-5 minutes on a modern CPU. Subsequent runs are incremental — only changed files are re-parsed.

Step 3: Configure Repository Context

In the admin dashboard, add your Git repositories:

  1. Navigate to http://localhost:8080 and log in as admin
  2. Go to the “Repositories” tab
  3. Add your repository URL (supports GitHub, GitLab, and self-hosted instances)
  4. For private repositories, provide a Personal Access Token (PAT) with read access
  5. Click “Sync Now” to trigger an immediate index build

Tabby supports multi-branch indexing as of v0.32.0. You can configure which branches to index in the repository settings.

Step 4: Test Completions

Open a file in your IDE and start typing. Tabby shows ghost text completions inline. Accept with Tab, reject with Esc.

# Type this in a Python file:
def calculate_risk_score(
    user_profile: dict,
    transaction_history: list,
    threshold: float = 0.8
) -> float:
    # Tabby suggests the body based on your repository's patterns
    total_score = 0.0
    for transaction in transaction_history:
        if transaction["amount"] > 10000:
            total_score += transaction["risk_weight"]
    return min(total_score / len(transaction_history), 1.0)

The completion uses RAG context from your indexed repository. If your codebase has similar functions in other files, Tabby retrieves those patterns and incorporates them into the prompt.

Step 5: Use the Chat Panel and Answer Engine

Open the Tabby chat panel in your IDE (Ctrl+Shift+P > “Tabby: Open Chat”). Use @-mention syntax to reference context:

  • @file src/main.rs — reference a specific file
  • @repo my-org/my-repo — reference an entire repository
  • @doc deployment-guide — reference ingested documentation

The Answer Engine (available in the web dashboard at http://localhost:8080/answers) provides a shareable Q&A interface over your indexed codebase. Team members can ask questions and get answers with citations to specific files and line numbers.

Use Cases

1. Air-Gapped Development Environment

When you would use this: Your organization operates in a classified or air-gapped network. No data can enter or leave the environment. Developers need AI assistance but cannot use any cloud service.

Why Tabby fits: Tabby runs entirely on your infrastructure. No external API calls. No telemetry leaving the network. The Docker image can be pre-loaded with model weights and deployed from an internal registry. The admin dashboard provides user management without external authentication dependencies.

2. Fintech Compliance Team

When you would use this: Your compliance team has mandated that no source code can be sent to third-party AI services. You have 15 developers who need AI code completion.

Why Tabby fits: Tabby’s centralized server gives you a single point of control. You configure the models, set access policies, and audit usage through the admin dashboard. The per-user acceptance rate analytics let you measure ROI without sending data to a third party. A single RTX 4070 Ti (16 GB VRAM) running StarCoder2-3B for completion and Qwen2.5-Coder-7B for chat serves the entire team.

3. Shared GPU Infrastructure for a Remote Team

When you would use this: Your team of 8 developers is distributed across time zones. Each developer has a laptop without a GPU. You want to provide AI code completion without requiring every developer to buy a GPU.

Why Tabby fits: Deploy Tabby on a single GPU server in your cloud VPC. Every developer connects to the same server. The GPU is shared — Tabby queues concurrent requests and batches inference where possible. The server cost is $50-100/month for a cloud GPU instance, versus $500-800 per developer for local GPUs.

4. Multi-Language Monorepo

When you would use this: Your codebase spans Python, Go, TypeScript, Rust, and Java. You need code completion that understands cross-language patterns and repository conventions.

Why Tabby fits: Tree-sitter supports 40+ languages out of the box. The BM25 index captures symbol definitions across all languages in the same repository. A Go function that calls a Python service via gRPC gets context from both the Go caller and the Python protobuf definitions. Tabby’s additional_languages config lets you add custom language support for proprietary DSLs.

5. On-Premises Enterprise with SSO Requirements

When you would use this: Your enterprise requires GitHub, GitLab, or LDAP-based single sign-on for all internal tools. You have 50+ developers who need AI code completion.

Why Tabby fits: Tabby supports GitHub SSO, GitLab SSO, and LDAP authentication. The admin dashboard provides role-based access control, API token management, and per-user usage analytics. The Answer Engine serves as an internal knowledge base for engineering teams. The entire deployment runs on your hardware with no external dependencies.

Cheat Sheet

Aspect Detail
Repository github.com/TabbyML/tabby
License Apache 2.0
Language Rust (92.9%), TypeScript (5.1%), Python (1.2%)
Latest release v0.32.0 (January 2026)
GitHub stars 33,500+
GPU requirement Minimum 4 GB VRAM (StarCoder2-3B), recommended 8 GB (7B models)
CPU mode Supported but not practical (8-15 tok/s vs 80-120 tok/s on GPU)
Setup time 10 minutes (Docker pull + run)
IDE support VS Code, JetBrains, Vim/Neovim, Emacs
Key features FIM completion, chat panel, Answer Engine, RAG indexing, team management, SSO, usage analytics
Model roles Completion (FIM), Chat (instruction-tuned), Embedding (semantic search)
Local backends llama.cpp (CUDA, Metal, ROCm, Vulkan, CPU)
Remote backends OpenAI, Ollama, vLLM, Mistral, Azure, any OpenAI-compatible API
Index engine Tantivy (BM25 token reverse index)
Code parser Tree-sitter (40+ languages)
Database SQLite (no external database required)
Default ports 8080 (HTTP)
Data directory ~/.tabby/ (config, models, indexes, SQLite DB)
Common gotchas Wrong FIM prompt template = garbage output; embedding model change requires re-index; CPU mode is too slow for interactive use; 3 models loaded simultaneously consume more VRAM than the completion model alone

Vibe Coding Projects

Project 1: Self-Hosted Code Completion for a Personal Blog Engine

What it does: Deploy Tabby on a $35/month cloud GPU instance (Lambda Labs or Vast.ai) and configure it to provide code completion for a personal blog engine written in Python and JavaScript. Index the repository, test completions across both languages, and measure acceptance rates.

What you will learn: Docker deployment on cloud GPU infrastructure, Tabby configuration for multi-language projects, RAG indexing workflow, and acceptance rate analytics.

Effort: 2-3 hours. One evening to deploy and configure, one evening to test and tune.

Project 2: Team Tabby Server with SSO and Usage Analytics

What it does: Deploy Tabby on a team GPU server with GitHub SSO authentication. Configure separate models for completion (StarCoder2-3B) and chat (Qwen2.5-Coder-7B). Set up repository indexing for a monorepo with 3 languages. Invite 5 team members and measure per-user acceptance rates over one week.

What you will learn: SSO configuration, multi-model deployment, team management, usage analytics interpretation, and production monitoring.

Effort: 4-6 hours. One session for deployment and SSO setup, one session for team onboarding, and periodic check-ins to review analytics.

Project 3: Air-Gapped Tabby with Custom Model

What it does: Deploy Tabby in a fully air-gapped environment (no internet access). Pre-download model weights (StarCoder2-3B GGUF), load them onto the server via USB drive, and configure Tabby to use them without any external API calls. Index a private codebase and verify that no network requests are made during operation.

What you will learn: Air-gapped deployment patterns, offline model distribution, network egress verification, and compliance documentation.

Effort: 3-4 hours. One session for model download and transfer, one session for deployment and verification.

Problems Solved Efficiently

Problem Type Why Tabby Fits When to Look Elsewhere
Team needs AI code completion but cannot use cloud services Centralized server, no external API calls, full data residency Solo developer with a GPU laptop (Continue.dev + Ollama has lower overhead)
Compliance requires audit trail of AI usage Built-in per-user analytics, event logging, admin dashboard Need agentic multi-file refactoring (use Aider or Cline)
Team of 5-20 developers sharing one GPU Single server serves all developers, queues concurrent requests Need maximum raw completion quality (GitHub Copilot with GPT-4o still wins on complex reasoning)
Multi-language monorepo needs cross-file context Tree-sitter parsing + BM25 indexing across all languages Need deep GitHub integration (PR reviews, issue comments)
Enterprise needs SSO and role-based access GitHub, GitLab, LDAP SSO; admin dashboard with API tokens Need a VS Code fork with built-in AI (use Cursor)
Offline/air-gapped development environment Fully self-contained, no network dependencies, Docker image pre-loadable Need chat with web-browsing or real-time data access
Organization wants to evaluate AI coding ROI Per-user acceptance rate analytics, API call volume metrics Need a quick personal trial without infrastructure (use Copilot free tier)

Architectural Tradeoffs

What we gained

  • Zero external dependencies. No database server, no message queue, no Redis cache, no cloud API keys. The entire system is one Rust binary and one SQLite file. This makes deployment, backup, and disaster recovery trivial.

  • Centralized GPU sharing. A single RTX 3090 serves 2-5 developers concurrently. The GPU cost per developer drops from $500-800 to $50-100. The break-even point is 3 developers.

  • Full data residency. Code never leaves your network. The model weights, the index, the configuration, and the telemetry all live in ~/.tabby/. This is the only way to satisfy SOC 2, HIPAA, ITAR, and air-gap requirements.

  • Three independent model roles. Completion, chat, and embedding are separate model slots. You can run a 3B model for fast autocomplete, a 7B model for chat quality, and a lightweight embedding model for indexing. Each can be local or remote independently.

  • Repository-level RAG context. Tree-sitter parsing + BM25 indexing means even a 3B model produces contextually aware completions. The index is incremental and persists across restarts.

What we sacrificed

  • No agentic capabilities. Tabby will not write multi-file features, spawn shell commands, or plan refactors autonomously. It is a completion and chat server, not an AI coding agent. If you need agentic coding, pair Tabby with Aider or Cline.

  • Smaller models lag on complex reasoning. A 3B FIM model produces good single-line and single-function completions, but it cannot reason about architectural decisions or suggest multi-file refactors. The 7B chat model helps, but it still lags behind GPT-4o on complex tasks.

  • Infrastructure burden. You own GPU maintenance, driver updates, model version management, and server uptime. A cloud GPU instance costs $50-100/month. A local GPU workstation costs $500-2,000 upfront. This is not zero-cost — it is a different cost center.

  • CPU mode is not practical. Running on CPU produces 8-15 tok/s, which is too slow for interactive code completion. The first-token latency exceeds 2 seconds, making completions feel sluggish. Tabby requires a GPU for acceptable performance.

  • Embedding model changes invalidate the index. If you switch from Nomic-Embed-Text to a different embedding model, the entire BM25 index must be rebuilt. This takes 2-5 minutes for a 100,000-line repository.

  • No chat in base install. The base Docker image runs completion only. Chat and the Answer Engine require a separate chat model, which consumes additional VRAM. On a 4 GB GPU, you can only run the completion model.

The real lesson: Tabby optimizes for operational simplicity and data privacy at the cost of model quality and agentic capability. It is the right tool when your primary constraint is “code cannot leave this network” and your secondary constraint is “we want to share one GPU across the team.” It is the wrong tool when your primary constraint is “I want the smartest possible AI coding assistant” — in that case, use Copilot or Continue.dev with a cloud model.

Course-Style Deep Dive

How Tabby Works Under the Hood

When you type in your IDE, the following sequence executes in under 200 milliseconds:

  1. IDE captures context. The VS Code extension reads the text before the cursor (prefix) and after the cursor (suffix), along with the file path, language, and recently edited files. This is sent as a POST request to /v1/completions on the Tabby server.

  2. Server retrieves repository context. The CompletionService calls CodeSearchService, which tokenizes the prefix and suffix and runs a BM25 search against the Tantivy index. The top 5-10 matching snippets are retrieved. Each snippet is a symbol definition extracted by Tree-sitter — function signatures, type definitions, method declarations.

  3. Snippets are formatted as line comments. Retrieved snippets are prepended to the prompt as line comments so they do not break the semantics of the existing code:

// Path: src/services/risk_calculator.rs
// fn calculate_risk_score(user: &User, history: &[Transaction]) -> f64
// Path: src/models/user.rs
// struct User { id: String, name: String, risk_score: f64 }
  1. Prompt is assembled with FIM sentinel tokens. The PromptBuilder wraps the prefix, suffix, and retrieved snippets in the model-specific FIM template. For StarCoder2:
<fim_prefix>// Path: src/services/risk_calculator.rs
// fn calculate_risk_score(user: &User, history: &[Transaction]) -> f64

fn calculate_risk_score(
    user: &User,
    history: &[Transaction],
) -> f64 {
    let total: f64 = history.iter()
        .map(|t| t.amount * t.risk_weight)
        .sum()
    total / history.len() as f64
}
<fim_suffix>

fn main() {
    let score = calculate_risk_score(
        <fim_middle>

  1. Inference runs on the model. The assembled prompt is sent to the inference backend (llama.cpp subprocess or HTTP API). The model generates the middle section — the code that belongs at the cursor position. For a 3B model on an RTX 3080, this takes 8-12 milliseconds.

  2. Response is returned to the IDE. The generated text is streamed back as ghost text. The IDE renders it inline. The developer accepts with Tab or rejects with Esc.

Advanced Pattern 1: Hybrid Local/Remote Deployment

Run completion locally for speed and privacy, but route chat through a cloud API for higher quality:

# ~/.tabby/config.toml

# Fast local completion
[model.completion.local]
model_id = "StarCoder2-3B"

# High-quality remote chat
[model.chat.http]
kind = "openai/chat"
model_name = "gpt-4o-mini"
api_endpoint = "https://api.openai.com/v1"
# API key from environment variable
api_key = "${OPENAI_API_KEY}"

# Local embedding for privacy
[model.embedding.local]
model_id = "Nomic-Embed-Text"

This pattern gives you sub-200ms completions (local) and GPT-4o-quality chat (remote), while keeping your codebase index entirely local. The embedding model never sends code to an external API.

Advanced Pattern 2: Custom FIM Template for a Non-Standard Model

If you are using a model from Hugging Face that is not in Tabby’s registry, you must provide the correct FIM template:

[model.completion.http]
kind = "vllm/completion"
model_name = "my-org/custom-coder-7b"
api_endpoint = "http://vllm-server:8000/v1"
api_key = "${VLLM_API_KEY}"
prompt_template = "<|fim_prefix|>{prefix}<|fim_suffix|>{suffix}<|fim_middle|>"

To find the correct template, inspect the model’s tokenizer config on Hugging Face:

# Run this locally to discover FIM tokens
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("my-org/custom-coder-7b")
print("FIM prefix:", tokenizer.fim_prefix)  # e.g., "<|fim_prefix|>"
print("FIM suffix:", tokenizer.fim_suffix)  # e.g., "<|fim_suffix|>"
print("FIM middle:", tokenizer.fim_middle)  # e.g., "<|fim_middle|>"

Advanced Pattern 3: Multi-Instance Load Balancing

For teams larger than 5 concurrent developers, run multiple Tabby instances behind a reverse proxy:

# docker-compose.yml with 2 Tabby instances
services:
  tabby-1:
    image: tabbyml/tabby:v0.32.0
    volumes:
      - tabby_data_1:/data
    command: serve --model StarCoder2-3B --device cuda --parallelism 2
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    restart: unless-stopped

  tabby-2:
    image: tabbyml/tabby:v0.32.0
    volumes:
      - tabby_data_2:/data
    command: serve --model StarCoder2-3B --device cuda --parallelism 2
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    restart: unless-stopped

  nginx:
    image: nginx:alpine
    ports:
      - "8080:80"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf
    depends_on:
      - tabby-1
      - tabby-2
# nginx.conf — round-robin load balancing
upstream tabby_backend {
    server tabby-1:8080;
    server tabby-2:8080;
}

server {
    listen 80;
    location / {
        proxy_pass http://tabby_backend;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}

Production Considerations

Monitoring: Tabby exposes a /v1/health endpoint that returns HTTP 200 when the server is ready. Use this for container orchestration health checks. Monitor GPU utilization, VRAM usage, and request latency with nvidia-smi and Prometheus.

# Monitor GPU metrics
watch -n 1 nvidia-smi --query-gpu=utilization.gpu,memory.used,temperature.gpu --format=csv

# Check Tabby health
curl -s http://localhost:8080/v1/health | jq .

Error handling: Tabby returns standard HTTP error codes. A 503 indicates the model is still loading. A 429 indicates rate limiting (configurable via --parallelism). A 400 indicates a malformed request — usually a missing or incorrect FIM template.

Rate limiting: The --parallelism flag controls the maximum number of concurrent completion requests. On an RTX 3090 with StarCoder2-3B, set parallelism to 4-6. Each concurrent request consumes approximately 2 GB of VRAM for the inference context. Monitor VRAM usage and adjust accordingly.

Backup strategy: Back up the ~/.tabby/ directory regularly. It contains:

  • config.toml — server configuration
  • data/ — SQLite database (users, sessions, events)
  • index/ — BM25 search index (can be rebuilt, but takes time)
  • models/ — cached GGUF model weights (can be re-downloaded)
# Backup script
#!/bin/bash
BACKUP_DIR="/backups/tabby/$(date +%Y%m%d)"
mkdir -p "$BACKUP_DIR"
tar czf "$BACKUP_DIR/tabby-backup.tar.gz" -C "$HOME" .tabby
echo "Backup saved to $BACKUP_DIR/tabby-backup.tar.gz"

The Results

Metric Before Tabby After Tabby Improvement
Code completion latency N/A (no AI) 80-120 tok/s (StarCoder2-3B) Real-time ghost text
First-token latency N/A <200ms (GPU) Imperceptible delay
Repository context Manual lookup Automatic RAG retrieval 10x faster context access
GPU cost per developer $500-800 (per-device) $50-100 (shared) 5-10x cost reduction
Setup time per developer 15 minutes (per-device) 10 minutes (one-time) 15x faster team onboarding
Code privacy Sent to cloud Zero egress Full compliance
Team management None Admin dashboard + SSO Centralized control
Usage analytics None Per-user acceptance rates Data-driven decisions

What this means for you: Tabby is the most operationally complete self-hosted code completion server available in 2026. It solves the team-scale AI coding problem that no other tool addresses: how to give every developer AI assistance without sending code to a third party, without managing per-device installations, and without buying a GPU for every developer.

The tradeoff is real: smaller models produce lower-quality completions than GPT-4o. But for the vast majority of coding tasks — writing boilerplate, implementing known patterns, filling in function bodies — a 3B model with RAG context is good enough. And “good enough” with full privacy and zero per-seat cost beats “excellent” with a compliance violation and a $39/dev/month bill.

What to Watch Out For

  1. Do not run Tabby on CPU for interactive use. CPU inference produces 8-15 tok/s with 2+ second first-token latency. Completions arrive after you have already typed the next 5 characters. The autocomplete experience is broken. Tabby requires a GPU — a used RTX 3060 (12 GB VRAM) for $200 is the minimum viable investment.

  2. Match the FIM template to your model exactly. The most common cause of garbage completions is a mismatched prompt_template. If you use a model from Tabby’s registry, the template is handled automatically. If you use a custom model or a remote backend, verify the template against the model’s tokenizer config. One wrong sentinel token and every completion is nonsense.

  3. Account for all three models when sizing VRAM. Tabby loads three models: completion, chat, and embedding. A 7B completion model might fit in 8 GB VRAM, but adding a 7B chat model and an embedding model pushes total VRAM usage to 16+ GB. On a 12 GB GPU, you can run StarCoder2-3B for completion and skip the chat model.

  4. Rebuild the index after changing the embedding model. The BM25 index is tied to the embedding model. Switching from Nomic-Embed-Text to a different model invalidates the existing index. Run scheduler --now after any embedding model change.

  5. Set TABBY_WEBSERVER_JWT_TOKEN_SECRET in production. Without this environment variable, Tabby generates a random secret on startup. If the container restarts, all existing JWT tokens are invalidated and every user must re-authenticate. Set it to a stable, secret value.

  6. Do not expose Tabby directly to the internet. The admin dashboard has no built-in rate limiting or DDoS protection. Deploy behind a reverse proxy (nginx, Caddy) with HTTPS termination and IP allowlisting.

  7. Monitor VRAM usage with nvidia-smi. Tabby does not automatically adjust parallelism based on available VRAM. If you set --parallelism 8 on a 12 GB GPU, you will get out-of-memory errors. Start with --parallelism 2 and increase while monitoring VRAM.

Lesson 1: A self-hosted AI coding assistant is not free — it shifts cost from per-seat SaaS fees to GPU hardware and maintenance. The total cost of ownership for a 10-person team over 2 years is approximately $1,000 (GPU) + $1,200 (electricity/hosting) = $2,200, versus $4,560-9,360 for Copilot. The GPU investment breaks even in 4-6 months.

Lesson 2: Model quality matters less than context quality. A 3B model with RAG context from your indexed repository produces better completions than a 7B model without context. Invest in indexing your codebase properly before upgrading to a larger model.

Lesson 3: Tabby is not a Copilot replacement for every scenario. It is a Copilot replacement for teams that prioritize privacy, control, and cost over maximum model quality. If your team needs the absolute best completions and has no compliance constraints, Copilot is still the better choice. If your team needs AI assistance that stays on your network and costs nothing per developer, Tabby is the only option.

Advice for Getting Started

Start with the smallest viable setup: a single Docker container with StarCoder2-3B on a GPU with at least 4 GB VRAM. Do not add the chat model initially — completion-only mode uses less VRAM and is simpler to debug. Once completions are working, add the chat model and test the Answer Engine. Once the team is comfortable, add SSO and repository indexing.

The most common mistake is trying to run a 7B model on a 4 GB GPU. StarCoder2-3B is the sweet spot for consumer GPUs. It produces 80-120 tok/s on an RTX 3080, fits in 4 GB VRAM, and with RAG context, it handles most completion tasks well. Upgrade to a 7B model only when you have 8+ GB of VRAM available.

Next in the Open-Source AI Tools Mastery series: Cody

NL

Written by Nivant Labs Team

Engineer at Nivant Labs

Share this post

XLinkedInHN