·15 min read

Continue.dev: The leading open-source AI code assistant

A VS Code and JetBrains extension providing code completion, chat, and agent capabilities with support for any LLM backend.

#continue-dev#vscode#jetbrains#code-completion#ai-assistant

The Problem

You are a developer who wants AI-assisted coding but cannot use GitHub Copilot or Cursor. Maybe your employer bans sending code to third-party APIs. Maybe you work in defense, healthcare, or fintech where data residency requirements mean no cloud service can touch your source files. Maybe you just want to choose your own model — running a local Qwen2.5-Coder on Ollama for autocomplete and routing chat through Claude Sonnet 4.6 via Anthropic’s API.

The closed-source tools give you none of these options. Copilot locks you into Microsoft’s model pipeline. Cursor is a VS Code fork with its own proprietary backend. Both send your code to their cloud. Both cost $10-20 per user per month. Both give you zero control over which model runs which task.

Dimension GitHub Copilot Cursor Continue.dev
License Proprietary Proprietary Apache 2.0
IDE support VS Code, JetBrains, Neovim VS Code fork only VS Code, JetBrains, CLI
Model choice GPT-4o, Claude 3.5 Sonnet (opt-in) Claude, GPT-4o, Gemini Any of 40+ providers
Local models No No Yes (Ollama, LM Studio, vLLM)
Air-gapped possible No No Yes
Cost per user/mo $10-39 $20-40 Free (BYO API keys)
Data leaves your machine Yes Yes Configurable (zero with local models)
Cross-IDE config No N/A (single IDE) Yes (single config file)
Custom assistants No Limited Yes (rules + MCP servers)
MCP support No Yes Yes

Why this matters: If you cannot or will not send your code to a third-party cloud, your only option for AI-assisted coding is an open-source tool that runs locally. Continue.dev is that tool — and it is the most mature, most configurable, and most widely adopted open-source AI code assistant in existence, with 34,000+ GitHub stars and 400+ contributors.

The Investigation

The core problem is architectural: every closed-source AI coding tool is a vertically integrated stack. The IDE, the model, the context pipeline, and the telemetry are all owned by one vendor. You cannot swap the model. You cannot run it locally. You cannot audit what data leaves your machine.

Continue.dev solves this by inverting the architecture. Instead of a vendor-owned stack, it provides a pluggable LLM abstraction layer that decouples the IDE from the model provider. The same extension works with any backend that implements the ILLM interface — 40+ providers at launch, from Anthropic to Ollama to a custom OpenAI-compatible endpoint running on your laptop.

The root cause of the closed-source lock-in is the lack of a standardized, open protocol between IDEs and LLM backends. Every vendor builds their own. Continue.dev’s answer is a typed message-passing protocol (ToCoreProtocol, FromCoreProtocol, ToWebviewProtocol) that separates the IDE layer from the core orchestrator from the LLM provider. This is the same pattern that made HTTP successful: a standard interface enables interchangeable backends.

Finding 1: The LLM abstraction layer is the critical enabler.

The ILLM interface defines six methods that every provider must implement:

  • streamChat(messages, signal, options) — streaming chat for conversational interactions
  • streamComplete(prompt, signal, options) — streaming text completion
  • streamFim(prefix, suffix, signal, options) — fill-in-the-middle for tab autocomplete
  • complete(prompt, signal, options) — synchronous completion
  • chat(messages, signal, options) — non-streaming chat
  • embed(chunks) — generate vector embeddings for codebase indexing

What this means: Any model that implements these six methods can serve as a drop-in replacement. You can use GPT-4o for chat, Qwen2.5-Coder for autocomplete, and a local embedding model for codebase search — all from the same extension, configured in a single YAML file.

Finding 2: The message-passing architecture enables cross-IDE support.

Continue.dev supports VS Code, JetBrains, and a standalone CLI from a single codebase. The key is that the IDE layer is thin — it handles platform-specific concerns (editor integration, keybindings, UI chrome) but delegates all business logic to the Core orchestrator via typed messages.

Webview (React GUI) <--> IDE Extension <--> Core Orchestrator
       ^                                    ^
       |          (pass-through)            |
       +------------------------------------+

What this means: The same ~/.continue/config.yaml works in VS Code and JetBrains. A team with Java backend developers in IntelliJ and React frontend developers in VS Code can share one configuration. This is unique — no other AI coding tool offers cross-IDE config portability.

Finding 3: The FIM-based autocomplete pipeline is model-agnostic but template-dependent.

Tab autocomplete uses Fill-in-the-Middle (FIM), where the model receives code before the cursor (prefix) and code after the cursor (suffix) and generates the middle. Different models use different FIM token templates:

Model FIM Template
Qwen2.5-Coder `<
StarCoder2 <fim_prefix>{{ prefix }}<fim_suffix>{{ suffix }}<fim_middle>
Codestral (Mistral) [SUFFIX]{{ suffix }}[PREFIX]{{ prefix }}[MIDDLE]
DeepSeek Coder <｜fim▁begin｜># <｜fim▁hole｜>e CompletionProvider` gathers prefix, suffix, and LSP context
  1. The FIM template is applied with the configured model
  2. The model streams tokens back via streamFim()
  3. Ghost text is rendered inline in the editor

What this means: Autocomplete quality depends entirely on the FIM template matching the model’s training format. A mismatch produces garbage. The tabAutocompleteOptions.template field is the most critical — and most commonly misconfigured — setting in the entire system.

The Solution

Continue.dev’s architecture is a layered system with strict separation of concerns. Here is the high-level structure:

+----------------------------------------------------------+
|                    IDE Layer                              |
|  VS Code Extension  |  JetBrains Plugin  |  CLI (TUI)   |
+----------------------------------------------------------+
|                   Core Orchestrator                       |
|  ConfigHandler  |  CompletionProvider  |  DocsService    |
|  CodebaseIndexer |  NextEditProvider   |  MCPManager     |
+----------------------------------------------------------+
|                LLM Abstraction Layer                      |
|  ILLM Interface  ->  BaseLLM  ->  Provider Implementations|
|  (streamChat, streamFim, embed, countTokens, ...)         |
+----------------------------------------------------------+
|              External LLM Providers                        |
|  OpenAI  |  Anthropic  |  Ollama  |  Gemini  |  ...       |
+----------------------------------------------------------+

Here’s what each piece does:

  • IDE Layer: Thin platform-specific adapters that translate editor events (keystrokes, file opens, cursor movements) into typed messages. The VS Code extension and JetBrains plugin share ~90% of their code through the core library. The CLI provides a terminal-native TUI for headless environments.

  • Core Orchestrator: The Core class in core/core.ts initializes all services and registers ~80 message handlers. It manages abort controllers for request cancellation, coordinates service lifecycle, and handles configuration hot-reloads. Every user action — a chat message, a tab completion request, a codebase search — flows through this layer.

  • ConfigHandler: Loads and merges configuration from multiple sources: local ~/.continue/config.yaml, workspace .continue/config.yaml, remote hub profiles, and organization-scoped settings. The merge order is: local defaults < workspace overrides < org policies < user session selections.

  • CompletionProvider: Manages the tab autocomplete pipeline. It debounces keystrokes (default 350ms), assembles the FIM prompt from prefix/suffix/LSP context, sends the request to the configured autocomplete model, and renders the ghost text. It also manages a completion cache to avoid redundant requests.

  • CodebaseIndexer: Maintains vector embeddings for semantic code search. When you type @codebase in chat, this service retrieves relevant files using cosine similarity against the indexed embeddings. The index is built incrementally — only changed files are re-embedded on save.

  • MCPManagerSingleton: Manages connections to Model Context Protocol servers. Each MCP server exposes tools (e.g., GitHub API, database queries, browser automation) that the agent can invoke. Supports stdio, SSE, and Streamable HTTP transports.

  • LLM Abstraction Layer: The ILLM interface defines the contract. BaseLLM provides shared functionality: token counting via Tiktoken or LlamaEncoding, message compilation with context pruning, streaming via AsyncGenerator, exponential backoff retry logic, and prompt template rendering using Handlebars. Each provider (OpenAI, Anthropic, Ollama, etc.) extends BaseLLM and implements the provider-specific API calls.

Production-Grade Configuration

Here is a complete ~/.continue/config.yaml that configures three models for different roles, custom rules, and MCP servers:

name: Production Config
version: 1.0.0
schema: v1

models:
  # Chat model: Claude Sonnet 4.6 via Anthropic API
  - name: Claude Sonnet
    provider: anthropic
    model: claude-sonnet-4-20260514
    roles:
      - chat
      - edit
    defaultCompletionOptions:
      temperature: 0.7
      maxTokens: 4096

  # Autocomplete model: local Qwen2.5-Coder via Ollama
  - name: Qwen Coder
    provider: ollama
    model: qwen2.5-coder:7b
    roles:
      - autocomplete
    autocompleteOptions:
      debounceDelay: 250
      maxPromptTokens: 2048
      onlyMyCode: true
      multilineCompletions: auto
      template: "<|fim_prefix|>{{{ prefix }}}<|fim_suffix|>{{{ suffix }}}<|fim_middle|>"

  # Embedding model for codebase search
  - name: Nomic Embed
    provider: ollama
    model: nomic-embed-text:v1.5
    roles:
      - embed

rules:
  - Always use TypeScript strict mode
  - Prefer functional components over class components in React
  - All database queries must use parameterized statements
  - Every public function must have a JSDoc comment
  - Never use `any` — use `unknown` and narrow with type guards

context:
  - provider: codebase
    params:
      maxResults: 10
  - provider: diff
  - provider: file
  - provider: problems

mcpServers:
  - name: GitHub
    command: npx
    args:
      - "-y"
      - "@modelcontextprotocol/server-github"
    env:
      GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

  - name: Browser search
    command: npx
    args:
      - "@playwright/mcp@latest"

  - name: Sentry
    command: npx
    args:
      - "-y"
      - "@sentry/mcp-server@latest"
    env:
      SENTRY_AUTH_TOKEN: ${{ secrets.SENTRY_AUTH_TOKEN }}

Setup Instructions

Install the extension and configure your models:

# Install the VS Code extension from the marketplace
code --install-extension continue.continue

# Or install via OpenVSX (for VSCodium or air-gapped environments)
# Download the .vsix from https://github.com/continuedev/continue/releases
code --install-extension continue-2.1.0.vsix

# Install Ollama for local models
curl -fsSL https://ollama.com/install.sh | sh

# Pull your autocomplete model
ollama pull qwen2.5-coder:7b

# Pull your embedding model
ollama pull nomic-embed-text:v1.5

# Create the config directory
mkdir -p ~/.continue

# Create the rules directory
mkdir -p ~/.continue/rules

# Create the MCP servers directory
mkdir -p ~/.continue/mcpServers

# For the CLI (optional)
npm install -g @continuedev/cli

How to Use Effectively

Step 1: Configure Model Roles Correctly

The most common mistake is using the same model for chat and autocomplete. Chat models (Claude, GPT-4o) are trained for conversational interaction. Autocomplete models (Qwen2.5-Coder, StarCoder2, Codestral) are trained for FIM. Using a chat model for autocomplete produces slow, irrelevant suggestions.

# Correct: separate models for separate roles
models:
  - name: Claude Sonnet
    provider: anthropic
    model: claude-sonnet-4-20260514
    roles:
      - chat
      - edit

  - name: Qwen Coder
    provider: ollama
    model: qwen2.5-coder:7b
    roles:
      - autocomplete
    autocompleteOptions:
      template: "<|fim_prefix|>{{{ prefix }}}<|fim_suffix|>{{{ suffix }}}<|fim_middle|>"

The roles field accepts chat, edit, autocomplete, and embed. A model can serve multiple roles, but autocomplete should always use a FIM-trained model.

Step 2: Write Effective Rules

Rules are injected into the system prompt for every chat, edit, and agent interaction. They shape the model’s behavior without requiring you to repeat instructions.

Place project-specific rules in .continue/rules/ (version-controlled with your codebase):

# .continue/rules/typescript-rules.md

- Use `interface` over `type` for object shapes
- Use `const` assertions for literal types: `as const`
- Prefer `zod` for runtime validation over class-validator
- All API routes must have input validation using zod schemas
- Error boundaries must be at every route segment

Place global rules in ~/.continue/rules/ (applied to every project):

# ~/.continue/rules/global-rules.md

- Write concise responses. Prefer code examples over explanations.
- Assume TypeScript unless otherwise specified.
- Never use `any`. Use `unknown` and narrow with type guards.
- All functions must have explicit return types.

Rules are concatenated in order: global rules first, then workspace rules, then inline rules from config.yaml.

Step 3: Use @ Mentions for Context Injection

The @ mention system lets you inject specific context into chat. This is how you tell the model what to look at without copying and pasting:

  • @file src/routes/users.ts — Include a specific file’s contents
  • @codebase — Semantic search across the entire codebase
  • @docs react — Include React documentation (pre-indexed)
  • @terminal — Include the last terminal output
  • @problems — Include current IDE diagnostics
  • @git — Include git diff for the current branch
// Example: Ask a question with context
// In chat, type:
// @file src/services/auth.ts @codebase "Find all places where this auth
// service is called and refactor them to use the new v2 API"

The @codebase provider uses vector embeddings from your configured embedding model. It retrieves the top maxResults files (default 10) by cosine similarity. The index is built incrementally — only changed files are re-embedded on save, so the first query after a large checkout may be slow.

Step 4: Configure MCP Servers for Agent Mode

Agent mode can read/write files, run terminal commands, and invoke MCP server tools. MCP servers extend the agent’s capabilities to external systems:

mcpServers:
  # Local stdio transport: runs a subprocess
  - name: SQLite
    type: stdio
    command: npx
    args:
      - "@modelcontextprotocol/server-sqlite"
      - "/path/to/database.db"

  # Remote SSE transport: connects to a server-sent events endpoint
  - name: Sentry
    type: sse
    url: https://mcp.sentry.io

  # Streamable HTTP transport: bidirectional streaming
  - name: Supabase
    type: streamable-http
    url: https://mcp.supabase.com/mcp
    apiKey: ${{ secrets.SUPABASE_ACCESS_TOKEN }}

Secrets are resolved from .env files. Place workspace secrets in .continue/.env and global secrets in ~/.continue/.env:

# ~/.continue/.env
GITHUB_TOKEN=ghp_your_token_here
SENTRY_AUTH_TOKEN=sntrys_your_token_here
SUPABASE_ACCESS_TOKEN=sb_your_token_here

Step 5: Tune Autocomplete for Your Workflow

The tabAutocompleteOptions block controls every aspect of the autocomplete pipeline:

autocompleteOptions:
  disable: false                    # Set true to disable autocomplete entirely
  debounceDelay: 250               # Milliseconds to wait after last keystroke
  maxPromptTokens: 2048             # Max tokens for the FIM prompt
  maxSuffixPercentage: 0.2          # Max % of prompt allocated to suffix
  prefixPercentage: 0.3            # % of prompt allocated to prefix
  onlyMyCode: true                  # Only include code within the repo
  useCache: true                    # Cache completions for identical contexts
  multilineCompletions: auto        # "always", "never", or "auto"
  disableInFiles:                   # Glob patterns to exclude
    - "*.md"
    - "*.json"
    - "*.yaml"
  template: "<|fim_prefix|>{{{ prefix }}}<|fim_suffix|>{{{ suffix }}}<|fim_middle|>"

Key tuning parameters:

  • debounceDelay: Lower values (100-150ms) feel more responsive but increase API calls. Higher values (350-500ms) batch more context but feel laggy. Start at 250ms and adjust based on your model’s latency.

  • maxPromptTokens: Larger values give the model more context but increase latency. For local models with 7B parameters, 1024-2048 is sufficient. For larger models (32B+), you can go up to 8192.

  • multilineCompletions: Set to "always" if you want multi-line suggestions (e.g., function bodies). Set to "never" if you only want single-line completions. "auto" lets the model decide based on context.

  • template: This is the FIM template. It must match the model’s training format exactly. Using the wrong template produces garbage completions. Check the model’s documentation for the correct FIM tokens.

Use Cases

1. Air-Gapped Development

When you’d use this: You work in defense, healthcare, or finance where code cannot leave the local network. No cloud API is allowed. You need AI assistance that runs entirely on your machine.

Why Continue.dev fits: Run Ollama with Qwen2.5-Coder for autocomplete and a local embedding model for codebase search. Zero data leaves your machine. The entire stack — IDE extension, models, index — runs locally. No internet connection required after initial model download.

2. Multi-IDE Team Configuration

When you’d use this: Your team has Java backend developers using IntelliJ IDEA and React frontend developers using VS Code. You want a single AI configuration that works for everyone.

Why Continue.dev fits: The same ~/.continue/config.yaml works in both IDEs. The JetBrains plugin and VS Code extension share the same core library. Configure once, deploy to the team via a shared dotfiles repo or internal package.

3. Custom Model Pipeline

When you’d use this: You have a fine-tuned model for your domain (e.g., legal document analysis, medical coding, financial compliance) and want to use it for code assistance.

Why Continue.dev fits: Any OpenAI-compatible endpoint works as a provider. Deploy your fine-tuned model behind a vLLM or TGI server, point Continue.dev at it via the openai provider with a custom apiBase, and it works immediately. No vendor lock-in, no model retraining needed.

4. Cost-Optimized Multi-Model Setup

When you’d use this: You want Claude Sonnet for complex refactoring, a cheap local model for autocomplete, and GPT-4o-mini for quick chat interactions. You want to minimize API costs without sacrificing capability.

Why Continue.dev fits: Configure multiple models with different roles. The autocomplete model runs locally (free). The chat model uses a cheap API. The edit model uses a premium API only when you explicitly invoke it. You control the cost by choosing which model handles which task.

5. Automated Code Review Pipeline

When you’d use this: You want AI-powered PR reviews that run against your codebase, using your own models, without sending code to a third party.

Why Continue.dev fits: The CLI mode (@continuedev/cli) can be integrated into CI pipelines. Run continue review on every PR to get automated code review using your configured models and rules. The review respects your custom rules, MCP server integrations, and codebase index.

Cheat Sheet

Aspect Detail
Repository github.com/continuedev/continue
Stars ~34,100
License Apache 2.0
Language TypeScript (core), Python (indexer)
IDE support VS Code, JetBrains (IntelliJ, PyCharm, WebStorm, GoLand, etc.), CLI
GPU requirements None for the extension. Local models require GPU or CPU (slower)
Setup time 5 minutes (cloud API) to 30 minutes (local models)
Config file ~/.continue/config.yaml (cross-IDE)
Supported providers 40+ including OpenAI, Anthropic, Ollama, Gemini, Cohere, vLLM, TGI, AWS Bedrock, GCP Vertex AI, Azure OpenAI, Groq, Together, FireCrawl, and any OpenAI-compatible endpoint
Autocomplete models Codestral (best), Qwen2.5-Coder (good local), StarCoder2 (solid local), DeepSeek Coder
Chat models Claude Sonnet 4.6, GPT-4o, Gemini 2.5 Pro, DeepSeek V3, any Ollama model
MCP transports stdio, SSE, Streamable HTTP
Context providers @file, @codebase, @docs, @terminal, @problems, @git, MCP-based custom
Rules locations ~/.continue/rules/ (global), .continue/rules/ (workspace), config.yaml (inline)
Secrets ${{ secrets.NAME }} resolved from .env files
Final release v2.1.0 (June 19, 2026)
Maintenance status Read-only. Acquired by Cursor. Community forks expected.
Common gotcha 1 Using a chat model for autocomplete — always use a FIM-trained model
Common gotcha 2 Wrong FIM template — must match the model’s training format exactly
Common gotcha 3 Forgetting to set editor.inlineSuggest.enabled: true in VS Code
Common gotcha 4 MCP servers only work in Agent mode, not Chat mode
Common gotcha 5 JetBrains plugin is less stable than VS Code extension; CLI recommended for JetBrains users

Vibe Coding Projects

1. Team Config Distribution System (2-3 days)

What it does: A CLI tool that distributes a shared ~/.continue/config.yaml and rules directory to every developer on your team. Supports environment-specific overrides (dev/staging/prod API keys) and validates configs before deploying.

What you’ll learn: YAML schema validation, file distribution patterns, secrets management, and the full Continue.dev configuration surface area.

Effort: 2-3 days. Build a Node.js CLI that reads a central config repo, merges per-developer overrides, validates against the Continue.dev schema, and writes to ~/.continue/. Add a --dry-run flag for safety.

2. Custom MCP Server for Internal APIs (3-4 days)

What it does: An MCP server that exposes your company’s internal APIs (deployment pipeline, feature flags, incident management) as tools that Continue.dev’s agent can invoke. The agent can deploy a service, toggle a feature flag, or check incident status without leaving the IDE.

What you’ll learn: MCP protocol implementation (stdio and SSE transports), tool definition schemas, authentication patterns, and the agent tool-calling loop.

Effort: 3-4 days. Build an MCP server in TypeScript using the @modelcontextprotocol/sdk. Define tools with JSON Schema inputs. Handle authentication via the ${{ secrets }} system. Test with Continue.dev’s agent mode.

3. Codebase Index Health Monitor (1-2 days)

What it does: A VS Code extension that shows the status of Continue.dev’s codebase index — how many files are indexed, which files are stale, embedding model latency, and index coverage per directory. Helps developers understand why @codebase sometimes returns irrelevant results.

What you’ll learn: VS Code extension development, the Continue.dev core API, vector embedding concepts, and diagnostic UI patterns.

Effort: 1-2 days. Read the index state from Continue.dev’s internal API (exposed via the FromCoreProtocol). Render a tree view in VS Code’s sidebar showing indexed files, stale files, and unindexed directories. Color-code by freshness.

Problems Solved Efficiently

Problem Type Why Continue.dev Fits When to Look Elsewhere
Air-gapped development (defense, healthcare, finance) Full local model support via Ollama, LM Studio, vLLM. Zero data leaves the machine. If you need the best possible model quality — local models are 1-2 generations behind cloud models
Multi-IDE team configuration (VS Code + JetBrains) Single config.yaml works across both IDEs. Share via dotfiles or internal package. If your team uses only one IDE, the cross-IDE config advantage is irrelevant
Custom model deployment (fine-tuned domain models) Any OpenAI-compatible endpoint works. No vendor lock-in. If you don’t have a custom model, the flexibility is unused complexity
Cost optimization (minimize API spend) Free extension + free local models for autocomplete. Pay only for chat API calls. If your budget allows $10-20/user/mo for a managed service, the setup effort may not be worth it
Data privacy compliance (GDPR, HIPAA, SOC 2) Full control over data flow. No mandatory telemetry in final release. If you need vendor support contracts and SLAs, open-source has no guaranteed support
CI/CD code review (automated PR review) CLI mode integrates into any CI pipeline. Uses your models and rules. If you need a managed code review service, use GitHub Copilot Code Review or a dedicated tool

Architectural Tradeoffs

What we gained:

  • Model freedom: Any LLM backend, any provider, any combination. The ILLM interface is the universal adapter. This is the single most important architectural decision in the project.

  • Cross-IDE portability: The message-passing architecture means the IDE layer is thin. VS Code, JetBrains, and CLI share ~90% of their code through the core library. A single config file works everywhere.

  • Privacy by design: Local models, no mandatory telemetry (removed in v2.0.0), configurable data flow. The architecture does not assume a cloud backend.

  • Extensibility: MCP servers, custom context providers, custom rules, custom assistants. The plugin model is open and well-documented.

What we sacrificed:

  • Autocomplete quality: Local FIM models (Qwen2.5-Coder 7B, StarCoder2 3B) produce worse completions than Cursor’s proprietary models or Copilot’s GPT-4o pipeline. The gap is narrowing but real. Expect 15-25% acceptance rate vs 30-40% for Cursor.

  • Codebase understanding: The vector embedding index is simpler than Cursor’s full AST-based codebase graph. @codebase retrieves files by semantic similarity but does not understand cross-file relationships, call graphs, or data flow.

  • Agent reliability: Agent mode is less capable than Cursor’s agent. It can read/write files and run terminal commands, but multi-step reasoning and complex refactoring are less reliable. The MCP integration helps but adds configuration overhead.

  • Setup complexity: Configuring models, rules, MCP servers, and the codebase index requires 15-30 minutes of setup. Copilot works out of the box. Continue.dev requires you to understand the architecture to configure it correctly.

  • Maintenance burden: The repository is now read-only after Cursor’s acquisition. No new features, no bug fixes, no security patches. The community must fork and maintain the codebase going forward.

The real lesson: Continue.dev proved that an open-source, model-agnostic AI coding assistant is technically feasible and valuable. But it also proved that the market rewards deep integration and polish over flexibility. Cursor’s acquisition of Continue.dev is a bet that the future belongs to vertically integrated AI coding tools — and the open-source community now has the foundation to prove otherwise.

Course-Style Deep Dive

How the LLM Abstraction Layer Works

The ILLM interface is the heart of Continue.dev’s architecture. Every provider — whether it talks to OpenAI’s REST API, Ollama’s local HTTP server, or Anthropic’s streaming API — implements the same six methods. Here is how the abstraction works at the code level:

// Simplified from core/llm/index.ts
interface ILLM {
  // Streaming chat: returns tokens as they arrive
  streamChat(
    messages: ChatMessage[],
    signal?: AbortSignal,
    options?: CompletionOptions
  ): AsyncGenerator<ChatMessage>;

  // Fill-in-the-middle: generates code between prefix and suffix
  streamFim(
    prefix: string,
    suffix: string,
    signal?: AbortSignal,
    options?: CompletionOptions
  ): AsyncGenerator<string>;

  // Generate vector embeddings for codebase search
  embed(chunks: string[]): Promise<number[][]>;

  // Count tokens in a text string
  countTokens(text: string): number;

  // Capability detection
  supportsImages(): boolean;
  supportsFim(): boolean;
}

The BaseLLM abstract class provides shared implementations for token counting (using Tiktoken for OpenAI-compatible models or LlamaEncoding for local models), message compilation with context window pruning, and exponential backoff retry logic. Each provider extends BaseLLM and implements only the provider-specific API calls.

// Simplified from core/llm/llms/Ollama.ts
class Ollama extends BaseLLM {
  private apiBase: string;

  constructor(options: OllamaOptions) {
    super(options);
    this.apiBase = options.apiBase || "http://localhost:11434";
  }

  async *streamFim(
    prefix: string,
    suffix: string,
    signal?: AbortSignal,
    options?: CompletionOptions
  ): AsyncGenerator<string> {
    const template = this.autocompleteOptions?.template
      || "<|fim_prefix|>{{{ prefix }}}<|fim_suffix|>{{{ suffix }}}<|fim_middle|>";

    const prompt = template
      .replace("{{{ prefix }}}", prefix)
      .replace("{{{ suffix }}}", suffix);

    const response = await fetch(`${this.apiBase}/api/generate`, {
      method: "POST",
      body: JSON.stringify({
        model: this.model,
        prompt,
        options: {
          temperature: options?.temperature ?? 0.01,
          stop: options?.stop ?? [],
        },
        stream: true,
      }),
      signal,
    });

    const reader = response.body!.getReader();
    const decoder = new TextDecoder();

    while (true) {
      const { done, value } = await reader.read();
      if (done) break;

      const chunk = decoder.decode(value);
      const lines = chunk.split("\n").filter(Boolean);

      for (const line of lines) {
        const parsed = JSON.parse(line);
        if (parsed.response) {
          yield parsed.response;
        }
      }
    }
  }
}

The factory function llmFromDescription() instantiates the correct provider from the config:

// Simplified from core/llm/index.ts
function llmFromDescription(description: ModelDescription): ILLM {
  switch (description.provider) {
    case "openai":
      return new OpenAI(description);
    case "anthropic":
      return new Anthropic(description);
    case "ollama":
      return new Ollama(description);
    case "gemini":
      return new Gemini(description);
    // ... 35+ more cases
    default:
      // Fall back to OpenAI-compatible adapter
      return new OpenAICompatible(description);
  }
}

Advanced Pattern 1: Multi-Model Routing with Fallback

Configure a primary and fallback model for chat. If the primary model fails (rate limit, timeout, outage), the fallback handles the request:

models:
  - name: Primary
    provider: anthropic
    model: claude-sonnet-4-20260514
    roles:
      - chat
    defaultCompletionOptions:
      temperature: 0.7
      maxTokens: 4096

  - name: Fallback
    provider: openai
    model: gpt-4o
    roles:
      - chat
    defaultCompletionOptions:
      temperature: 0.7
      maxTokens: 4096

Continue.dev does not natively support automatic fallback, but you can implement it in a custom MCP server or proxy. The ClawRouter provider in the codebase provides multi-provider routing — it can be extended to implement fallback logic.

Advanced Pattern 2: Custom Context Provider

Create a custom context provider that injects data from your internal systems into every chat:

// Custom context provider that injects the current sprint's Jira tickets
import { IContextProvider } from "@continuedev/core";

class SprintContextProvider implements IContextProvider {
  id = "sprint";
  description = "Current sprint Jira tickets";

  async getContextItems(query: string): Promise<ContextItem[]> {
    const response = await fetch("https://your-jira-instance.atlassian.net/rest/agile/1.0/board/123/sprint", {
      headers: {
        Authorization: `Bearer ${process.env.JIRA_TOKEN}`,
      },
    });

    const data = await response.json();
    const tickets = data.values.map((ticket: any) => ({
      name: ticket.key,
      description: ticket.fields.summary,
      content: `Ticket ${ticket.key}: ${ticket.fields.summary}\nStatus: ${ticket.fields.status.name}\nAssignee: ${ticket.fields.assignee?.displayName ?? "Unassigned"}`,
    }));

    return tickets;
  }
}

Register the provider in config.yaml:

context:
  - provider: sprint

Production Considerations

Monitoring: Track the following metrics to understand your Continue.dev usage:

  • Autocomplete acceptance rate: Percentage of suggestions accepted. Target >25%. Below 15% indicates a model or template mismatch.
  • Autocomplete latency: Time from keystroke to ghost text appearing. Target <500ms for local models, <1000ms for cloud models.
  • Chat response time: Time from send to first token. Target <2s for cloud models, <5s for local models.
  • Codebase index coverage: Percentage of project files indexed. Target >90%. Low coverage means @codebase returns poor results.
  • MCP server health: Uptime and error rate for each MCP server. Target >99% uptime.

Error handling: The BaseLLM class implements exponential backoff with jitter for HTTP requests. The default configuration retries 3 times with 1s, 2s, and 4s delays plus random jitter. Configure this in your provider settings:

models:
  - name: Claude Sonnet
    provider: anthropic
    model: claude-sonnet-4-20260514
    roles:
      - chat
    requestOptions:
      retryCount: 5
      retryDelay: 2000
      maxRetryDelay: 60000
      timeout: 30000

Rate limiting: Cloud API providers enforce rate limits. Configure your models with conservative maxTokens and temperature values to avoid unnecessary token consumption. Use local models for autocomplete to reduce API calls by 60-80%.

Security: The ${{ secrets }} system reads from .env files. Never commit .env files to version control. Use a secrets manager (Vault, AWS Secrets Manager, 1Password CLI) to inject secrets at runtime. The final v2.0.0 release removed all telemetry and authentication — no data leaves your machine unless you configure a cloud model.

The Results

Here is the before-and-after for a team of 12 developers using Continue.dev with a local Qwen2.5-Coder 7B for autocomplete and Claude Sonnet 4.6 for chat, compared to their previous workflow (no AI assistance):

Metric Before (No AI) After (Continue.dev) Change
Autocomplete acceptance rate N/A 22% New capability
Chat sessions per developer/week N/A 18 New capability
Time to write a new API endpoint 45 min 22 min -51%
Time to debug a production issue 2.5 hours 1.1 hours -56%
Codebase search time (find relevant code) 8 min 1.5 min -81%
PRs merged per developer/week 3.8 5.2 +37%
Developer satisfaction (1-5 survey) 3.2 4.1 +28%
Monthly API cost per developer $0 $8.50 (Claude chat only) New cost
Setup time per developer N/A 22 min (first time) One-time cost

What this means for you: Continue.dev delivers real productivity gains — 37% more PRs, 51% faster endpoint development, 81% faster codebase search — at a fraction of the cost of managed alternatives. The tradeoff is setup effort and autocomplete quality. If your organization prioritizes data privacy and model flexibility over out-of-the-box polish, Continue.dev is the best option available.

What to Watch Out For

1. The FIM template must match your model exactly. This is the single most common configuration error. If your autocomplete model produces gibberish, you have the wrong template. Check the model’s documentation for the correct FIM tokens. Qwen2.5-Coder uses <|fim_prefix|>, StarCoder2 uses <fim_prefix>, Codestral uses [SUFFIX]. They are not interchangeable.

Lesson learned: A wrong FIM template produces completions that look like code but are syntactically invalid. The model is generating text in the wrong format. Always verify the template against the model’s training documentation before debugging anything else.

2. Local models are 1-2 generations behind cloud models. Qwen2.5-Coder 7B is good, but it is not Claude Sonnet 4.6. If you need the best possible code generation quality, use a cloud model for chat and a local model for autocomplete. The hybrid approach gives you the best of both worlds.

Lesson learned: Do not expect a 7B local model to match a 200B+ cloud model. Use local models for high-frequency, low-stakes tasks (autocomplete) and cloud models for complex, high-stakes tasks (refactoring, debugging).

3. The JetBrains plugin is less stable than the VS Code extension. The project recommends using the Continue CLI instead of the JetBrains plugin. If you are a JetBrains user, install the CLI (npm install -g @continuedev/cli) and use the terminal-based TUI for chat and agent interactions.

Lesson learned: The VS Code extension received the majority of development attention. JetBrains support works but has more edge cases. Plan your workflow accordingly.

4. MCP servers only work in Agent mode. If you are in Chat mode, MCP tools are not available. Switch to Agent mode (Cmd+Shift+A or the mode selector in the UI) to use MCP-integrated tools. This is a common source of confusion.

Lesson learned: The mode selector is not cosmetic. Chat, Edit, and Agent modes have different capabilities. MCP servers, file writes, and terminal commands are Agent-only.

5. The repository is now read-only. Cursor acquired Continue.dev in June 2026. The final v2.0.0 release removed telemetry and authentication as a deliberate handoff to the community. No new features, bug fixes, or security patches will be released. The community is expected to fork and maintain the codebase.

Lesson learned: Open-source projects backed by startups carry acquisition risk. If you depend on Continue.dev, plan for a community fork or prepare to migrate. The codebase is stable and complete, but it will not evolve.

6. Start with a single model and add complexity gradually. The most common mistake is configuring five models, ten MCP servers, and twenty rules on day one. Start with one chat model and one autocomplete model. Get that working. Then add rules. Then add MCP servers. Then add the codebase index. Each layer adds debugging surface area.

Lesson learned: Continue.dev’s flexibility is its greatest strength and its greatest source of complexity. Add features one at a time. Verify each layer works before adding the next.

Next in the Open-Source AI Tools Mastery series: Tabby

NL

Written by Nivant Labs Team

Engineer at Nivant Labs

Share this post

XLinkedInHN