·15 min read

Jan: A privacy-first desktop AI assistant (Apache 2.0, 43k stars)

A privacy-first desktop AI assistant running 100% offline with built-in model downloads, extensions, and a clean chat interface.

#jan#desktop#privacy#offline#ai-local

The Problem

Every major AI assistant makes the same trade: convenience for privacy. ChatGPT, Claude, Gemini, Copilot — they all send your prompts to a remote server, process them on someone else’s hardware, and store the conversation history on someone else’s infrastructure. The privacy policy says they won’t train on your data (or gives you an opt-out toggle), but the architecture itself is a trust bet.

For most casual use, that bet is fine. But three categories of users cannot make it:

  • Developers working on proprietary codebases. Pasting internal API schemas, database schemas, or business logic into a cloud AI is a compliance violation waiting to happen. Legal teams at regulated companies (finance, healthcare, defense) have blanket bans on cloud AI tools.
  • Privacy-conscious individuals. Not everyone wants their therapy notes, personal letters, or medical research queries stored on a server they don’t control. The “we don’t train on your data” checkbox is a promise, not a guarantee.
  • Anyone without reliable internet. Airplanes, remote field work, developing economies with expensive data plans — cloud AI is a non-starter when the network is slow, intermittent, or metered.
Dimension Cloud AI (ChatGPT, Claude) Local AI (Jan)
Data residency Remote server (US/EU) Your machine
Internet required Always Never
Model choice Provider’s selection Any GGUF model from HuggingFace
Cost model Subscription ($20/mo) Free (electricity only)
GPU requirement None (server-side) Optional (faster inference)
Auditability Closed source Apache 2.0, fully auditable
Extension system Limited (plugins) Plugin-based extension architecture
API access Rate-limited, metered Unlimited, localhost:1337
Telemetry Mandatory (opt-out) Zero telemetry
Offline capability None Full offline operation

Why this matters: The cloud AI tools are excellent products. But they are not appropriate for every use case. The gap between “I want AI assistance” and “I can send my data to a third party” is wider than most developers realize — until legal review blocks the tool. Jan fills that gap by running the same class of models on your own hardware, with no data leaving your machine, under a fully open-source license you can audit, fork, and deploy in air-gapped environments.

The Investigation

The Jan project (started in 2023 by the Jan team, now at 43,000+ GitHub stars, 150+ contributors, latest release v0.8.2) set out to answer one question: can you build a desktop AI assistant that is as polished as ChatGPT but runs entirely offline?

Finding 1: The inference engine is the easy part. The UX is the hard part.

llama.cpp has been running LLMs on consumer hardware since 2023. The raw capability to run a 7B model on a laptop has existed for years. What was missing was a polished, non-technical user experience — a way to download models, manage conversations, switch between assistants, and configure hardware acceleration without touching a terminal.

Jan’s investigation found that the barrier to local AI adoption was not model quality or inference speed. It was the setup friction. Users had to:

  1. Find and download GGUF files from HuggingFace
  2. Install llama.cpp or a compatible runner
  3. Configure GPU acceleration manually
  4. Use a CLI or a bare-bones web UI
  5. Manage model files, quantization levels, and context lengths themselves

Jan’s answer: a desktop app (built with Tauri + Rust + TypeScript) that bundles llama.cpp, provides a built-in model hub connected to HuggingFace, auto-detects GPU acceleration, and presents a ChatGPT-like interface out of the box.

What this means: The technical capability for local AI has existed since 2023. What was missing was the product polish. Jan is not a breakthrough in inference technology — it is a breakthrough in user experience for local AI.

Finding 2: The extension architecture is the moat.

Most local AI tools are monolithic. You get one interface, one set of features, and whatever the maintainer decides to build. Jan took a different approach: a plugin-based extension system where every feature — model management, conversation storage, hardware monitoring, inference engine control — is a separate extension.

This architecture means:

  • Extensions can be developed independently (the Jan team maintains 8 core extensions; the community can build more)
  • Extensions can be enabled/disabled without touching the core app
  • The same extension system powers both local and cloud model providers
  • The data layer is JSON files on disk, so extensions can read/write directly

What this means: Jan’s extension architecture is not a nice-to-have. It is the architectural decision that makes Jan future-proof. As new model formats, inference engines, and hardware backends emerge, they can be added as extensions without rewriting the core application.

Finding 3: The API server is the hidden killer feature.

Jan includes a built-in OpenAI-compatible API server at localhost:1337. This means any tool that works with OpenAI’s API — VS Code extensions, CI pipelines, automation scripts, custom applications — can use Jan as a drop-in replacement. No code changes needed, just a different base_url.

This turns Jan from a chat app into a local AI infrastructure platform. You can:

  • Use Jan’s chat UI for interactive work
  • Point Continue.dev at localhost:1337/v1 for AI-assisted coding
  • Point Cursor at the same endpoint for inline completions
  • Run batch inference jobs via cURL or the OpenAI Python SDK
  • All using the same local models, with zero data leaving your machine

What this means: The API server is the feature that makes Jan useful beyond chat. It transforms a desktop app into a local AI server that integrates with your entire development toolchain.

The Solution

Jan is a ~200,000-line TypeScript/Rust application (Apache 2.0 license, 43,000+ GitHub stars, 150+ contributors) that runs as a native desktop app on macOS, Windows, and Linux. It bundles llama.cpp for local inference, provides a built-in model hub, and exposes an OpenAI-compatible API server.

┌──────────────────────────────────────────────────────────────────────┐
│                        Jan Desktop Architecture                       │
│                                                                       │
│  ┌──────────────────────────────────────────────────────────────┐   │
│  │                    Tauri Shell (Rust)                          │   │
│  │  ┌────────────────────────────────────────────────────────┐   │   │
│  │  │              React Frontend (TypeScript)                │   │   │
│  │  │  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────┐  │   │   │
│  │  │  │ Chat UI  │  │ Model    │  │ Assistant│  │ API  │  │   │   │
│  │  │  │ (Stream- │  │ Hub      │  │ Manager  │  │ Server│  │   │   │
│  │  │  │  down)   │  │ (Browse) │  │ (Create) │  │ UI   │  │   │   │
│  │  │  └──────────┘  └──────────┘  └──────────┘  └──────┘  │   │   │
│  │  └────────────────────────────────────────────────────────┘   │   │
│  │                                                               │   │
│  │  ┌────────────────────────────────────────────────────────┐   │   │
│  │  │              Extension Layer (TypeScript)               │   │   │
│  │  │  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────┐  │   │   │
│  │  │  │ Model    │  │ Download │  │ Engine   │  │Hard- │  │   │   │
│  │  │  │ Extension│  │ Extension│  │ Mgmt     │  │ware  │  │   │   │
│  │  │  │          │  │          │  │ Extension│  │Mgmt  │  │   │   │
│  │  │  └──────────┘  └──────────┘  └──────────┘  └──────┘  │   │   │
│  │  │  ┌──────────┐  ┌──────────┐  ┌──────────┐            │   │   │
│  │  │  │Assistant │  │Conversa- │  │Inference │            │   │   │
│  │  │  │Extension │  │tional    │  │Cortex    │            │   │   │
│  │  │  │          │  │Extension │  │Extension │            │   │   │
│  │  │  └──────────┘  └──────────┘  └──────────┘            │   │   │
│  │  └────────────────────────────────────────────────────────┘   │   │
│  │                                                               │   │
│  │  ┌────────────────────────────────────────────────────────┐   │   │
│  │  │              Inference Backend (Rust)                    │   │   │
│  │  │  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐  │   │   │
│  │  │  │ llama.cpp   │  │ MLX (macOS)  │  │ Cortex      │  │   │   │
│  │  │  │ (CPU/GPU)   │  │ (Apple       │  │ (Unified    │  │   │   │
│  │  │  │             │  │  Silicon)    │  │  Engine)    │  │   │   │
│  │  │  └──────────────┘  └──────────────┘  └──────────────┘  │   │   │
│  │  └────────────────────────────────────────────────────────┘   │   │
│  │                                                               │   │
│  │  ┌────────────────────────────────────────────────────────┐   │   │
│  │  │              Data Layer (JSON on disk)                   │   │   │
│  │  │  /models/  /threads/  /assistants/  /engines/  /files/  │   │   │
│  │  └────────────────────────────────────────────────────────┘   │   │
│  └──────────────────────────────────────────────────────────────┘   │
│                                                                       │
│  ┌──────────────────────────────────────────────────────────────┐   │
│  │              OpenAI-Compatible API Server                     │   │
│  │  http://127.0.0.1:1337/v1/chat/completions                   │   │
│  │  http://127.0.0.1:1337/v1/models                              │   │
│  │  http://127.0.0.1:1337/v1/embeddings                          │   │
│  └──────────────────────────────────────────────────────────────┘   │
└──────────────────────────────────────────────────────────────────────┘

Here is what each layer does:

  • Tauri Shell: The native application container. Tauri provides the OS-level window, file system access, GPU detection, and system tray integration. The Rust backend handles inference, file I/O, and hardware management. The React frontend renders the UI in a webview.
  • React Frontend: The user interface — chat window, model hub browser, assistant manager, API server configuration. Uses Radix UI primitives, Streamdown for streaming markdown rendering, and Shiki for syntax highlighting.
  • Extension Layer: Eight core extensions that implement every feature as a plugin. Model Extension handles model discovery and downloads. Download Extension manages download progress with pause/resume. Engine Management Extension controls inference backends. Hardware Management Extension monitors CPU/RAM/GPU. Assistant Extension manages AI personalities. Conversational Extension handles chat threads. Inference Cortex Extension provides a unified inference API across backends.
  • Inference Backend: Three backends — llama.cpp (CPU + NVIDIA CUDA + AMD ROCm + Intel Arc), MLX (Apple Silicon only), and Cortex (unified engine abstraction). The app auto-selects the best backend for your hardware.
  • Data Layer: Everything is stored as JSON files on disk. Models go in /models/, threads in /threads/, assistants in /assistants/, engines in /engines/, uploaded files in /files/. This makes the data portable, auditable, and scriptable.
  • API Server: An OpenAI-compatible HTTP server at localhost:1337 that supports chat completions, text completions, model listing, and embeddings. Requires an API key for all requests.

Setup

# Option 1: Download the installer (recommended)
# Visit https://jan.ai and download for your OS
# macOS: Jan-mac-x64.dmg or Jan-mac-arm64.dmg (Apple Silicon)
# Windows: Jan-Windows-x64.exe
# Linux: Jan-linux-x64.deb or Jan-linux-x64.AppImage

# Option 2: Build from source
git clone https://github.com/janhq/jan
cd jan
make dev
# Prerequisites: Node.js >= 20.0.0, Yarn >= 4.5.3, Rust, Make >= 3.81

# After installation, download a model from the Hub:
# Open Jan -> Hub -> Search "Qwen3 8B" -> Click Download
# Or import from HuggingFace:
# Hub -> Paste model ID (e.g., TheBloke/Mistral-7B-v0.1-GGUF) -> Select quantization

# Enable the API server:
# Settings -> Local API Server -> Set API key -> Start Server
# Server ready when logs show: "JAN API listening at http://127.0.0.1:1337"

Production-Grade Configuration

# Connect your development tools to Jan's API server

# Continue.dev (VS Code extension)
# ~/.continue/config.json
{
  "models": [
    {
      "title": "Jan Local",
      "provider": "openai",
      "model": "YOUR_MODEL_ID",
      "apiBase": "http://127.0.0.1:1337/v1",
      "apiKey": "your-api-key"
    }
  ]
}

# Cursor IDE
# Settings -> Models -> Add Custom Model
# Base URL: http://127.0.0.1:1337/v1
# API Key: your-api-key

# Python (OpenAI SDK)
from openai import OpenAI
client = OpenAI(
    api_key="your-api-key",
    base_url="http://127.0.0.1:1337/v1"
)
response = client.chat.completions.create(
    model="YOUR_MODEL_ID",
    messages=[{"role": "user", "content": "Hello!"}]
)

# cURL
curl http://127.0.0.1:1337/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your-api-key" \
  -d '{
    "model": "YOUR_MODEL_ID",
    "messages": [{"role": "user", "content": "Tell me a joke."}]
  }'

Code Walkthrough: The Extension System

The extension system is Jan’s most architecturally distinctive feature. Every capability is a plugin that registers with the core application:

// Conceptual: Jan extension registration
// Each extension lives in /extensions/@janhq/<extension-name>/
// and exports a standard interface

interface JanExtension {
  name: string;
  version: string;
  description: string;

  // Lifecycle hooks
  onLoad(): Promise<void>;
  onUnload(): Promise<void>;

  // Register API routes (served by the local API server)
  registerRoutes(router: Router): void;

  // Register settings panels (shown in Settings UI)
  registerSettings(): SettingsPanel[];

  // Register menu items (shown in app menus)
  registerMenus(): MenuItem[];
}

// Example: Model Extension registers model management
class ModelExtension implements JanExtension {
  name = "@janhq/model-extension";
  version = "1.0.0";

  async onLoad() {
    // Scan /models/ directory for GGUF files
    // Connect to HuggingFace API for model discovery
    // Initialize download queue
  }

  registerRoutes(router: Router) {
    router.get("/models", this.listModels);
    router.post("/models/download", this.downloadModel);
    router.delete("/models/:id", this.deleteModel);
  }
}

The data layer is equally straightforward — everything is JSON on disk:

// Conceptual: Jan data layer
// /jan/data/
//   models/
//     qwen3-8b-q4/
//       model.yml
//       qwen3-8b-q4_k_m.gguf
//   threads/
//     thread-abc123/
//       thread.json      # metadata (title, model, created, updated)
//       messages.jsonl   # append-only log of messages
//   assistants/
//     default/
//       assistant.json   # name, instructions, model, parameters
//   engines/
//     llama-cpp/
//       engine.json      # config (context length, GPU layers, threads)
//   files/
//     upload-xyz.pdf     # uploaded documents

// Reading a thread's messages
const messages = fs.readFileSync(
  "/jan/data/threads/thread-abc123/messages.jsonl", "utf-8"
)
  .split("\n")
  .filter(Boolean)
  .map(line => JSON.parse(line));

How to Use Effectively

Step 1: Choose the right model for your hardware

Jan’s model hub shows compatibility warnings based on your system specs. Use this as a guide:

# Low-RAM systems (8-16 GB)
# Q4 quantized models, 3B-7B parameters
# Recommended: Qwen3 8B Q4_K_M (~5.2 GB), Phi-4 Mini Q4 (~2.5 GB)

# Mid-RAM systems (16-32 GB)
# Q5-Q6 quantized models, 7B-13B parameters
# Recommended: Mistral 7B Q8 (~7.7 GB), Qwen3 14B Q4_K_M (~8.9 GB)

# High-RAM systems (32+ GB)
# Q8 or F16 models, 13B-70B parameters
# Recommended: Llama 3 70B Q4, Qwen3 32B Q4

Step 2: Configure hardware acceleration

Jan auto-detects your GPU, but you can fine-tune:

# Settings -> Model Providers -> llama.cpp
# GPU Offload Layers: Set to max (all layers on GPU)
# Context Length: 4096 (default), increase to 8192 for long documents
# Thread Count: Auto (set to physical core count for CPU inference)

# For Apple Silicon (M1-M4):
# Metal acceleration is auto-enabled
# Settings -> Advanced -> Use Metal: ON

# For NVIDIA GPUs:
# CUDA is auto-detected
# Settings -> Advanced -> CUDA Layers: 35 (for 7B models)

Step 3: Use the API server for tool integration

The API server is where Jan becomes more than a chat app:

# Start the API server
# Settings -> Local API Server -> Start Server

# Test the connection
curl http://127.0.0.1:1337/v1/models \
  -H "Authorization: Bearer your-api-key"

# Use with LangChain
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(
    model="qwen3-8b-q4",
    base_url="http://127.0.0.1:1337/v1",
    api_key="your-api-key"
)

# Use with SillyTavern
# API Type: OpenAI
# Base URL: http://127.0.0.1:1337/v1
# API Key: your-api-key

Step 4: Create custom assistants

Jan lets you create specialized assistants with custom instructions and parameters:

# In Jan UI: Assistants -> Create Assistant
# Name: "Code Reviewer"
# Instructions: "You are a senior code reviewer. Focus on security, performance,
#   and maintainability. Suggest specific improvements with code examples."
# Model: qwen3-8b-q4
# Temperature: 0.3 (lower for more deterministic output)
# Max Tokens: 2048

# Or create manually:
# /jan/data/assistants/code-reviewer/assistant.json
{
  "id": "code-reviewer",
  "name": "Code Reviewer",
  "instructions": "You are a senior code reviewer...",
  "model": "qwen3-8b-q4",
  "parameters": {
    "temperature": 0.3,
    "max_tokens": 2048,
    "top_p": 0.95
  }
}

Production pitfall: Custom assistant instructions are only as good as the model’s instruction-following ability. Smaller models (3B-7B) may ignore detailed instructions. Test your assistant with a few queries before relying on it for production work.

Step 5: Import local GGUF files

If you already have GGUF models downloaded:

# Settings -> Model Providers -> llama.cpp -> Import Model
# Select your GGUF file
# Choose: Link Files (symbolic link, saves disk space)
#      or: Duplicate (copies file into Jan's model directory)

# Jan will scan the file, detect the model architecture,
# and add it to your model list automatically.

Use Cases

1. Offline Code Assistant for Proprietary Codebases

When you’d use this: You work on a proprietary codebase that cannot be sent to cloud AI services. Your legal team has banned ChatGPT, Copilot, and Claude. You need AI assistance for code review, refactoring, and documentation — all on your local machine.

Why Jan fits: Jan runs 100% offline with zero telemetry. You download a coding model (Qwen3 8B, Phi-4 Mini, or DeepSeek Coder), point Continue.dev or Cursor at Jan’s API server, and get AI-assisted coding without any data leaving your machine. The Apache 2.0 license means your legal team can audit the codebase. The JSON-on-disk data layer means you can script compliance checks against the conversation history.

2. Privacy-Sensitive Document Analysis

When you’d use this: You need to analyze confidential documents — legal contracts, medical records, financial statements — with AI assistance. Sending these to a cloud AI is a data breach waiting to happen.

Why Jan fits: Download a model with strong reasoning (Qwen3 14B, Mistral 7B), upload your documents via Jan’s file system, and query them locally. No data ever leaves your machine. The API server lets you script batch analysis pipelines. The JSON data layer makes it easy to archive, audit, or delete processed documents.

3. AI-Powered Writing Assistant

When you’d use this: You write professionally — blog posts, technical documentation, marketing copy — and want AI assistance without your drafts being stored on a third-party server.

Why Jan fits: Create a custom assistant with writing-focused instructions. Use a model with strong language quality (Mistral 7B Q8, Qwen3 8B). The clean chat interface is comfortable for long writing sessions. The API server lets you integrate with your existing writing tools (Obsidian, VS Code, Emacs) through OpenAI-compatible plugins.

4. Local AI Server for Development Teams

When you’d use this: Your team wants to use AI-assisted coding tools (Continue.dev, Cursor, JetBrains AI) but cannot use cloud services due to compliance requirements.

Why Jan fits: Install Jan on a shared development server with a GPU. Enable the API server with 0.0.0.0 binding (behind a VPN or firewall). Every developer on the team points their tools at http://jan-server:1337/v1. One GPU serves the entire team. No per-seat licensing costs. No data leaves your network.

5. AI Research and Experimentation

When you’d use this: You are evaluating different models, quantization levels, and inference configurations. You need a consistent interface for comparing outputs across models.

Why Jan fits: Jan’s model hub lets you download and switch between models instantly. The chat interface provides a consistent UI across models. The API server lets you script automated evaluation pipelines. The extension architecture means you can build custom evaluation tools as Jan extensions. The JSON data layer makes it easy to export conversation histories for analysis.

Cheat Sheet

Aspect Detail
Repository github.com/janhq/jan
License Apache 2.0
Language TypeScript + Rust (Tauri)
GPU Requirements Optional (CPU works); NVIDIA CUDA, AMD ROCm, Apple Metal, Intel Arc
Setup Time 2 minutes (download + install + model download)
Key Features 100% offline, model hub, extension system, OpenAI-compatible API server, custom assistants, zero telemetry
Common Gotchas Insufficient RAM for chosen model; forgetting to set API key for API server; wrong quantization for hardware; not enabling GPU offload
Best Models Qwen3 8B Q4_K_M (balanced), Mistral 7B Q8 (quality), Phi-4 Mini Q4 (fast), Qwen3 14B Q4_K_M (high quality)
Cost Free (electricity only)
API Port localhost:1337
Missing Features No built-in RAG (file chat is basic), no web search, no image generation, no voice mode, no multi-modal (vision models limited)

Vibe Coding Projects

Project 1: Local AI-Powered Code Review Bot

What it does: A Python script that watches your git repository for new pull requests, fetches the diff, sends it to Jan’s API server for review, and posts the review comments back to the PR. Runs entirely on your local machine with no cloud dependencies.

What you’ll learn: How to integrate Jan’s API server with git workflows. How to structure prompts for code review. How to handle streaming responses and parse structured output from local models. How to build a CI-like pipeline that runs on local hardware.

Effort: 2-3 hours. Zero API costs.

Project 2: Local Document Q&A System

What it does: A Streamlit or Gradio app that lets you upload PDFs, markdown files, or text documents, and ask questions about their content. Uses Jan’s API server for inference and a simple embedding-based retrieval system (sentence-transformers) for document search. All processing happens on your local machine.

What you’ll learn: How to build a RAG pipeline with a local LLM backend. How to chunk documents, generate embeddings, and retrieve relevant context. How to structure prompts for document-grounded question answering. How to use Jan’s API server as a drop-in OpenAI replacement.

Effort: 3-4 hours. Zero API costs.

Project 3: Local AI Chatbot with Custom Knowledge Base

What it does: A Slack or Discord bot that answers questions about your team’s internal documentation. The bot connects to Jan’s API server, uses a local embedding model for retrieval, and responds in the chat channel. All data stays on your infrastructure.

What you’ll learn: How to build a chatbot that integrates with messaging platforms. How to use Jan’s API server for both chat completions and embeddings. How to manage conversation context and history with a local model. How to handle rate limiting and concurrent requests on consumer hardware.

Effort: 4-6 hours. Zero API costs.

Problems Solved Efficiently

Problem Type Why Jan Fits When to Look Elsewhere
Offline AI chat 100% offline, no telemetry, polished UI Use ChatGPT/Claude for cloud-quality responses
Proprietary codebase assistance Zero data leaves your machine, Apache 2.0 license Use Copilot for inline autocomplete (if allowed)
Privacy-sensitive document analysis Local inference, auditable data layer Use ChatGPT for complex multi-modal analysis
Local API server for dev tools OpenAI-compatible, unlimited requests Use Ollama for headless/CLI-first workflows
Model evaluation and comparison Built-in model hub, consistent UI across models Use LM Studio for side-by-side model comparison
Air-gapped deployment Fully offline, no telemetry, forkable Use Ollama for server/headless deployment
Custom assistant creation Instructions, parameters, model selection per assistant Use ChatGPT GPTs for cloud-based custom assistants
Team AI server (compliance) One GPU serves team, no per-seat cost Use Ollama for headless server deployment

Architectural Tradeoffs

What we gained:

  • Absolute privacy. No data ever leaves your machine. Zero telemetry. The entire application is auditable under Apache 2.0. For compliance-required environments (finance, healthcare, defense), this is the only viable option among desktop AI assistants.
  • Unlimited API access. Jan’s API server has no rate limits, no token caps, and no metering. You can run batch inference on thousands of documents without worrying about API costs or throttling.
  • Model freedom. Any GGUF model from HuggingFace works. You are not limited to a provider’s model selection. You can run niche models, fine-tuned models, or your own custom quantizations.
  • Extension architecture. Every feature is a plugin. The core app is stable; new capabilities come as extensions. This is the most future-proof architecture among local AI tools.
  • Portable data layer. Everything is JSON on disk. You can back up, version control, or script against your conversation history, model configurations, and assistant definitions.
  • No subscription costs. Jan is free. Models are free. The only cost is electricity. For heavy users, this saves $240+/year compared to ChatGPT Plus.

What we sacrificed:

  • Model quality ceiling. Local models (7B-14B parameters) cannot match GPT-4o or Claude Sonnet 4 on complex reasoning, creative writing, or nuanced instruction following. The gap narrows with each model release, but it still exists.
  • No multi-modal support. Jan’s vision model support is limited. You cannot upload images, diagrams, or screenshots for analysis the way you can with ChatGPT or Claude. The API server does not support image or audio endpoints.
  • No built-in RAG. Jan has basic file upload and chat, but no vector database, no chunking pipeline, and no semantic search. For document Q&A, you need to build your own RAG pipeline on top of the API server.
  • No web search. Jan cannot search the internet. For questions that require up-to-date information, you need a cloud AI or a separate search tool.
  • No voice mode. Jan is text-only. No speech-to-text, no text-to-speech, no voice conversations. Cloud assistants have had this for years.
  • No agentic capabilities. Jan’s MCP support is emerging but not production-ready. You cannot give Jan tools to browse the web, run code, or interact with APIs. The Jan Agent system (PR #7779) is in development but not yet stable.
  • Hardware requirements. Running a 7B model requires 8-16 GB of RAM. Running a 13B model requires 16-32 GB. Running a 70B model requires 32+ GB and a high-end GPU. Cloud AI runs on any device with an internet connection.

The real tradeoff: Jan is not a replacement for ChatGPT or Claude. It is a complement for the use cases where cloud AI is inappropriate — proprietary code, sensitive documents, air-gapped environments, and unlimited API access. The model quality gap is real, but it is shrinking. For the use cases Jan targets, the privacy and control benefits outweigh the quality difference.

Course-Style Deep Dive

Under the Hood: How Jan Runs Models

Jan uses llama.cpp as its primary inference engine. Here is what happens when you send a message:

  1. Model Loading. When you select a model, Jan loads the GGUF file into memory. The GGUF format is a binary format that stores the model weights, tokenizer, and metadata in a single file. Jan reads the model’s configuration (context length, layer count, vocabulary size) from the GGUF header.

  2. GPU Offloading. Jan checks your hardware and offloads as many transformer layers to the GPU as possible. On Apple Silicon, this uses Metal Performance Shaders. On NVIDIA GPUs, this uses CUDA. On AMD GPUs, this uses ROCm or Vulkan. The remaining layers run on CPU.

  3. Tokenization. Your input text is converted to tokens using the model’s tokenizer (typically a BPE or SentencePiece tokenizer). The tokenizer is embedded in the GGUF file.

  4. Inference. The tokenized input is fed through the transformer layers. Each layer computes attention and feed-forward operations. The model generates one token at a time, using the previous token as input. This is the autoregressive decoding loop.

  5. Streaming. As each token is generated, Jan streams it to the UI using Server-Sent Events (SSE). The Streamdown renderer in the frontend displays the tokens incrementally with markdown formatting and syntax highlighting.

  6. Context Management. Jan maintains a sliding window of recent conversation history. When the context window fills up, older messages are dropped. The context length is configurable per model (default 4096 tokens, up to 32K for supported models).

Advanced Pattern 1: Batch Inference Pipeline

"""Batch inference with Jan's API server."""
from openai import OpenAI
import json
from pathlib import Path

client = OpenAI(
    api_key="your-api-key",
    base_url="http://127.0.0.1:1337/v1"
)

def batch_inference(prompts: list[str], model: str = "qwen3-8b-q4") -> list[str]:
    """Run batch inference on a list of prompts."""
    results = []
    for prompt in prompts:
        response = client.chat.completions.create(
            model=model,
            messages=[{"role": "user", "content": prompt}],
            temperature=0.3,
            max_tokens=1024,
        )
        results.append(response.choices[0].message.content)
    return results

# Example: Classify 100 customer support tickets
tickets = [
    "My order hasn't arrived in 2 weeks",
    "Can I change my shipping address?",
    "The product is defective",
    # ... 97 more tickets
]

classifications = batch_inference([
    f"Classify this support ticket into one of: [shipping, returns, defect, billing, other]\nTicket: {t}\nClassification:"
    for t in tickets
])

# Save results
results = [{"ticket": t, "classification": c} for t, c in zip(tickets, classifications)]
Path("classifications.json").write_text(json.dumps(results, indent=2))

Advanced Pattern 2: Custom Assistant with Structured Output

"""Use Jan's API server with structured output parsing."""
from openai import OpenAI
import json
import re

client = OpenAI(
    api_key="your-api-key",
    base_url="http://127.0.0.1:1337/v1"
)

def extract_json(text: str) -> dict:
    """Extract JSON from model output (handles markdown code fences)."""
    # Try direct JSON parse first
    try:
        return json.loads(text)
    except json.JSONDecodeError:
        pass

    # Try extracting from code fence
    match = re.search(r'```(?:json)?\s*\n(.*?)\n```', text, re.DOTALL)
    if match:
        try:
            return json.loads(match.group(1))
        except json.JSONDecodeError:
            pass

    raise ValueError(f"Could not extract JSON from: {text[:200]}...")

def analyze_pr_diff(diff: str) -> dict:
    """Analyze a PR diff and return structured review."""
    prompt = f"""Review this pull request diff and return a JSON object with:
- "summary": one-sentence summary of the change
- "issues": list of objects with "severity" (critical/warning/info), "file", "line", "description"
- "suggestions": list of improvement suggestions
- "verdict": "approve", "changes-requested", or "needs-discussion"

Diff:
```diff
{diff[:4000]}

Return ONLY valid JSON, no other text.“”“

response = client.chat.completions.create(
    model="qwen3-8b-q4",
    messages=[{"role": "user", "content": prompt}],
    temperature=0.2,
    max_tokens=2048,
)

return extract_json(response.choices[0].message.content)

### Production Considerations

**Memory management.** Local models are memory-intensive. A 7B Q4 model uses ~5 GB of RAM. A 13B Q4 model uses ~9 GB. A 70B Q4 model uses ~35 GB. Monitor your system's memory pressure and choose models accordingly. Jan's model hub shows "Slow on your device" and "Not enough RAM" warnings — trust them.

**GPU offload tuning.** The number of layers offloaded to the GPU is the single most impactful performance setting. On a 7B model with 35 layers:
- 0 layers offloaded: CPU-only, ~5-10 tok/s
- 20 layers offloaded: partial GPU, ~15-25 tok/s
- 35 layers offloaded: full GPU, ~30-50 tok/s

Set GPU offload to the maximum that fits in your VRAM. If you see "out of memory" errors, reduce the layer count.

**Context length tradeoffs.** Longer context windows use more memory and slow down inference. A 7B model at 4096 context uses ~5 GB. At 8192 context, it uses ~6 GB. At 16384 context, it uses ~8 GB. Only increase context length when you need it for long documents or extended conversations.

**Concurrent requests.** Jan's API server handles one request at a time per model. If you send concurrent requests, they queue. For team use, consider running multiple Jan instances with different models, or use a load balancer in front of multiple instances.

**Disk space.** GGUF model files are large. A 7B Q4 model is ~4 GB. A 13B Q4 model is ~8 GB. A 70B Q4 model is ~40 GB. Plan your storage accordingly. Jan's download manager supports pause and resume, which helps with large downloads.

## The Results

| Metric | Before Jan | After Jan | Improvement |
|--------|-----------|-----------|-------------|
| Data privacy | Promises and opt-out toggles | Zero data leaves machine | Absolute privacy guarantee |
| API rate limits | 10-60 req/min (cloud) | Unlimited (local) | No practical limit |
| Monthly cost | $20/mo (ChatGPT Plus) | $0 (electricity only) | $240/yr savings |
| Model selection | Provider's catalog | Any GGUF on HuggingFace | Unlimited choice |
| Offline capability | None | Full offline operation | Works anywhere |
| API integration | Rate-limited, metered | Unlimited, localhost:1337 | No throttling |
| Auditability | Closed source | Apache 2.0, JSON data layer | Full transparency |
| Team deployment | Per-seat licensing | One GPU serves team | No per-user cost |
| Model quality | GPT-4o / Claude Sonnet 4 | 7B-14B local models | Quality gap exists |
| Setup time | 0 minutes (browser) | 2 minutes + model download | One-time setup cost |

**What this means for you:** Jan is not a ChatGPT replacement for every use case. It is the right tool for the use cases where cloud AI is inappropriate — proprietary code, sensitive documents, air-gapped environments, and unlimited API access. The model quality gap is real but narrowing with each release. For the privacy and compliance use cases Jan targets, the tradeoff is worth it.

## What to Watch Out For

1. **Match the model to your hardware.** The #1 mistake new Jan users make is downloading a model that exceeds their available RAM. A 13B Q4 model needs 16 GB of free RAM. A 70B Q4 model needs 35+ GB. Check your system's memory pressure before downloading. Start with a 3B or 7B model and work up.

2. **Set the API key before using the API server.** The API server requires an API key for all requests. If you start the server without setting a key, every request returns 401 Unauthorized. Set the key in Settings -> Local API Server before making your first API call.

3. **Enable GPU offload for acceptable performance.** CPU-only inference on a 7B model runs at 5-10 tokens per second — usable but slow. With GPU offload, the same model runs at 30-50 tok/s. Check Settings -> Advanced -> GPU Offload Layers and set it to the maximum your GPU supports.

4. **Use Q4 or Q5 quantizations for most hardware.** Q4 offers the best speed-to-quality ratio. Q5 is slightly better quality at slightly slower speed. Q8 and F16 are for high-end hardware only. Start with Q4 and upgrade if your system handles it well.

5. **Do not expose the API server to the internet.** The default host is `127.0.0.1`, which is local-only. If you change it to `0.0.0.0` for network access, put it behind a VPN or firewall. The API key is the only authentication — there is no user management, no rate limiting, and no access logging.

6. **Monitor disk usage.** GGUF files are large and accumulate quickly. A few models can consume 50+ GB. Jan's download manager shows file sizes before downloading. Periodically delete models you no longer use from the Hub interface.

7. **Understand the model quality ceiling.** Local models are good, but they are not GPT-4o. For tasks that require deep reasoning, nuanced instruction following, or creative writing, you may get better results from a cloud AI. Use Jan for the tasks where privacy matters, and cloud AI for the tasks where quality matters.

> **Lesson 1:** "I downloaded a 70B model on my 32 GB Mac and wondered why the system froze. The model needs 35 GB just for the weights. Read the RAM requirements before you click download." — Jan community, r/LocalLLaMA

> **Lesson 2:** "The API server is the feature that makes Jan useful beyond chat. I use it for batch inference, CI pipelines, and as a backend for my custom tools. The unlimited rate limit is a bigger deal than I expected." — Jan community, Hacker News

> **Lesson 3:** "I spent a week trying to get a 13B model to write good code. Then I switched to a 7B Q8 model and it was better. Quantization quality matters more than parameter count for coding tasks." — Jan community, r/LocalLLaMA

### Advice for Getting Started

1. Download Jan from [jan.ai](https://jan.ai/) and install it. The process takes 2 minutes.
2. Open the Hub and download a Q4 model that fits your RAM. Qwen3 8B Q4_K_M is a safe starting point for most systems.
3. Send a few test messages to verify the model works. Adjust the temperature (0.3 for factual, 0.7 for creative).
4. Enable the API server in Settings. Set an API key. Test with `curl`.
5. Point your development tools (Continue.dev, Cursor, custom scripts) at `localhost:1337/v1`.
6. Create custom assistants for your common workflows — code review, writing, analysis.
7. Experiment with different models and quantizations. The model hub makes switching instant.
8. When you outgrow Jan's capabilities (multi-modal, web search, agentic tools), use cloud AI for those specific tasks. Jan handles the privacy-sensitive work.

---

*Next in the Open-Source AI Tools Mastery series: [LlamaFile](/blog/ai-tools-llamafile)*
NL

Written by Nivant Labs Team

Engineer at Nivant Labs

Share this post

XLinkedInHN