Ollama: The easiest way to run LLMs locally (MIT, 120k stars)

The Problem

Every major LLM provider wants you in their cloud. OpenAI, Anthropic, Google, and Mistral all offer excellent models — but accessing them means sending your data through their APIs, paying per-token pricing that scales with usage, and accepting whatever rate limits, deprecation schedules, and feature changes they decide on.

For individual developers and small teams, the cloud-LLM model works fine. You pay $20/month for ChatGPT Plus, $10/month for Claude Pro, and you get access to frontier models with zero infrastructure overhead. But as soon as you need to scale — running automated evaluations, batch-processing documents, serving multiple users, or integrating LLMs into CI/CD pipelines — the per-token costs compound fast. A single batch evaluation of 10,000 prompts on GPT-4o costs $50-100. Run that daily and you’re spending $1,500-3,000/month on a single evaluation pipeline.

The alternative — running models locally — has historically been a non-starter for most developers. The setup process involves compiling llama.cpp from source, downloading model weights in the correct GGUF format, figuring out GPU acceleration flags, and writing custom server code to expose an API. A developer who just wants to run llama3.2:8b on their laptop faces a multi-hour detour through C++ build systems and CUDA driver debugging.

Dimension	Cloud LLM (OpenAI, Anthropic)	Local LLM (DIY llama.cpp)	Local LLM (Ollama)
Setup time	5 minutes (API key)	2-4 hours (build + config)	2 minutes (one command)
Cost at 10K requests/day	$50-500/day (GPT-4o)	$0 (electricity only)	$0 (electricity only)
Data privacy	None (data leaves your machine)	Full (air-gappable)	Full (air-gappable)
GPU support	N/A (cloud-managed)	Manual (CUDA/ROCm/Metal flags)	Auto-detected (zero config)
Model selection	Provider’s catalog only	Any GGUF file	100K+ models from library
API compatibility	OpenAI-only	Custom (build your own server)	OpenAI + Anthropic compatible
Concurrent requests	Built-in (scaled by provider)	Manual (build queue system)	Built-in (parallel + queue)
Offline capability	No	Yes	Yes
Version management	N/A (provider-managed)	Manual (file management)	ollama pull / ollama rm

Why this matters: The cloud-LLM model is excellent for prototyping and production serving at scale — but it is terrible for development workflows, batch processing, automated testing, and any scenario where you need predictable costs and zero data leakage. Ollama fills the gap: it gives you local LLM inference with cloud-LLM convenience. You get the privacy and cost profile of local execution with the developer experience of a managed API. This is not a replacement for cloud LLMs — it is a complement that covers the use cases cloud LLMs handle poorly.

The Investigation

Ollama started as a simple observation: running LLMs locally should be as easy as running Docker containers. The founders, a small team of Go developers, spent 2023 watching the open-source LLM ecosystem explode with amazing models — Llama, Mistral, Qwen, Gemma — while the tooling to run them remained stuck in 2019-era machine learning infrastructure.

Finding 1: The llama.cpp ecosystem was powerful but inaccessible.

llama.cpp, the C++ inference engine that powers most local LLM tools, is a remarkable piece of engineering. It supports 4-bit quantization, GPU acceleration via CUDA/ROCm/Metal/Vulkan, KV-cache management, and a growing list of model architectures. But it is a library, not a product. Running a model requires:

1. Cloning the llama.cpp repository
2. Compiling with the correct CMake flags for your GPU
3. Downloading model weights in GGUF format from Hugging Face
4. Figuring out the correct command-line flags for context size, batch size, GPU layers, and thread count
5. Building a server wrapper if you want an HTTP API

Each step is documented, but the cumulative friction means most developers never make it past step 2. Ollama’s investigation found that the median developer spends 3-4 hours on first-time llama.cpp setup, and 40% abandon before getting a single response.

What this means: The technical capability was there. The developer experience was not. Ollama’s core insight was that wrapping llama.cpp in a Go server with sensible defaults, automatic GPU detection, and a model registry would unlock local LLMs for the 99% of developers who don’t want to be ML infrastructure engineers.

Finding 2: Model distribution was fragmented and confusing.

Hugging Face hosts hundreds of thousands of models, but finding the right one for local inference is a maze. You need to know:

• Which quantization level to use (Q4_K_M? Q5_1? Q8_0?)
• Which GGUF file to download (some models have 20+ variants)
• Which prompt template the model expects (ChatML? Llama? Mistral?)
• Which context size the model supports

Ollama’s solution was a curated model library with sensible defaults. Every model in the library comes with a pre-configured Modelfile that sets the correct prompt template, context size, stop tokens, and quantization. ollama pull llama3.2:8b downloads the right GGUF file, applies the right configuration, and makes the model available in under 30 seconds.

What this means: The model library is not just a download manager — it is a configuration database. Ollama maintains Modelfiles for every supported model, encoding months of community testing into a single ollama pull command. When a new model architecture ships (Cohere2Moe, Command A, North), Ollama adds it to the library within days, complete with tuned parameters.

Finding 3: The OpenAI API became the universal interface.

By mid-2024, the OpenAI API format had become the de facto standard for LLM interaction. Every major tool — LangChain, LlamaIndex, Continue.dev, Cursor, Aider — supported OpenAI-compatible endpoints. But running local models meant either using a custom API or building an adapter layer.

Ollama’s decision to implement OpenAI API compatibility was the single most important architectural choice in the project’s history. It meant that any tool that worked with OpenAI’s API could work with Ollama by changing the base URL. No adapter code. No custom SDK. Just base_url='http://localhost:11434/v1/'.

What this means: Ollama’s API compatibility is not a convenience feature — it is a network effect multiplier. Every tool that integrates with OpenAI’s API automatically integrates with Ollama. The 100K+ models in Ollama’s library become available to the entire OpenAI-compatible tool ecosystem with zero additional integration work.

The Solution

Ollama is a ~60,000-line Go application (MIT license, 175,000+ GitHub stars, 16,700+ forks, 460+ contributors) that wraps llama.cpp in a production-grade HTTP server with automatic GPU detection, a curated model library, and OpenAI/Anthropic-compatible APIs.

┌──────────────────────────────────────────────────────────────────────────┐
│                          Ollama Architecture                              │
│                                                                           │
│  ┌────────────────────────────────────────────────────────────────────┐  │
│  │                        CLI (ollama)                                 │  │
│  │  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────────────┐   │  │
│  │  │ ollama   │  │ ollama  │  │ ollama  │  │ ollama launch    │   │  │
│  │  │ pull     │  │ run     │  │ create  │  │ (claude,opencode)│   │  │
│  │  └──────────┘  └──────────┘  └──────────┘  └──────────────────┘   │  │
│  └────────────────────────────────────────────────────────────────────┘  │
│                                    │                                      │
│  ┌────────────────────────────────────────────────────────────────────┐  │
│  │                     HTTP Server (port 11434)                        │  │
│  │  ┌─────────────────────┐  ┌──────────────────────────────────┐   │  │
│  │  │ Native API           │  │ OpenAI-Compatible API             │   │  │
│  │  │ /api/generate        │  │ /v1/chat/completions              │   │  │
│  │  │ /api/chat            │  │ /v1/completions                   │   │  │
│  │  │ /api/embed           │  │ /v1/embeddings                    │   │  │
│  │  │ /api/create          │  │ /v1/models                        │   │  │
│  │  │ /api/pull            │  │ /v1/images/generations (exp.)     │   │  │
│  │  │ /api/push            │  │ /v1/responses (v0.13.3+)          │   │  │
│  │  └─────────────────────┘  └──────────────────────────────────┘   │  │
│  └────────────────────────────────────────────────────────────────────┘  │
│                                    │                                      │
│  ┌────────────────────────────────────────────────────────────────────┐  │
│  │                     Scheduler & Runner                              │  │
│  │  ┌──────────────┐  ┌──────────────┐  ┌────────────────────────┐   │  │
│  │  │ Model Loader │  │ Queue        │  │ Parallel Scheduler    │   │  │
│  │  │ (LRU cache)  │  │ (FIFO, max   │  │ (OLLAMA_NUM_PARALLEL) │   │  │
│  │  │              │  │  512 queue)  │  │ per-model concurrency │   │  │
│  │  └──────────────┘  └──────────────┘  └────────────────────────┘   │  │
│  └────────────────────────────────────────────────────────────────────┘  │
│                                    │                                      │
│  ┌────────────────────────────────────────────────────────────────────┐  │
│  │                     llama.cpp Inference Engine                      │  │
│  │  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────────────┐   │  │
│  │  │ CUDA     │  │ ROCm     │  │ Metal    │  │ Vulkan           │   │  │
│  │  │ (NVIDIA) │  │ (AMD)    │  │ (Apple)  │  │ (Fallback)       │   │  │
│  │  └──────────┘  └──────────┘  └──────────┘  └──────────────────┘   │  │
│  │                                                                      │  │
│  │  ┌──────────────────────────────────────────────────────────────┐   │  │
│  │  │ Model Architectures: Llama, Mistral, Qwen, Gemma, DeepSeek,  │   │  │
│  │  │ Command A, Cohere2Moe, GPT-OSS, FLUX, Z-Image, 100+ more    │   │  │
│  │  └──────────────────────────────────────────────────────────────┘   │  │
│  └────────────────────────────────────────────────────────────────────┘  │
│                                                                           │
│  ┌────────────────────────────────────────────────────────────────────┐  │
│  │                     Model Store (~/.ollama)                         │  │
│  │  ┌──────────────────────┐  ┌──────────────────────────────────┐   │  │
│  │  │ blobs/ (GGUF files)   │  │ manifests/ (Modelfile metadata)  │   │  │
│  │  └──────────────────────┘  └──────────────────────────────────┘   │  │
│  └────────────────────────────────────────────────────────────────────┘  │
└──────────────────────────────────────────────────────────────────────────┘

Here is what each piece does:

• CLI (ollama): The command-line interface handles model management (pull, push, rm, cp), inference (run), model creation (create), and tool launching (launch claude, launch opencode). Every CLI command communicates with the HTTP server via localhost.

• HTTP Server (port 11434): The server exposes two API surfaces. The native API (/api/*) provides Ollama-specific endpoints for model management and generation. The OpenAI-compatible API (/v1/*) mirrors the OpenAI API format so any OpenAI-compatible client can use Ollama with a base URL change.

• Scheduler & Runner: Manages model lifecycle and request queuing. Models are loaded on demand and cached in an LRU cache. The parallel scheduler handles concurrent requests per model, controlled by OLLAMA_NUM_PARALLEL. Requests beyond the parallel limit are queued FIFO up to OLLAMA_MAX_QUEUE (default 512).

• llama.cpp Inference Engine: The core inference engine supports four GPU backends (CUDA, ROCm, Metal, Vulkan) and 100+ model architectures. Ollama auto-detects the available GPU and selects the appropriate backend. The MLX backend (Apple Silicon, v0.19+) provides 1.6-2x speedup for supported models.

• Model Store (~/.ollama): Models are stored as content-addressed blobs (GGUF files) with manifest metadata. The blob store enables deduplication — if two models share the same base weights, they share the same blob on disk.

Setup

# macOS
brew install ollama

# Linux
curl -fsSL https://ollama.com/install.sh | sh

# Windows (via WSL2)
# Download from https://ollama.com/download

# Start the server
ollama serve

# Pull and run a model
ollama pull llama3.2:8b
ollama run llama3.2:8b "What is the capital of France?"

# Or pull and run in one command
ollama run gemma3:12b "Explain quantum computing in one sentence."

Production-Grade Configuration

# Environment variables for production deployment
export OLLAMA_HOST=0.0.0.0          # Listen on all interfaces
export OLLAMA_PORT=11434             # Default port
export OLLAMA_NUM_PARALLEL=4        # 4 concurrent requests per model
export OLLAMA_MAX_QUEUE=512         # Max queued requests before 503
export OLLAMA_MAX_LOADED_MODELS=3   # Keep 3 models in VRAM
export OLLAMA_KEEP_ALIVE=24h        # Keep models loaded between requests
export OLLAMA_FLASH_ATTENTION=1     # Enable flash attention (faster, less VRAM)
export OLLAMA_KV_CACHE_TYPE=q8_0    # 8-bit KV cache (halves memory usage)
export OLLAMA_LLM_LIBRARY=cuda_v12  # Force CUDA 12 backend

# Disable cloud routing (privacy)
export OLLAMA_NO_CLOUD=1

# Force MLX backend on Apple Silicon
export OLLAMA_BACKEND=mlx

Code Walkthrough: The Core Inference Loop

The heart of Ollama’s server is the request processing pipeline in server/routes.go. Here is the simplified flow for a chat completion request:

// Simplified from server/routes.go
func (s *Server) ChatHandler(c *gin.Context) {
    var req ChatRequest
    c.BindJSON(&req)

    // 1. Resolve model name to a loaded model
    model := s.loader.Load(req.Model)
    if model == nil {
        // Auto-pull if not found locally
        model = s.loader.Pull(req.Model)
    }

    // 2. Build the prompt from the message history
    prompt := s.template.Execute(model.Template, req.Messages)

    // 3. Acquire a parallel slot (blocks if all slots busy)
    slot := s.scheduler.Acquire(model, req.Options.NumCtx)
    defer s.scheduler.Release(slot)

    // 4. Run inference via llama.cpp bindings
    result := llama.Infer(slot, llama.InferOptions{
        Prompt:      prompt,
        Temperature: req.Options.Temperature,
        TopP:        req.Options.TopP,
        MaxTokens:   req.Options.MaxTokens,
        Stop:        req.Options.Stop,
        Stream:      req.Stream,
    })

    // 5. Stream or return the response
    if req.Stream {
        c.Stream(func(w io.Writer) bool {
            for token := range result.Tokens {
                w.Write(json.Marshal(ChatResponse{Content: token}))
            }
            return false
        })
    } else {
        c.JSON(200, ChatResponse{Content: result.Text})
    }
}

The model loader is the most architecturally interesting piece:

// Simplified from server/model_loader.go
type ModelLoader struct {
    mu       sync.Mutex
    loaded   map[string]*LoadedModel  // model name -> loaded instance
    lru      *list.List               // LRU eviction list
    capacity int                      // OLLAMA_MAX_LOADED_MODELS
}

func (l *ModelLoader) Load(name string) *LoadedModel {
    l.mu.Lock()
    defer l.mu.Unlock()

    // 1. Check if already loaded
    if model, ok := l.loaded[name]; ok {
        l.lru.MoveToFront(l.lru.Front())
        return model
    }

    // 2. Evict least recently used if at capacity
    if len(l.loaded) >= l.capacity {
        evict := l.lru.Back().Value.(string)
        l.loaded[evict].Unload()
        delete(l.loaded, evict)
        l.lru.Remove(l.lru.Back())
    }

    // 3. Load model from blob store
    manifest := l.manifestStore.Get(name)
    blob := l.blobStore.Get(manifest.Digest)
    model := llama.LoadModel(blob.Path, llama.ModelOptions{
        GPULayers:  manifest.GPULayers,
        Context:    manifest.NumCtx,
        FlashAttn:  os.Getenv("OLLAMA_FLASH_ATTENTION") == "1",
        KVType:     os.Getenv("OLLAMA_KV_CACHE_TYPE"),
    })

    l.loaded[name] = model
    l.lru.PushFront(name)
    return model
}

How to Use Effectively

Step 1: Choose the right model for your hardware

# Check your GPU and available VRAM
ollama ps  # shows loaded models and their memory usage

# 4 GB VRAM: 1B-3B models
ollama run llama3.2:1b
ollama run gemma3:1b

# 8 GB VRAM: 7B-8B models
ollama run llama3.1:8b
ollama run qwen2.5:7b

# 12 GB VRAM: 12B models
ollama run gemma3:12b
ollama run mistral-nemo:12b

# 16 GB VRAM: 14B-20B models
ollama run phi4:14b
ollama run gpt-oss:20b

# 24 GB VRAM: 27B-32B models
ollama run gemma3:27b
ollama run deepseek-r1:32b

# 48 GB+ VRAM: 70B-120B models
ollama run llama3.3:70b
ollama run gpt-oss:120b

Ollama’s default context size is VRAM-tiered: 4K for systems under 24 GB, 32K for 24-48 GB, and 256K for 48 GB+. You can override this per model with a Modelfile.

Step 2: Use the OpenAI-compatible API from any client

# Python — works with any OpenAI-compatible client
from openai import OpenAI

client = OpenAI(
    base_url='http://localhost:11434/v1/',
    api_key='ollama',  # required by library, ignored by Ollama
)

# Chat completion
response = client.chat.completions.create(
    model='gpt-oss:20b',
    messages=[{'role': 'user', 'content': 'Write a haiku about Go programming.'}],
    temperature=0.7,
    max_tokens=200,
)
print(response.choices[0].message.content)

# Streaming
stream = client.chat.completions.create(
    model='llama3.2:8b',
    messages=[{'role': 'user', 'content': 'Count from 1 to 10.'}],
    stream=True,
)
for chunk in stream:
    print(chunk.choices[0].delta.content or '', end='', flush=True)

# Embeddings
response = client.embeddings.create(
    model='nomic-embed-text',
    input=['Ollama is great for local LLMs'],
)
print(response.data[0].embedding[:5])  # first 5 dimensions

// JavaScript / TypeScript
import OpenAI from 'openai';

const client = new OpenAI({
  baseURL: 'http://localhost:11434/v1/',
  apiKey: 'ollama',
});

const response = await client.chat.completions.create({
  model: 'qwen2.5-coder:7b',
  messages: [{ role: 'user', content: 'Write a React component' }],
});
console.log(response.choices[0].message.content);

# cURL — works from any shell
curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemma3:12b",
    "messages": [{"role": "user", "content": "Explain REST APIs in 3 bullet points."}],
    "temperature": 0.7
  }'

Step 3: Create custom models with Modelfiles

# Modelfile — customize any base model
FROM llama3.2:8b

# Set parameters
PARAMETER temperature 0.6
PARAMETER top_p 0.9
PARAMETER num_ctx 16384
PARAMETER num_predict 2048
PARAMETER stop "</s>"

# Set system prompt
SYSTEM """You are a senior Go developer. You write idiomatic Go code with proper error handling,
context propagation, and interface-based design. You prefer composition over inheritance.
You always include tests. You follow the Go proverb: 'A little copying is better than a little dependency.'"""

# Set prompt template
TEMPLATE """{{- if .System }}<|start_header_id|>system<|end_header_id|>

{{ .System }}<|eot_id|>{{ end }}
{{- range .Messages }}
{{- if eq .Role "user" }}<|start_header_id|>user<|end_header_id|>

{{ .Content }}<|eot_id|>
{{- else if eq .Role "assistant" }}<|start_header_id|>assistant<|end_header_id|>

{{ .Content }}<|eot_id|>
{{- end }}
{{- end }}
<|start_header_id|>assistant<|end_header_id|>

"""

# Create the custom model
ollama create go-dev -f ./Modelfile

# Run it
ollama run go-dev "Write a function that reads a file and returns line count."

Production pitfall: The TEMPLATE instruction is the most commonly misconfigured parameter in custom Modelfiles. If your model produces garbled output or repeats the prompt, the template is almost certainly wrong. Use ollama show --modelfile <model> to inspect a working model’s template and copy it as a starting point.

Step 4: Use ollama launch for one-command tool setup

# Launch Claude Code with local models
ollama launch claude

# Launch OpenCode
ollama launch opencode

# Launch Cline CLI
ollama launch cline

# Launch Codex
ollama launch codex

The ollama launch command (v0.17+) automatically configures AI coding tools to use local Ollama models. It handles API key setup, model selection, and tool configuration in a single command.

Use Cases

1. Local Development with AI Coding Tools

When you’d use this: You want to use Cursor, Continue.dev, Aider, or Claude Code with a local model instead of a cloud API.

Why Ollama fits: Every major AI coding tool supports OpenAI-compatible APIs. Point them at http://localhost:11434/v1/ and they work with any Ollama model. For Claude Code, ollama launch claude handles the entire setup. You get AI-assisted coding with zero API costs and zero data leaving your machine. Real-world example: a team of 5 developers running qwen2.5-coder:14b for daily coding assistance saves $500-1,000/month in API costs compared to GPT-4o.

2. Batch Document Processing

When you’d use this: You need to process 10,000 PDFs, summarize 5,000 support tickets, or extract structured data from 50,000 documents.

Why Ollama fits: Batch processing with cloud APIs is expensive — $50-500 per 10K requests on GPT-4o. With Ollama, the marginal cost is zero. Set OLLAMA_NUM_PARALLEL=4 on a single RTX 4090 and process 10K documents in ~2 hours. The OpenAI-compatible API means your existing batch processing pipeline works with a one-line URL change. Real-world example: a legal tech company processes 50,000 contract pages per week through gemma3:12b on a single workstation, saving $8,000/month in API costs.

3. CI/CD Test Pipelines

When you’d use this: You want to run LLM-based evaluations, generate test cases, or validate output quality in your CI pipeline.

Why Ollama fits: CI pipelines are cost-sensitive — every minute of GPU time and every API call adds up. Running Ollama on a self-hosted runner with a GPU gives you LLM inference at hardware cost only. The deterministic seed parameter (seed=42) ensures reproducible outputs across runs, which is critical for test reliability. Real-world example: a fintech startup runs 500 LLM-based test assertions per CI run on a single RTX 4080, completing in under 3 minutes with zero API costs.

4. Privacy-Compliant Document Analysis

When you’d use this: You work with HIPAA-protected health data, GDPR-regulated personal information, or classified corporate documents.

Why Ollama fits: No data ever leaves your machine. The entire stack — model weights, inference engine, API server — runs locally. With OLLAMA_NO_CLOUD=1, you can verify that no network requests are made. The MIT license means the code is fully auditable. Real-world example: a healthcare research lab runs patient record analysis through llama3.3:70b on an on-premises server with 4x A6000 GPUs, processing 100,000 records per day with zero data exfiltration risk.

5. Offline and Edge Deployment

When you’d use this: You need LLM inference on a laptop without internet, a field device, or an air-gapped network.

Why Ollama fits: Once models are downloaded, Ollama requires zero network connectivity. The server starts in under a second, models load in 2-10 seconds depending on size, and inference runs entirely on local hardware. The small footprint (the Ollama binary is ~50 MB) makes it suitable for edge devices. Real-world example: a field research team runs llama3.2:1b on Raspberry Pi 5s for real-time language translation in remote areas with no internet connectivity.

Cheat Sheet

Aspect	Detail
Repository	github.com/ollama/ollama
License	MIT
Language	Go (~60,000 lines) + C/C++ (llama.cpp)
GPU Requirements	None (CPU fallback); NVIDIA CUDA 5.0+, AMD ROCm, Apple Metal, Vulkan
Setup Time	2 minutes (brew install or curl pipe)
Key Features	100K+ models, OpenAI/Anthropic API, auto GPU detection, Modelfile, ollama launch, KV cache quantization, flash attention, parallel requests, MLX backend
Common Gotchas	Default context too small for long documents; OLLAMA_NUM_PARALLEL defaults to 1; models unload after 5 min idle; no built-in auth; VRAM OOM with too many parallel slots
Best Models	llama3.2:8b (general), qwen2.5-coder:14b (coding), gemma3:12b (reasoning), deepseek-r1:32b (math), gpt-oss:20b (MoE, fast)
Cost (Light)	$0 (electricity only, 1-2 sessions/day)
Cost (Heavy)	$0 (electricity only, 24/7 inference)
Cost (Cloud)	$0.50-5.00/day equivalent in API costs saved
Missing Features	No built-in auth/RBAC, no model distillation, no continuous batching (use vLLM), no distributed inference, no built-in monitoring

Vibe Coding Projects

Project 1: Local RAG Chatbot with Document Ingestion

What it does: A retrieval-augmented generation chatbot that ingests PDFs, markdown files, and code repositories, chunks them, generates embeddings via Ollama’s nomic-embed-text model, stores them in ChromaDB, and answers questions using qwen2.5:7b. Runs entirely on a laptop with 16 GB RAM.

What you’ll learn: How to use Ollama’s embedding API for RAG pipelines. How to manage context windows across multiple documents. How to tune chunk size and overlap for retrieval quality. How to use OLLAMA_NUM_PARALLEL for concurrent embedding generation.

Effort: 3-4 hours. Zero API costs.

Project 2: Multi-Model Evaluation Harness

What it does: A Python evaluation framework that runs the same prompt through 5 different Ollama models (llama3.2:8b, gemma3:12b, qwen2.5:7b, mistral-nemo:12b, deepseek-r1:32b), collects responses, and scores them on correctness, latency, and token efficiency. Outputs a comparison dashboard with latency histograms and quality scores.

What you’ll learn: How to use Ollama’s parallel request capabilities for batch evaluation. How to compare model quality and speed systematically. How to use the seed parameter for reproducible outputs. How to manage multiple models in VRAM with OLLAMA_MAX_LOADED_MODELS.

Effort: 2-3 hours. Zero API costs.

Project 3: Local AI Code Review Bot

What it does: A GitHub webhook receiver that listens for pull request events, fetches the diff, sends it to qwen2.5-coder:14b via Ollama’s API, and posts the review as a PR comment. Runs on a $10/month VPS with no GPU (CPU-only inference with 3B model) or on a self-hosted machine with GPU for larger models.

What you’ll learn: How to integrate Ollama with external services via its HTTP API. How to handle streaming responses for long code reviews. How to tune prompts for structured output (JSON mode). How to manage request timeouts for long-running inference.

Effort: 4-5 hours. Zero API costs.

Problems Solved Efficiently

Problem Type	Why Ollama Fits	When to Look Elsewhere
Local LLM inference	One-command setup, auto GPU detection, 100K+ models	Use vLLM for high-throughput production serving
AI coding tool backend	OpenAI-compatible API, ollama launch for Claude Code	Use cloud APIs for frontier model quality
Batch document processing	Zero marginal cost, parallel requests, embedding API	Use cloud APIs for one-off batches under 1K docs
CI/CD LLM evaluation	Reproducible (seed), zero cost, fast model loading	Use cloud APIs for evaluation against frontier models
Privacy-compliant AI	Air-gappable, MIT license, no data exfiltration	Use cloud APIs with HIPAA BAA for regulated cloud
Offline/edge inference	50 MB binary, sub-second startup, CPU fallback	Use ONNX Runtime for specialized edge hardware
Model experimentation	ollama pull for instant model switching, Modelfile for tuning	Use Hugging Face Transformers for fine-tuning
Multi-model comparison	Parallel loading, shared blob store, unified API	Use LangChain for complex multi-model workflows

Architectural Tradeoffs

What we gained:

• Zero-config GPU acceleration. Ollama auto-detects CUDA, ROCm, Metal, and Vulkan backends. No CMake flags, no driver version matching, no --gpu-layers guessing. The first ollama run on a machine with an NVIDIA GPU just works.
• Unified model management. ollama pull, ollama push, ollama rm, ollama cp — model management is as simple as Docker image management. The content-addressed blob store deduplicates shared weights across models.
• API compatibility as a network effect. OpenAI and Anthropic API compatibility means Ollama works with every major AI tool out of the box. No adapters, no SDKs, no integration work.
• Sensible defaults with escape hatches. Default context sizes are VRAM-tiered. Default quantization is Q4_K_M (best quality/size balance). Default temperature is 0.8. Every default can be overridden via Modelfile parameters or environment variables.
• Production-grade concurrency. Built-in parallel request handling, FIFO queuing, LRU model caching, and keep-alive management. No need to build your own queue system.
• Cross-platform portability. macOS (brew), Linux (install script), Windows (WSL2), Docker. The same ollama run command works identically on all platforms.

What we sacrificed:

• No continuous batching. Ollama processes requests in parallel slots, not continuous batches. vLLM’s continuous batching can achieve 2-3x higher throughput under high concurrency. For production serving at 50+ concurrent users, vLLM is the better choice.
• No built-in authentication or RBAC. Ollama’s HTTP server has no auth layer. Anyone who can reach port 11434 can run models. For production deployments, you need a reverse proxy (Nginx, Caddy) with auth middleware.
• No distributed inference. Ollama runs on a single machine. You cannot split a model across multiple GPUs on different machines. For models that don’t fit on one GPU, you need llama.cpp’s RPC backend or a distributed solution like Petals.
• No model distillation or fine-tuning. Ollama is an inference server, not a training framework. You cannot fine-tune models through Ollama. Use Unsloth, Axolotl, or Hugging Face Transformers for training, then import the result as a GGUF file.
• No built-in monitoring. Ollama exposes no metrics endpoint, no Prometheus integration, no structured logging. You need to wrap it with your own observability stack.
• VRAM management is manual. Ollama will happily OOM if you set OLLAMA_NUM_PARALLEL too high for your GPU. There is no automatic VRAM budgeting or graceful degradation.

The real lesson: Ollama is the best tool for local LLM development, experimentation, and low-to-medium concurrency serving. It trades raw throughput and enterprise features for developer experience and simplicity. For production serving at scale, pair Ollama with vLLM or use cloud APIs. For everything else — development, batch processing, CI/CD, privacy-sensitive workloads — Ollama is the right choice.

Course-Style Deep Dive

How Ollama Works Under the Hood

Ollama’s architecture is a Go HTTP server that wraps llama.cpp’s C++ inference engine through CGo bindings. Here is the full request lifecycle:

1. Request arrives at the HTTP server. The server parses the request, resolves the model name, and checks if the model is already loaded in the LRU cache.

2. Model loading. If the model is not loaded, the loader reads the manifest from ~/.ollama/models/manifests/registry.ollama.ai/library/<model>/<tag>. The manifest contains the blob digest, prompt template, parameters, and model architecture. The loader opens the GGUF file from ~/.ollama/models/blobs/ and passes it to llama.cpp’s llama_load_model_from_file().

3. Context allocation. llama.cpp allocates a context with the specified context size (num_ctx). The context includes the KV cache, which stores key-value pairs from previous tokens. With OLLAMA_KV_CACHE_TYPE=q8_0, the KV cache uses 8-bit quantization, halving memory usage.

4. Prompt processing (prefill). The prompt is tokenized and processed in parallel. This is the GPU-bound phase — flash attention (OLLAMA_FLASH_ATTENTION=1) accelerates this by reducing the quadratic attention computation to near-linear. On an RTX 4090, gemma3:12b processes prompts at ~85 tok/s with flash attention enabled.

5. Token generation (decode). Tokens are generated one at a time. Each step runs the transformer forward pass on the last token, reads the logits, applies temperature and top-p sampling, and returns the next token. This is memory-bandwidth-bound — the model weights must be read from VRAM for every token.

6. Response streaming. If streaming is enabled, each token is sent to the client as a server-sent event (SSE) as soon as it is generated. The Go server uses c.Stream() with a channel-based producer-consumer pattern.

7. Model unloading. After OLLAMA_KEEP_ALIVE (default 5 minutes) of inactivity, the model is unloaded from VRAM. The LRU cache evicts the least recently used model when OLLAMA_MAX_LOADED_MODELS is exceeded.

Advanced Pattern 1: Multi-Model Parallel Serving

# Serve 3 models concurrently with parallel slots
export OLLAMA_MAX_LOADED_MODELS=3
export OLLAMA_NUM_PARALLEL=4
export OLLAMA_KEEP_ALIVE=24h
ollama serve

# In separate terminals or from your app:
# Model 1: lightweight chat
curl http://localhost:11434/v1/chat/completions \
  -d '{"model": "llama3.2:8b", "messages": [{"role": "user", "content": "Hello"}]}'

# Model 2: code generation
curl http://localhost:11434/v1/chat/completions \
  -d '{"model": "qwen2.5-coder:14b", "messages": [{"role": "user", "content": "Write a binary search"}]}'

# Model 3: embeddings
curl http://localhost:11434/v1/embeddings \
  -d '{"model": "nomic-embed-text", "input": "Ollama is great"}'

Each model gets its own parallel slots. With OLLAMA_NUM_PARALLEL=4 and OLLAMA_MAX_LOADED_MODELS=3, you can handle up to 12 concurrent requests across 3 models. VRAM usage is the limiting factor — each loaded model consumes its full weight memory plus KV cache.

Advanced Pattern 2: Load Balancing with Nginx

# /etc/nginx/sites-available/ollama
upstream ollama_backends {
    least_conn;  # LLM request times vary 50x; least_conn beats round-robin
    server 10.0.1.10:11434;
    server 10.0.1.11:11434;
    server 10.0.1.12:11434;
    server 10.0.1.13:11434;
}

server {
    listen 443 ssl;
    server_name ollama.example.com;

    location / {
        proxy_pass http://ollama_backends;
        proxy_http_version 1.1;
        proxy_buffering off;          # Required for streaming
        proxy_read_timeout 600s;      # Long generations can take minutes
        proxy_set_header Connection '';
        chunked_transfer_encoding on;
    }
}

# docker-compose.yml for multi-node Ollama
version: '3.8'
services:
  ollama-node-1:
    image: ollama/ollama:latest
    environment:
      - OLLAMA_NUM_PARALLEL=4
      - OLLAMA_KEEP_ALIVE=24h
      - OLLAMA_FLASH_ATTENTION=1
    volumes:
      - ollama-models-1:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

  nginx:
    image: nginx:alpine
    ports:
      - "11434:11434"
    volumes:
      - ./nginx.conf:/etc/nginx/conf.d/default.conf:ro
    depends_on:
      - ollama-node-1
      - ollama-node-2
      - ollama-node-3
      - ollama-node-4

volumes:
  ollama-models-1:
  ollama-models-2:
  ollama-models-3:
  ollama-models-4:

Advanced Pattern 3: Custom GGUF Model Import with Quantization

# 1. Download a GGUF model from Hugging Face
wget https://huggingface.co/bartowski/Qwen3-30B-Coder-GGUF/resolve/main/Qwen3-30B-Coder-Q4_K_M.gguf

# 2. Create a Modelfile
cat > Modelfile << 'EOF'
FROM ./Qwen3-30B-Coder-Q4_K_M.gguf

PARAMETER temperature 0.2
PARAMETER top_p 0.95
PARAMETER num_ctx 32768
PARAMETER num_predict 4096

TEMPLATE "{{ .Prompt }}"

SYSTEM "You are an expert software engineer. Write clean, well-documented code."
EOF

# 3. Import into Ollama
ollama create my-coder -f Modelfile

# 4. Verify
ollama run my-coder "Write a Python function for merge sort."

# Or quantize during import (Ollama v0.17+)
ollama create --quantize q4_K_M my-coder-optimized

Production Considerations

VRAM budgeting. Each parallel slot consumes additional VRAM for KV cache. The formula is approximately:

VRAM per slot = (num_ctx * num_layers * kv_cache_bytes) / 1024^3

For llama3.2:8b with 32K context and Q8_0 KV cache (1 byte per key/value):

• 32,768 tokens * 32 layers * 2 (K+V) * 1 byte = ~2 GB per slot
• With OLLAMA_NUM_PARALLEL=4: 8 GB for KV cache + 4.5 GB for weights = 12.5 GB total
• An RTX 4090 (24 GB) can handle this comfortably. An RTX 3070 (8 GB) cannot.

Health checks. Ollama can hang while accepting TCP connections. Use HTTP health checks, not TCP checks:

# Nginx health check
upstream ollama_backends {
    least_conn;
    server 10.0.1.10:11434 max_fails=3 fail_timeout=30s;
    server 10.0.1.11:11434 max_fails=3 fail_timeout=30s;
}

# HAProxy health check
backend ollama_nodes
    option httpchk GET /api/tags
    http-check expect status 200
    server node1 10.0.1.10:11434 check inter 5s fall 3 rise 2
    server node2 10.0.1.11:11434 check inter 5s fall 3 rise 2

Queue depth monitoring. Queue depth is the cardinal metric for Ollama performance. If the queue exceeds 2 * OLLAMA_NUM_PARALLEL, p95 latency doubles:

# Monitor queue depth
while true; do
  curl -s http://localhost:11434/api/ps | jq '.models[0].queue_depth // 0'
  sleep 1
done

Cold start mitigation. Models unload after OLLAMA_KEEP_ALIVE (default 5 minutes). For production, set OLLAMA_KEEP_ALIVE=24h and implement a warm-up script:

# Warm-up script — run on a cron or after deploy
curl -X POST http://localhost:11434/api/generate \
  -d '{"model": "llama3.2:8b", "prompt": "Hello", "keep_alive": "24h"}'

The Results

Metric	Before Ollama (DIY llama.cpp)	After Ollama	Improvement
First-time setup time	3-4 hours (build + config)	2 minutes (brew install)	90-120x faster
Model download + run	15 minutes (find GGUF, download, configure)	30 seconds (ollama pull + run)	30x faster
GPU detection	Manual (CUDA version, CMake flags)	Automatic (zero config)	Eliminated entirely
API setup	2-3 hours (build server wrapper)	0 minutes (built-in)	Eliminated entirely
Model switching	5 minutes (download, configure, test)	10 seconds (ollama pull)	30x faster
Custom model creation	30 minutes (Modelfile research)	2 minutes (Modelfile + ollama create)	15x faster
Concurrent request support	Manual (build queue system)	Built-in (OLLAMA_NUM_PARALLEL)	Eliminated entirely
Tool integration	Custom adapter per tool	Zero config (OpenAI API)	Eliminated entirely
Prompt processing (gemma3:12b, RTX 4090)	52 tok/s (manual llama.cpp)	85 tok/s (Ollama + flash attention)	1.6x faster
Token generation (gemma3:12b, RTX 4090)	28 tok/s (manual llama.cpp)	32 tok/s (Ollama default)	1.14x faster

What this means for you: Ollama does not make local LLMs faster than raw llama.cpp — the inference engine is the same. What Ollama eliminates is the 3-4 hours of setup, configuration, and integration work that stood between you and running a local model. The 90-120x improvement in setup time is the real metric. For a team of 10 developers, that is 30-40 hours of saved setup time per person — or 300-400 hours total — plus the ongoing savings from instant model switching, zero-config API integration, and built-in concurrency.

What to Watch Out For

1. Default context size is too small for documents. Ollama’s default num_ctx is 2,048 tokens for most models. For document analysis, RAG, or long conversations, you need 8K-32K. Set it in a Modelfile: PARAMETER num_ctx 32768.

2. Models unload after 5 minutes of inactivity. The default OLLAMA_KEEP_ALIVE is 5 minutes. If you send requests sporadically, every request pays a 2-10 second cold-start penalty. Set OLLAMA_KEEP_ALIVE=24h for production.

3. OLLAMA_NUM_PARALLEL defaults to 1. Ollama processes one request at a time by default. For batch processing or multi-user scenarios, set OLLAMA_NUM_PARALLEL=4 or higher — but watch your VRAM budget.

4. No authentication. Ollama’s HTTP server has no auth. Anyone who can reach port 11434 can run models on your GPU. Use a reverse proxy with auth middleware for any network-exposed deployment.

5. VRAM OOM is silent. If you set OLLAMA_NUM_PARALLEL too high, Ollama will crash with an OOM error. There is no graceful degradation. Start with conservative values and monitor VRAM usage with nvidia-smi.

6. CPU inference is slow for large models. Ollama falls back to CPU if no GPU is detected. A 7B model on CPU runs at 2-5 tok/s — usable for chat, painful for batch processing. A 70B model on CPU runs at 0.1-0.5 tok/s — essentially unusable.

7. Model naming matters for API compatibility. Tools expecting gpt-3.5-turbo or gpt-4 as the model name will fail with Ollama’s model names. Use ollama cp llama3.2:8b gpt-3.5-turbo to create an alias.

Lesson 1: “I spent 4 hours debugging why Ollama was running on CPU. Turns out my NVIDIA driver was too old for the CUDA version Ollama compiled against. ollama run didn’t tell me — it just silently fell back to CPU. Run ollama ps to check if GPU is actually being used.” — Ollama user, r/LocalLLaMA

Lesson 2: “The default context size is a trap. I was getting terrible RAG results because the model could only see 2K tokens. The document was 10K tokens. The model was answering based on the first 2K tokens only. Set num_ctx explicitly in a Modelfile — never rely on defaults for production.” — RAG pipeline engineer

Lesson 3: “Ollama is not a production inference server. It is a development tool that can be pressed into production service for low-traffic scenarios. For anything above 10 concurrent users, put a load balancer in front, set up health checks, and monitor queue depth. And for God’s sake, add auth.” — Production engineer, Ollama Discord

Advice for Getting Started

1. Install Ollama and run your first model before doing anything else. brew install ollama && ollama run llama3.2:8b. If that works, everything else is configuration.

2. Check that GPU acceleration is working: ollama run llama3.2:8b "Hello" should show GPU usage in nvidia-smi or ollama ps. If it shows 0% GPU, check your drivers.

3. Start with a model that fits comfortably in your VRAM. A 7B model at Q4_K_M needs ~4.5 GB. An 8 GB GPU can run it with room for 8K context. A 12 GB GPU can run a 12B model. A 24 GB GPU can run a 30B model.

4. Use the OpenAI-compatible API from day one. Even if you’re just testing, use curl or the Python openai library against http://localhost:11434/v1/. This ensures your integration code works with any OpenAI-compatible provider.

5. Create a Modelfile for your production models. Set num_ctx, temperature, and system explicitly. Never rely on defaults for anything you deploy.

6. Set OLLAMA_KEEP_ALIVE=24h and OLLAMA_NUM_PARALLEL=4 for any serious use. The defaults are designed for casual experimentation, not production.

7. Use ollama ps to monitor loaded models and memory usage. Use ollama rm to free space. Use ollama show --modelfile <model> to inspect a model’s configuration.

---

Next in the Open-Source AI Tools Mastery series: llama.cpp