·15 min read

Milvus: The most scalable open-source vector database (Apache 2.0, 32k stars)

The most scalable open-source vector database managing trillion-scale vector datasets with GPU-accelerated indexing and hybrid search.

#milvus#vector-database#scalable#gpu-indexing#ai-data

The Problem

Every vector database on the market makes the same implicit promise: that you can throw vectors at it and get fast similarity search. The reality is that most vector databases hit a wall somewhere between 100 million and 1 billion vectors. Latency spikes, recall drops, and the operational cost curve goes vertical.

The problem is architectural. Most vector databases were designed for a single-node deployment with memory-mapped files. They scale vertically until the RAM runs out, then they scale horizontally by sharding — but sharding introduces cross-node coordination overhead, rebalancing costs, and query fan-out that degrades p99 latency by 3-5x.

For teams building at scale — recommendation engines serving billions of users, RAG pipelines indexing the entire internet, multimodal search over video and image embeddings — the existing options fall into a painful gap: too slow, too expensive, or too operationally complex.

Dimension Single-Node Vector DBs (Chroma, pgvector) Mid-Scale DBs (Qdrant, Weaviate) Milvus
Max vector capacity 10-50M 100M-1B 10B+
GPU-accelerated indexing No No Yes (CAGRA, IVF_PQ)
Index build time (100M, 768-dim) 2-4 hours 30-90 minutes 4 minutes (GPU)
Query throughput (100M, 100 QPS target) 1,200 QPS (degrades at scale) 8,000 QPS 20,000+ QPS
p99 latency at 500M vectors N/A (crashes) 47-89 ms 62 ms (GPU node)
Hybrid search (vector + scalar) Limited Good Good
Multi-tenancy None Basic Enterprise (millions of tables)
Hot/cold tiering No No Yes (StorageV2, Parquet)
Operational complexity Low (single binary) Low-Medium High (Kubernetes)
License Varies Apache 2.0 / BSD-3 Apache 2.0

Why this matters: The vector database market has bifurcated. Below 100M vectors, you have excellent options — Qdrant, Weaviate, pgvector — that are simple to operate and fast. Above 1B vectors, the field narrows to exactly one open-source option: Milvus. If your dataset will grow past 100M vectors, the cost of migrating databases later is higher than the cost of starting with Milvus now. This is not a tool for every team — it is the tool for teams that have outgrown every other option.

The Investigation

The Milvus project started at Zilliz in 2019 as a research project on GPU-accelerated similarity search. The founding insight was that CPU-based indexing could not keep pace with the growth of vector data — and that the GPU parallelism that powered deep learning training could be repurposed for inference-time vector search.

Finding 1: The CPU indexing wall is real and getting worse.

Vector indexes are compute-bound, not memory-bound. Building an HNSW or IVF index requires millions of distance calculations, each involving floating-point operations on high-dimensional vectors. On CPU, these calculations are serialized across cores with limited SIMD utilization. On GPU, they run in parallel across thousands of CUDA cores.

Milvus’s investigation found that GPU-accelerated index building is 30-50x faster than CPU for the same index quality. The NVIDIA cuVS library (formerly RAFT) can build a CAGRA index on 106M 2048-dim vectors in ~4 minutes — a task that takes 2+ hours on a 32-core CPU node.

What this means: If you are building indexes on CPU at scale, you are spending 30-50x more time and energy than necessary. The GPU is not a luxury — it is the economically rational choice for any dataset over 10M vectors that requires frequent index rebuilds.

Finding 2: Disaggregated storage is the only path to trillion-scale.

Milvus’s architecture separates compute (query nodes, index nodes) from storage (object store, WAL, metadata). This is the same pattern that made Snowflake and BigQuery successful for structured data. The insight is that vector search workloads have asymmetric scaling: you may need 100 query nodes during peak traffic and 10 at night, but your storage footprint is constant.

With disaggregated storage, scaling is linear and independent. Add query nodes for throughput. Add index nodes for build speed. Storage scales independently via S3/MinIO. No data migration, no rebalancing, no downtime.

What this means: Every vector database that couples compute and storage (Chroma, pgvector, single-node Qdrant) has a hard scaling ceiling. Milvus’s ceiling is determined by your object store budget — which is effectively unlimited.

Finding 3: Hybrid search requires a unified storage engine.

Early vector databases treated scalar filtering as an afterthought — a post-filter applied after the vector search returned candidates. This works at small scale but breaks at large scale: a post-filter on a 1M-result set can take longer than the vector search itself.

Milvus’s investigation found that pre-filtering (applying scalar filters before vector search) requires bitmap indexing on the scalar side and tight integration with the vector index traversal. This is architecturally complex but delivers 10-100x better filtered search performance at scale.

What this means: If your application requires filtered vector search (and most production applications do — “find similar products in the Electronics category priced under $50”), the database must support pre-filtering natively. Post-filtering is not a viable strategy above 10M vectors.

The Solution

Milvus is a cloud-native vector database (Apache 2.0 license, 32,000+ GitHub stars) built on a fully disaggregated architecture. It separates storage, compute, and coordination into independent layers that scale horizontally.

┌──────────────────────────────────────────────────────────────────────────┐
│                          Milvus Architecture                               │
│                                                                           │
│  ┌────────────────────────────────────────────────────────────────────┐  │
│  │                    Access Layer (Proxies)                           │  │
│  │  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐           │  │
│  │  │ Proxy 1  │  │ Proxy 2  │  │ Proxy 3  │  │ Proxy N  │           │  │
│  │  └────┬─────┘  └────┬─────┘  └────┬─────┘  └────┬─────┘           │  │
│  │       │             │             │             │                  │  │
│  │  Load Balancer (Nginx / K8s Ingress)                               │  │
│  └────────────────────────────────────────────────────────────────────┘  │
│                                    │                                      │
│  ┌────────────────────────────────────────────────────────────────────┐  │
│  │                   Coordinator Layer (The Brain)                     │  │
│  │  ┌──────────────┐  ┌──────────────┐  ┌────────────────────────┐  │  │
│  │  │ RootCoord    │  │ QueryCoord   │  │ DataCoord             │  │  │
│  │  │ (DDL, TSO,   │  │ (query       │  │ (index building,       │  │  │
│  │  │  topology)   │  │  scheduling) │  │  compaction, GC)       │  │  │
│  │  └──────────────┘  └──────────────┘  └────────────────────────┘  │  │
│  └────────────────────────────────────────────────────────────────────┘  │
│                                    │                                      │
│  ┌────────────────────────────────────────────────────────────────────┐  │
│  │                    Worker Nodes (Stateless)                        │  │
│  │  ┌──────────────┐  ┌──────────────┐  ┌────────────────────────┐  │  │
│  │  │ Query Nodes  │  │ Index Nodes  │  │ Data Nodes             │  │  │
│  │  │ (search,     │  │ (GPU/CPU     │  │ (compaction, WAL       │  │  │
│  │  │  load data)  │  │  index build)│  │  flush, log backup)    │  │  │
│  │  └──────────────┘  └──────────────┘  └────────────────────────┘  │  │
│  │  ┌──────────────┐                                                │  │
│  │  │ Stream Nodes │  (real-time writes, WAL, growing data)         │  │
│  │  └──────────────┘                                                │  │
│  └────────────────────────────────────────────────────────────────────┘  │
│                                    │                                      │
│  ┌────────────────────────────────────────────────────────────────────┐  │
│  │                      Storage Layer                                 │  │
│  │  ┌──────────────┐  ┌──────────────┐  ┌────────────────────────┐  │  │
│  │  │ Meta Store   │  │ Object Store │  │ WAL / Log Broker       │  │  │
│  │  │ (etcd)       │  │ (MinIO/S3/   │  │ (Kafka / Pulsar /      │  │  │
│  │  │              │  │  Azure Blob) │  │  Woodpecker)           │  │  │
│  │  └──────────────┘  └──────────────┘  └────────────────────────┘  │  │
│  └────────────────────────────────────────────────────────────────────┘  │
└──────────────────────────────────────────────────────────────────────────┘

Here is what each layer does:

  • Access Layer (Proxies): Stateless front-end endpoints that handle request routing, authentication, rate limiting, and result aggregation. Proxies implement MPP (Massively Parallel Processing) — they fan out queries to multiple query nodes, collect partial results, and merge them before returning to the client. Proxies are stateless and scale horizontally behind a load balancer.

  • Coordinator Layer: The cluster’s brain. RootCoord manages DDL operations (create/drop collections, partitions, indexes), time-stamp ordering (TSO), and cluster topology. QueryCoord schedules queries across query nodes and manages segment distribution. DataCoord manages index building, data compaction, garbage collection, and segment lifecycle. Exactly one active coordinator per cluster, with standby replicas for HA.

  • Worker Nodes: Stateless executors that do the actual work. Query Nodes load vector segments from object storage and execute search requests. Index Nodes build vector indexes (GPU or CPU) and write them back to object storage. Data Nodes handle WAL log consumption, data compaction, and flushing sealed segments to object storage. Stream Nodes handle real-time writes and growing data queries — they are the “mini-brain” at the shard level.

  • Storage Layer: Fully disaggregated. Meta Store (etcd) holds cluster metadata, schema definitions, and service registry. Object Store (MinIO, S3, Azure Blob, or GCS) stores log snapshots, index files, and intermediate query results. WAL/Log Broker (Kafka, Pulsar, or the new Woodpecker engine) provides a zero-disk, cloud-native write-ahead log for durability and consistency.

Setup

# Option 1: Milvus Standalone (development / <10M vectors)
# Single binary, no Kubernetes required
curl -sfL https://raw.githubusercontent.com/milvus-io/milvus/master/scripts/standalone_embed.sh | bash

# Verify it's running
curl http://localhost:19530/metrics

# Option 2: Milvus Cluster on Kubernetes (production)
# Requires Helm 3.12+
helm repo add milvus https://milvus-io.github.io/milvus-helm/
helm repo update
helm install my-milvus milvus/milvus \
  --set cluster.enabled=true \
  --set persistence.enabled=true \
  --set minio.mode=standalone \
  --namespace milvus --create-namespace

# Option 3: Milvus Lite (embedded, for edge/dev)
pip install pymilvus milvus-lite

Production-Grade Configuration

# values.yaml for Helm deployment
cluster:
  enabled: true

# Proxy layer — stateless, scale for throughput
proxy:
  replicas: 3
  resources:
    requests:
      memory: "4Gi"
      cpu: "2"

# Query nodes — memory-bound, scale for data volume
queryNode:
  replicas: 5
  resources:
    requests:
      memory: "32Gi"
      cpu: "8"
  limits:
    memory: "64Gi"

# Index nodes — compute-bound, GPU recommended
indexNode:
  replicas: 2
  resources:
    requests:
      memory: "16Gi"
      cpu: "4"
      nvidia.com/gpu: 1
  limits:
    memory: "32Gi"
    nvidia.com/gpu: 1

# Data nodes — I/O bound, moderate resources
dataNode:
  replicas: 3
  resources:
    requests:
      memory: "8Gi"
      cpu: "4"

# Storage
minio:
  mode: distributed
  replicas: 4
  persistence:
    size: 500Gi

etcd:
  replicaCount: 3
  persistence:
    size: 10Gi

# WAL broker
pulsar:
  enabled: true
  bookkeeper:
    replicaCount: 3

Code Walkthrough: Connecting and Searching

# pymilvus — the official Python SDK
from pymilvus import (
    connections, Collection, FieldSchema, CollectionSchema,
    DataType, utility, AnnSearchRequest, RRFRanker
)
import numpy as np

# 1. Connect to the cluster
connections.connect(
    alias="default",
    host="localhost",
    port=19530,
)

# 2. Create a collection with schema
collection_name = "product_embeddings"

# Drop if exists for clean demo
if utility.has_collection(collection_name):
    utility.drop_collection(collection_name)

fields = [
    FieldSchema(name="id", dtype=DataType.INT64, is_primary=True, auto_id=True),
    FieldSchema(name="product_id", dtype=DataType.INT64),
    FieldSchema(name="category", dtype=DataType.VARCHAR, max_length=64),
    FieldSchema(name="price", dtype=DataType.FLOAT),
    FieldSchema(name="in_stock", dtype=DataType.BOOL),
    FieldSchema(name="embedding", dtype=DataType.FLOAT_VECTOR, dim=768),
]

schema = CollectionSchema(
    fields=fields,
    description="Product embeddings for semantic search",
    enable_dynamic_field=True,
)

collection = Collection(name=collection_name, schema=schema)

# 3. Create indexes
# Vector index — GPU_CAGRA for maximum throughput
collection.create_index(
    field_name="embedding",
    index_params={
        "metric_type": "IP",  # Inner Product (use IP, not COSINE, for GPU)
        "index_type": "GPU_CAGRA",
        "params": {
            "intermediate_graph_degree": 128,
            "graph_degree": 64,
            "build_algo": "NN_DESCENT",
        },
    },
)

# Scalar index for filtered search
collection.create_index(
    field_name="category",
    index_params={"index_type": "INVERTED"},
)

collection.create_index(
    field_name="price",
    index_params={"index_type": "INVERTED"},
)

# 4. Insert data
import random

vectors = np.random.random((10000, 768)).astype(np.float32)
data = [
    [i for i in range(10000)],           # product_id
    [random.choice(["electronics", "clothing", "home", "sports"]) for _ in range(10000)],
    [random.uniform(5.0, 500.0) for _ in range(10000)],
    [random.choice([True, False]) for _ in range(10000)],
    vectors,
]

collection.insert(data)
collection.flush()

# 5. Load collection into memory (required before search)
collection.load()

# 6. Basic vector search
search_params = {
    "metric_type": "IP",
    "params": {"nprobe": 64},
}

query_vector = np.random.random((1, 768)).astype(np.float32)

results = collection.search(
    data=query_vector,
    anns_field="embedding",
    param=search_params,
    limit=10,
    output_fields=["product_id", "category", "price"],
)

for result in results[0]:
    print(f"ID: {result.id}, Score: {result.score:.4f}, "
          f"Product: {result.entity.get('product_id')}, "
          f"Category: {result.entity.get('category')}")

# 7. Hybrid search with scalar filtering
results = collection.search(
    data=query_vector,
    anns_field="embedding",
    param=search_params,
    limit=10,
    expr="category == 'electronics' and price < 100.0 and in_stock == true",
    output_fields=["product_id", "category", "price"],
)

# 8. Multi-vector hybrid search (Milvus 2.6+)
# Combine results from multiple vector fields with RRF
query_vector_2 = np.random.random((1, 768)).astype(np.float32)

req_1 = AnnSearchRequest(
    data=query_vector,
    anns_field="embedding",
    param=search_params,
    limit=100,
)

req_2 = AnnSearchRequest(
    data=query_vector_2,
    anns_field="embedding",
    param=search_params,
    limit=100,
)

hybrid_results = collection.hybrid_search(
    reqs=[req_1, req_2],
    rerank=RRFRanker(k=60),
    limit=10,
    output_fields=["product_id", "category"],
)

How to Use Effectively

Step 1: Choose the right index type

Milvus supports 10+ index types. The wrong choice costs 10x in latency or memory.

# GPU_CAGRA — best for high-throughput production (GPU required)
index_params = {
    "index_type": "GPU_CAGRA",
    "metric_type": "IP",
    "params": {
        "intermediate_graph_degree": 128,
        "graph_degree": 64,
        "build_algo": "NN_DESCENT",
    },
}

# IVF_SQ8 — best for memory-constrained CPU deployments
index_params = {
    "index_type": "IVF_SQ8",
    "metric_type": "L2",
    "params": {"nlist": 4096},
}

# HNSW — best for low-latency CPU search (<10ms p99)
index_params = {
    "index_type": "HNSW",
    "metric_type": "L2",
    "params": {"M": 16, "efConstruction": 500},
}

# DiskANN — best for billion-scale on limited RAM
index_params = {
    "index_type": "DISKANN",
    "metric_type": "L2",
}

Index selection guide:

Index Type Memory Build Speed Search Speed Recall Best For
GPU_CAGRA 1.8x data Fastest (GPU) Fastest High Production, GPU available
IVF_FLAT 1.0x data Fast Moderate High Balanced, CPU
IVF_SQ8 0.3x data Fast Moderate Medium Memory-constrained
IVF_PQ 0.1x data Moderate Fast Low-Medium Extreme compression
HNSW 1.5x data Slow Fastest (CPU) High Low-latency CPU
DISKANN 0.1x data Slow Moderate High Billion-scale, limited RAM

Step 2: Configure GPU memory properly

# Set GPU memory pool in Milvus config
# config/milvus.yaml
gpu:
  initMemSize: 4096   # MB — initial GPU memory allocation
  maxMemSize: 16384   # MB — maximum GPU memory pool
  enable: true
  cache_capacity: 8   # GB — GPU cache for index data

Production pitfall: GPU_CAGRA does not support COSINE distance. Use Inner Product (IP) with normalized vectors instead. Normalize your vectors before insertion: vec = vec / np.linalg.norm(vec). IP on normalized vectors is equivalent to cosine similarity.

Step 3: Use partitions for logical data isolation

# Create partitions for logical grouping
collection.create_partition(partition_name="electronics")
collection.create_partition(partition_name="clothing")
collection.create_partition(partition_name="home")

# Insert into specific partition
collection.insert(data, partition_name="electronics")

# Search within a partition (faster than filtering)
collection.search(
    data=query_vector,
    anns_field="embedding",
    param=search_params,
    limit=10,
    partition_names=["electronics"],
)

Partitions are not the same as shards. Partitions are logical groupings within a collection. Shards are physical data divisions. Use partitions for tenant isolation or category-level segmentation. Use shards for horizontal scaling.

Step 4: Monitor with Prometheus and Grafana

Milvus exposes 200+ metrics. The critical ones:

# Prometheus scrape config
scrape_configs:
  - job_name: 'milvus'
    static_configs:
      - targets:
        - 'proxy:9091'      # Proxy metrics
        - 'querynode:9091'   # Query node metrics
        - 'indexnode:9091'   # Index node metrics
        - 'datanode:9091'    # Data node metrics

Key metrics to watch:

Metric Warning Critical Action
milvus_querynode_search_latency_p99 >50ms >100ms Add query nodes
milvus_querynode_memory_usage_bytes >80% >90% Scale up or add nodes
milvus_indexnode_gpu_utilization <30% <10% Check index build queue
milvus_datanode_compaction_queue_length >100 >500 Add data nodes
milvus_proxy_request_rate >80% capacity >95% capacity Add proxies

Step 5: Tune nprobe and nlist for your workload

# nlist controls index granularity (set at build time)
# Higher nlist = better recall, slower build, more memory
index_params = {
    "index_type": "IVF_FLAT",
    "params": {"nlist": 4096},  # Default is 1024
}

# nprobe controls search quality (set at query time)
# Higher nprobe = better recall, slower search
search_params = {
    "params": {"nprobe": 64},  # Start here, tune up/down
}

A good starting point: nlist = 4 * sqrt(N) where N is the number of vectors. For 100M vectors, nlist = 4 * 10000 = 40000. For nprobe, start at nlist / 64 and adjust based on your recall target.

Use Cases

1. Enterprise RAG at Scale

When you’d use this: Your organization has millions of internal documents (technical docs, policy manuals, support tickets) and you need a RAG pipeline that indexes everything and returns relevant chunks in under 100ms.

Why Milvus fits: Milvus is the only open-source vector database that handles 1B+ document chunks with consistent sub-100ms latency. The GPU-accelerated index building means you can rebuild indexes nightly without a maintenance window. The hybrid search (vector + scalar filtering) lets you filter by document type, department, date range, and access level in a single query. Real-world deployments at companies like eBay, Walmart, and NVIDIA use Milvus for RAG at this scale.

2. Recommendation Engines

When you’d use this: You need to serve personalized recommendations to millions of users, using user behavior embeddings and product embeddings, with sub-50ms latency at 10,000+ QPS.

Why Milvus fits: Milvus’s MPP architecture fans out queries across query nodes and merges results in the proxy layer, delivering linear QPS scaling. The GPU_CAGRA index delivers the highest throughput of any open-source vector index — up to 100x higher than CPU HNSW for batch queries. Multi-vector hybrid search (Milvus 2.6+) lets you combine user embedding, session embedding, and context embedding into a single ranked result.

3. Multimodal Search (Image + Text + Video)

When you’d use this: You have a catalog of millions of products with images, descriptions, and video previews, and you want users to search across all modalities.

Why Milvus fits: Milvus supports multiple vector fields per collection (Milvus 2.6+ unified data model). You can store image embeddings, text embeddings, and video embeddings in the same collection and run hybrid search across all three with Reciprocal Rank Fusion (RRF) or weighted scoring. The BLOB data type (v3.0) lets you store thumbnails and previews directly in the database.

When you’d use this: You need to find transactions, login attempts, or network events that are similar to known fraud patterns, across billions of historical records.

Why Milvus fits: Fraud detection requires both speed (real-time scoring) and scale (billions of historical events). Milvus’s DiskANN index lets you search billion-scale datasets from disk, keeping only the index in memory. The scalar filtering (expr parameter) lets you narrow searches by time window, transaction type, geographic region, and risk score simultaneously. The multi-tenancy support (millions of tables per cluster) lets you isolate tenants without separate infrastructure.

5. Drug Discovery and Molecular Similarity

When you’d use this: You need to search a library of billions of molecular fingerprints to find compounds similar to a target molecule.

Why Milvus fits: Molecular fingerprints are typically binary vectors (1024-2048 bits) that map naturally to Milvus’s BINARY_VECTOR type. The GPU_IVF_PQ index with product quantization compresses binary vectors to 10% of their original size while maintaining 95%+ recall. Milvus’s billion-scale capacity means you can index the entire PubChem database (~110M compounds) or ChEMBL (~20M compounds) on a single cluster. The GPU-accelerated Tanimoto similarity search (via custom distance functions in v3.0) is 50-100x faster than CPU-based molecular search tools.

Cheat Sheet

Aspect Detail
Repository github.com/milvus-io/milvus
License Apache 2.0
Language Go (core), C++ (indexing), Python (SDK)
GPU Requirements Optional (NVIDIA CUDA, cuVS); CPU-only mode available
Setup Time 5 min (Milvus Lite), 30 min (Standalone), 2-3 days (Cluster on K8s)
Key Features GPU-accelerated indexing, disaggregated storage, hybrid search, multi-tenancy, DiskANN, hot/cold tiering, MPP query execution
Common Gotchas GPU_CAGRA does not support COSINE distance; COSINE requires normalized IP; index rebuild spikes CPU 300%; scalar filters without indexes double latency; nprobe too low = poor recall
Best Index Types GPU_CAGRA (GPU, throughput), HNSW (CPU, latency), IVF_SQ8 (CPU, memory), DiskANN (billion-scale, disk)
Cost (Self-Host, 50M vec) ~$1,080/month (EC2, 3 proxy + 5 query + 2 index + 3 data nodes)
Cost (Zilliz Cloud) ~$2,400/month (managed, 50M vectors, 100 QPS)
Cost (Milvus Lite) $0 (embedded, dev/edge only)
Missing Features No built-in vectorization, no GraphQL API, no native BM25 (hybrid search via reranking), no built-in RBAC (v3.0 adds audit logging)

Vibe Coding Projects

Project 1: Semantic Product Search for E-Commerce

What it does: A FastAPI application that ingests product data (name, description, category, price), generates embeddings via a local sentence-transformer model, stores them in Milvus, and exposes a REST API for semantic search with category and price filters. Includes a simple React frontend for demo purposes.

What you’ll learn: How to set up Milvus Standalone, create collections with proper schema design, build GPU_CAGRA indexes, implement hybrid search with scalar filtering, and integrate with a FastAPI backend. You’ll also learn the embedding pipeline — generating, normalizing, and inserting vectors at scale.

Effort: 4-6 hours. ~$20-30 in cloud compute (or free on a local GPU machine).

Project 2: Multi-Tenant RAG Chatbot with Document Ingestion

What it does: A RAG chatbot that supports multiple tenants (organizations), each with their own document collection. Ingests PDFs, splits them into chunks, generates embeddings, and stores them in Milvus with tenant-level partitioning. The chatbot retrieves relevant chunks and generates answers via an LLM, filtered by tenant access controls.

What you’ll learn: Milvus multi-tenancy patterns (partition-based vs. collection-based), document chunking strategies, embedding generation pipelines, and hybrid search for combining semantic similarity with metadata filtering. You’ll also learn how to manage index rebuilds in a multi-tenant environment.

Effort: 8-12 hours. ~$50-80 in cloud compute.

Project 3: Real-Time Anomaly Detection Dashboard

What it does: A streaming application that ingests time-series data (server metrics, transaction logs, or network events), generates embeddings via an autoencoder model, stores them in Milvus, and runs continuous similarity search to flag anomalies. A Grafana dashboard visualizes anomaly scores over time.

What you’ll learn: Real-time data ingestion with Milvus Stream Nodes, time-windowed search patterns, DiskANN index configuration for billion-scale historical data, and GPU-accelerated index rebuilding for model retraining. You’ll also learn how to tune nprobe for the latency/recall tradeoff in real-time applications.

Effort: 10-15 hours. ~$100-150 in cloud compute.

Problems Solved Efficiently

Problem Type Why Milvus Fits When to Look Elsewhere
Billion-scale vector search Disaggregated storage, DiskANN, GPU indexing Use Qdrant for <100M vectors (simpler ops)
GPU-accelerated indexing CAGRA, IVF_PQ on NVIDIA GPUs, 30-50x faster than CPU Use CPU-only DBs if no GPU available
Hybrid search (vector + scalar) Pre-filtering with bitmap indexes, expr parameter Use Weaviate for native BM25 + vector hybrid
Multi-tenant SaaS Millions of tables per cluster, partition isolation Use Qdrant for simpler multi-tenancy at <1B
Real-time ingestion + search Stream Nodes, WAL-based consistency, growing data queries Use Pinecone for zero-ops real-time ingestion
Hot/cold tiering StorageV2, Parquet format, automatic tier migration Use single-node DBs for uniform data temperature
Enterprise compliance Apache 2.0, air-gapped, audit logging (v3.0) Use Pinecone for managed SOC 2 compliance
Drug discovery / molecular search BINARY_VECTOR, GPU Tanimoto, billion-scale Use specialized tools for small molecule libraries

Architectural Tradeoffs

What we gained:

  • Trillion-scale capacity. Milvus is the only open-source vector database proven at 10B+ vectors. The disaggregated architecture means storage is limited only by your object store budget. No other open-source option can make this claim.
  • GPU-accelerated everything. Index building is 30-50x faster on GPU. Search throughput is up to 100x higher for batch queries. The GPU memory pool caches index data for sub-millisecond access. If you have a GPU, Milvus uses it.
  • Linear horizontal scaling. Add query nodes for throughput, index nodes for build speed, data nodes for compaction throughput. Each dimension scales independently. No rebalancing, no data migration, no downtime.
  • Cloud-native storage separation. Compute scales up and down independently of storage. Run 100 query nodes during peak and 10 at night. Storage costs are decoupled from compute costs. This is the Snowflake model for vector search.
  • Enterprise multi-tenancy. Support for millions of tables in a single cluster. Partition-based tenant isolation. Tenant-level hot/cold tiering. This is the only vector database designed for SaaS-scale multi-tenancy from day one.

What we sacrificed:

  • Operational complexity. Milvus Cluster requires Kubernetes, etcd, object storage, and a message broker (Kafka/Pulsar). The Helm chart has 200+ configurable parameters. Setup takes 2-3 days for a production cluster. Qdrant runs as a single binary.
  • Overkill for small datasets. Below 100M vectors, Milvus’s complexity is not justified. A single Qdrant node or pgvector instance will serve the same workload with 10x less operational overhead. Milvus is a sledgehammer for a problem that may be a thumbtack.
  • No built-in vectorization. Milvus stores and searches vectors — it does not generate them. You need a separate embedding pipeline (sentence-transformers, OpenAI embeddings, etc.) and must manage the embedding lifecycle yourself. Weaviate and Pinecone offer built-in vectorization modules.
  • GPU dependency for peak performance. Without a GPU, Milvus is competitive but not dominant. CPU-only HNSW on Milvus is comparable to Qdrant’s HNSW — and Qdrant is simpler to operate. The GPU is where Milvus pulls ahead.
  • No native BM25. Milvus’s hybrid search uses vector + scalar filtering, not vector + keyword (BM25). For applications that need semantic + keyword search, Weaviate’s built-in BM25 integration is more natural. Milvus v3.0 adds TEXT data type and full-text search to address this.
  • Steeper learning curve. The architecture has 6+ component types, each with its own configuration, monitoring, and scaling characteristics. Understanding the full system takes weeks. Qdrant and Weaviate can be understood in a day.

The real lesson: Milvus is not competing with Qdrant, Weaviate, or Pinecone for the same workload. It is competing for workloads that those databases cannot handle. If your dataset fits in a single node, use a single-node database. If your dataset will grow past 100M vectors, the cost of migrating later is higher than the cost of starting with Milvus now. The complexity tax is real — but it is a one-time cost, while the scaling ceiling is a permanent constraint.

Course-Style Deep Dive

How GPU_CAGRA Indexing Works Under the Hood

The CAGRA (Cuda Anns for GRAph-based search) index is Milvus’s flagship GPU index, contributed by NVIDIA RAPIDS. Here is how it works:

  1. Graph Construction (Build Phase). CAGRA builds a k-NN graph where each node (vector) is connected to its nearest neighbors. The build uses the NN_DESCENT algorithm, which iteratively refines the graph: start with random edges, then for each node, check if its neighbors’ neighbors are better neighbors. This runs entirely on GPU, processing millions of distance calculations in parallel across CUDA cores.

  2. Graph Optimization. The raw k-NN graph is optimized for GPU traversal. CAGRA applies two key optimizations:

    • Intermediate graph degree (128): A high-degree intermediate graph is built during search to provide multiple starting points. This improves recall without increasing memory.
    • Graph degree (64): The final graph has a fixed out-degree of 64 edges per node. This fits in GPU shared memory for fast traversal.

  3. Search Phase. At query time, CAGRA uses a BFS-like traversal on the GPU:

    • Start from multiple seed nodes (determined by the intermediate graph)
    • Expand the frontier by evaluating all neighbors of visited nodes
    • Maintain a priority queue of the top-K candidates
    • Terminate when the frontier is exhausted or the queue stabilizes

The key insight: GPU memory bandwidth (1-2 TB/s on A100) is 10-20x higher than CPU memory bandwidth (100-200 GB/s on DDR5). CAGRA is designed to keep the graph in GPU memory and saturate that bandwidth with parallel distance computations.

# Conceptual: CAGRA search on GPU
# Pseudocode — actual implementation is in C++/CUDA
def cagra_search(query, graph, top_k, intermediate_degree=128, graph_degree=64):
    # 1. Initialize seed nodes from intermediate graph
    seeds = sample_seeds(intermediate_degree)

    # 2. Parallel BFS on GPU
    visited = set(seeds)
    candidates = MinHeap(top_k * 2)

    for seed in seeds:
        dist = cosine_distance(query, graph.nodes[seed])
        candidates.push((dist, seed))

    # 3. Iterative expansion (runs in CUDA kernel)
    while not candidates.stable:
        # Evaluate all neighbors of current frontier in parallel
        frontier = candidates.top(graph_degree)

        # GPU kernel: compute distances for all frontier neighbors
        distances = cuda_parallel_distance(query, frontier, graph)

        # Update candidates
        for node, dist in distances:
            if node not in visited:
                visited.add(node)
                candidates.push((dist, node))

    return candidates.top(top_k)

Advanced Pattern 1: Multi-Vector Hybrid Search with RRF

Milvus 2.6+ supports searching across multiple vector fields and combining results with Reciprocal Rank Fusion (RRF). This is useful for multimodal search (image + text + video embeddings) or multi-query search (same embedding, different search strategies).

from pymilvus import AnnSearchRequest, RRFRanker

# Create two search requests with different strategies
req_1 = AnnSearchRequest(
    data=query_vector,
    anns_field="embedding",
    param={"metric_type": "IP", "params": {"nprobe": 64}},
    limit=100,
)

req_2 = AnnSearchRequest(
    data=query_vector,
    anns_field="embedding",
    param={"metric_type": "IP", "params": {"nprobe": 128}},
    limit=100,
)

# RRF combines rankings: score = 1 / (k + rank)
# k=60 is the standard value (smooth rank normalization)
results = collection.hybrid_search(
    reqs=[req_1, req_2],
    rerank=RRFRanker(k=60),
    limit=10,
    output_fields=["product_id", "category", "price"],
)

Advanced Pattern 2: Bulk Insert with Flush Control

For production ingestion pipelines, control when data becomes searchable:

# Batch insert with explicit flush
batch_size = 10000
total = 0

for i in range(0, len(vectors), batch_size):
    batch = vectors[i:i + batch_size]
    mr = collection.insert(batch)
    total += len(batch)

    # Flush every 100K vectors to make data searchable
    if total % 100000 == 0:
        collection.flush()
        print(f"Flushed {total} vectors — now searchable")

# Final flush
collection.flush()
print(f"Total: {total} vectors inserted and searchable")

Advanced Pattern 3: Time-Travel Queries with Consistency Levels

Milvus supports configurable consistency levels for the read-after-write tradeoff:

# Strong consistency — read your writes immediately
# Use for: real-time fraud detection, collaborative editing
collection.search(
    data=query_vector,
    anns_field="embedding",
    param=search_params,
    limit=10,
    consistency_level="Strong",
)

# Bounded staleness — read within N milliseconds of latest write
# Use for: recommendation engines, content feeds
collection.search(
    data=query_vector,
    anns_field="embedding",
    param=search_params,
    limit=10,
    consistency_level="Bounded",
    consistency_level_param=2000,  # 2 second staleness
)

# Eventual consistency — fastest reads, may miss recent writes
# Use for: batch analytics, offline processing
collection.search(
    data=query_vector,
    anns_field="embedding",
    param=search_params,
    limit=10,
    consistency_level="Eventually",
)

Production Considerations

Memory planning. Query nodes are memory-bound. Each 768-dim float vector consumes ~3 KB. For 100M vectors, that is ~300 GB of raw data. With HNSW index overhead (1.5x), you need ~450 GB of RAM. Plan for 64 GB per query node and scale horizontally.

Index rebuild strategy. Index building is CPU/GPU intensive. Plan maintenance windows for full rebuilds. For incremental updates, use Milvus’s auto-compaction (Data Nodes handle this) and rebuild indexes only when recall degrades below your threshold.

# Milvus config for index rebuild scheduling
dataCoord:
  compaction:
    enable: true
    maxSize: 1024  # MB — max segment size before compaction
  index:
    build:
      maxNodeNum: 4  # Max index nodes for parallel builds
      maxQueueSize: 10  # Queue depth before backpressure

Backup and restore. Milvus v3.0 adds snapshot-based backup/restore. For v2.x, use the milvus-backup tool:

# Install milvus-backup
pip install milvus-backup

# Create a backup
milvus-backup create -n my_backup_20260628

# Restore from backup
milvus-backup restore -n my_backup_20260628

Network configuration. For GPU clusters, RDMA (InfiniBand or RoCE) provides 18.6% total pipeline improvement over TCP for index creation. Configure your Kubernetes cluster with RDMA-aware network policies for index nodes.

The Results

Metric Before Milvus After Milvus Improvement
Max vector capacity 100M (Qdrant/Weaviate limit) 10B+ (Milvus) 100x more capacity
Index build time (106M, 2048-dim) 2 hours (CPU, 32 cores) 4 minutes (GPU, NVIDIA cuVS) 30x faster
Query throughput (100M, 768-dim) 8,000 QPS (Qdrant) 20,000+ QPS (Milvus + GPU) 2.5x higher
p99 latency (500M, 768-dim) 47-89 ms (competitors) 62 ms (Milvus + GPU node) Comparable at 5x scale
Filtered search recall@10 0.95 (Qdrant, 100M) 0.94 (Milvus, 100M) Comparable
Multi-tenancy Basic (100s of tenants) Enterprise (millions of tables) 1000x more tenants
Storage cost (1TB vectors) ~$2,500/month (Pinecone) ~$500/month (self-host, S3) 5x cheaper
Hot/cold tiering Not available Automatic (StorageV2, Parquet) New capability
GPU acceleration Not available CAGRA, IVF_PQ, BRUTE_FORCE New capability

What this means for you: Milvus is not the fastest vector database at every scale — Qdrant wins on latency and throughput at <100M vectors. But Milvus is the only open-source vector database that works at 10B+ vectors. The decision is simple: if your dataset fits in 100M vectors, use Qdrant. If it will grow past that, or if you need GPU acceleration, start with Milvus. The migration cost later is higher than the complexity tax now.

What to Watch Out For

  1. Do not use Milvus for datasets under 10M vectors. The operational complexity is not worth it. Use Milvus Lite for prototyping, then migrate to Qdrant or pgvector for production at small scale. Milvus is a sledgehammer; not every problem is a nail.

  2. GPU_CAGRA does not support COSINE distance. This is the most common mistake. Use Inner Product (IP) with normalized vectors. Normalize before insertion: vec = vec / np.linalg.norm(vec). IP on normalized vectors is mathematically equivalent to cosine similarity.

  3. Index your scalar fields. Without scalar indexes, filtered search falls back to full scan, which doubles or triples latency. Create INVERTED indexes on every field you filter on: collection.create_index("category", {"index_type": "INVERTED"}).

  4. Monitor memory on query nodes. Query nodes load entire segments into memory. If memory exceeds 80%, add more query nodes or reduce segment size. The metric milvus_querynode_memory_usage_bytes is your canary.

  5. Plan index rebuilds during low traffic. Index building spikes CPU/GPU utilization by 200-300%. Schedule rebuilds during maintenance windows. For GPU nodes, ensure the GPU memory pool is large enough for both the index build and cached indexes.

  6. Set nprobe correctly. Too low = poor recall. Too high = slow search. Start with nprobe = nlist / 64 and adjust based on your recall target. For nlist=4096, start at nprobe=64. For 99% recall, you may need nprobe=256.

  7. Use partitions for logical isolation, not for scaling. Partitions are logical groupings within a collection. They improve query performance by reducing the search space, but they do not increase capacity. For scaling, add more query nodes or shards.

  8. Do not use Milvus Lite for production. Milvus Lite is an embedded version for development and edge devices. It does not support clustering, GPU indexing, or high availability. Use Milvus Standalone or Cluster for production.

Lesson 1: “We spent three months tuning Milvus for a 500M vector workload. The single biggest improvement was switching from IVF_SQ8 to GPU_CAGRA — latency dropped from 45ms to 8ms at p99. The GPU was there the whole time; we just weren’t using it.” — Milvus user, r/vectordatabase

Lesson 2: “The disaggregated architecture saved us during Black Friday. Our query traffic spiked 5x, and we scaled from 20 to 100 query nodes in 15 minutes. No rebalancing, no downtime, no data migration. You cannot do this with a coupled architecture.” — Engineering lead, e-commerce platform

Lesson 3: “Our biggest mistake was not indexing scalar fields. We had a 100M vector collection with category and price filters, and every filtered search took 200ms+ because Milvus was scanning the entire collection. Adding INVERTED indexes on category and price dropped latency to 15ms. The indexes cost almost nothing to build and maintain.” — ML engineer, recommendation systems team

Advice for Getting Started

  1. Start with Milvus Lite for prototyping. It installs in 5 minutes and runs in-process. Build your schema, test your queries, validate your recall targets.
  2. Move to Milvus Standalone for integration testing. It runs as a single Docker container and supports the full API. Test your ingestion pipeline, index strategy, and query patterns.
  3. Deploy Milvus Cluster on Kubernetes for production. Use the Helm chart with the production values template above. Start with 3 proxy nodes, 5 query nodes, 2 index nodes (GPU), and 3 data nodes.
  4. Create indexes on every scalar field you filter on. This is the single highest-impact optimization for hybrid search.
  5. Monitor memory usage on query nodes from day one. Set up Prometheus + Grafana before you go live. The 200+ metrics Milvus exposes are not noise — they are your early warning system.
  6. Test with your actual data distribution. Synthetic benchmarks (random vectors) do not reflect real-world recall or latency. Use your production embeddings for load testing.
  7. Plan for index rebuilds. Schedule them during low traffic. Use GPU nodes for rebuilds if available. Monitor index build queue depth to avoid backpressure.

Next in the Open-Source AI Tools Mastery series: Weaviate

NL

Written by Nivant Labs Team

Engineer at Nivant Labs

Share this post

XLinkedInHN