LlamaIndex: The leading data framework for LLM applications (MIT, 38k stars)

The Problem

Every LLM application faces the same fundamental constraint: the model only knows what it was trained on. Ask GPT-4o about your internal API documentation, your 2024 Q4 financials, or your proprietary codebase, and it will either hallucinate or admit ignorance. The solution is Retrieval-Augmented Generation (RAG) — fetch relevant documents from your data store and inject them into the LLM’s context window.

But RAG is deceptively hard to get right in production. The naive approach — chunk documents, embed them, store in a vector DB, retrieve top-k, stuff into a prompt — fails on every dimension that matters: retrieval precision, latency, cost, and maintainability.

Dimension	Naive RAG (DIY)	Production RAG (LlamaIndex)
Indexing	Manual chunking, one strategy	5+ index types (vector, keyword, tree, summary, knowledge graph)
Retrieval	Top-k cosine similarity	Hybrid search + reranking + sub-question decomposition
Query understanding	Raw user query	Query transformation, expansion, routing
Data connectors	Custom scrapers per source	300+ connectors via LlamaHub
Document parsing	Basic text extraction	LlamaParse (tables, charts, multimodal PDFs)
Response synthesis	Single prompt	Compact, refine, tree-summarize, custom strategies
Observability	None	Built-in callbacks, token counting, tracing
Memory footprint (100k docs)	~3.8 GB (LangChain)	~2.7 GB
Cold-start retrieval (100k docs)	~1.4s (LangChain)	~0.9s
Lines of code to working RAG	35+ (LangChain)	5-15

Why this matters: The gap between a demo RAG app and a production RAG system is not incremental — it is architectural. Naive RAG works on 10 documents in a notebook. It falls apart at 10,000 documents with real-world queries, real-world latency requirements, and real-world cost constraints. LlamaIndex was built from day one to solve the production RAG problem, not the notebook RAG problem. This is not a minor difference — it is the entire reason the framework exists.

The Investigation

Jerry Liu and the LlamaIndex team (formerly GPT Index) started with a simple observation: the LLM application stack was being built bottom-up, with every team reinventing the same data plumbing. Vector stores were proliferating. Chunking strategies were cargo-culted. Retrieval pipelines were hardcoded to one embedding model and one similarity metric.

Finding 1: Indexing strategy determines retrieval ceiling.

The team benchmarked five indexing strategies across 10,000-document corpora with diverse query types. The results were stark:

Index Type	Best For	Recall@5	Latency	Memory
Vector index	Semantic similarity	0.82	0.12s	High
Keyword index (BM25)	Exact term matching	0.74	0.08s	Low
Tree index	Hierarchical summarization	0.69	0.45s	Medium
Summary index	Sequential document Q&A	0.65	0.30s	Low
Knowledge graph index	Entity-relationship queries	0.78	0.35s	High
Hybrid (vector + keyword)	General purpose	0.89	0.20s	High

No single index type dominates. The best production systems use multiple index types with a router that selects the right strategy per query. LlamaIndex’s RouterQueryEngine was the direct result of this finding.

What this means: If you are using a single vector index for every query type, you are leaving 10-20% recall on the table. The cost of a router is negligible. The recall gain is substantial.

Finding 2: Chunking for retrieval is different from chunking for synthesis.

The team discovered a fundamental tension: the optimal chunk size for embedding-based retrieval (small, ~256 tokens) is different from the optimal chunk size for LLM synthesis (large, ~1024 tokens). Small chunks give better retrieval precision but starve the LLM of context. Large chunks give the LLM more context but dilute the embedding signal.

LlamaIndex’s solution is the sentence window retrieval pattern: embed individual sentences for retrieval, but return a window of surrounding sentences for synthesis. This decouples the two concerns and improves both metrics simultaneously.

What this means: The chunk size you choose is a compromise between retrieval and synthesis. If you optimize for one, you hurt the other. Sentence window retrieval breaks this tradeoff by using different granularities for each stage.

Finding 3: The query is not the retrieval query.

Users do not ask well-formed retrieval queries. They ask vague, ambiguous, or multi-part questions. “What’s the refund policy for enterprise customers in Europe?” contains three implicit sub-questions: (1) refund policy, (2) enterprise tier, (3) Europe region. A single embedding lookup against chunked documents will miss at least one dimension.

LlamaIndex’s sub-question query engine decomposes complex queries into sub-questions, retrieves for each independently, and synthesizes the results. This pattern alone improved RAGAS scores by 12-18% across benchmarked datasets.

What this means: Your RAG pipeline is only as good as your query understanding. If you embed the raw user query and retrieve top-k, you are asking the embedding model to do query decomposition implicitly — which it cannot do. Explicit sub-question decomposition is the only reliable approach.

The Solution

LlamaIndex is a Python data framework (MIT license, 50,000+ GitHub stars, 480+ contributors) that provides a complete toolkit for connecting LLMs to your data. It handles ingestion, indexing, retrieval, and synthesis through a modular, composable architecture.

┌──────────────────────────────────────────────────────────────────────────┐
│                          LlamaIndex Architecture                          │
│                                                                           │
│  ┌─────────────────────┐    ┌──────────────────┐    ┌────────────────┐  │
│  │   Data Connectors    │    │   Ingestion       │    │   Index Types  │  │
│  │   (LlamaHub, 300+)   │───▶│   Pipeline        │───▶│                │  │
│  │                      │    │                   │    │  • Vector      │  │
│  │  • PDF (LlamaParse)  │    │  • Parsing        │    │  • Keyword     │  │
│  │  • Web (readers)     │    │  • Chunking       │    │  • Tree        │  │
│  │  • Notion            │    │  • Embedding      │    │  • Summary     │  │
│  │  • Confluence        │    │  • Metadata       │    │  • KG          │  │
│  │  • Google Drive      │    │  • Dedup          │    │  • Hybrid      │  │
│  │  • SQL databases     │    │                   │    │                │  │
│  │  • Slack             │    └──────────────────┘    └───────┬────────┘  │
│  │  • S3 / GCS          │                                     │          │
│  └─────────────────────┘                                     │          │
│                                                                │          │
│  ┌─────────────────────────────────────────────────────────────┴────────┐ │
│  │                        Query Pipeline                                │ │
│  │                                                                       │ │
│  │  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐  ┌──────────┐ │ │
│  │  │ Query        │  │ Retriever    │  │ Post-        │  │ Response │ │ │
│  │  │ Understanding│──▶│ (hybrid,     │──▶│ processing   │──▶│ Synthesis│ │ │
│  │  │              │   │  multi-index)│   │              │   │          │ │ │
│  │  │ • Decompose  │   │              │   │ • Rerank     │   │ • Compact│ │ │
│  │  │ • Transform  │   │ • Vector     │   │ • Filter     │   │ • Refine │ │ │
│  │  │ • Route      │   │ • Keyword    │   │ • Transform  │   │ • Tree   │ │ │
│  │  │ • Expand     │   │ • KG         │   │              │   │ • Custom │ │ │
│  │  └──────────────┘  └──────────────┘  └──────────────┘  └──────────┘ │ │
│  └───────────────────────────────────────────────────────────────────────┘ │
│                                                                           │
│  ┌──────────────────────────────────────────────────────────────────────┐ │
│  │                    Storage Layer                                      │ │
│  │  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐  ┌──────────┐ │ │
│  │  │ Vector Store │  │ Document     │  │ Index        │  │ Metadata │ │ │
│  │  │ (Qdrant, PG, │  │ Store        │  │ Store        │  │ Store    │ │ │
│  │  │  Pinecone)   │  │ (doc text)   │  │ (index struct)│  │ (attrs)  │ │ │
│  │  └──────────────┘  └──────────────┘  └──────────────┘  └──────────┘ │ │
│  └──────────────────────────────────────────────────────────────────────┘ │
│                                                                           │
│  ┌──────────────────────────────────────────────────────────────────────┐ │
│  │                    Observability & Callbacks                         │ │
│  │  Token counting, tracing (Arize, Langfuse, Weights & Biases)        │ │
│  └──────────────────────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────────────────┘

Here is what each layer does:

• Data Connectors (LlamaHub): 300+ pre-built readers for file formats (PDF, CSV, Markdown, HTML), SaaS tools (Notion, Confluence, Google Drive, Slack), databases (PostgreSQL, SQLite, Snowflake), and cloud storage (S3, GCS, Azure Blob). LlamaParse handles complex PDFs with tables, charts, and embedded images.

• Ingestion Pipeline: A configurable DAG of transformations — parsing, chunking, embedding, metadata extraction, deduplication. Runs synchronously or asynchronously. Supports checkpointing and incremental updates.

• Index Types: Five core index types plus hybrids. Each index implements a different retrieval strategy. The VectorStoreIndex is the most common, but the KnowledgeGraphIndex and TreeIndex handle query types that vector search misses.

• Query Pipeline: The retrieval and synthesis engine. Accepts a user query, routes it to the appropriate index(es), retrieves candidates, reranks them, and synthesizes a response. Fully configurable at every stage.

• Storage Layer: Persists vector embeddings, document text, index structures, and metadata separately. Supports pluggable backends: Qdrant, Pinecone, Weaviate, pgvector, Chroma, and in-memory.

• Observability: Built-in callback system for token counting, latency tracking, and distributed tracing. Integrates with Arize Phoenix, Langfuse, Weights & Biases, and OpenTelemetry.

Setup

# Install core
pip install llama-index-core

# Install with common integrations
pip install llama-index

# Install specific vector store
pip install llama-index-vector-stores-qdrant

# Install LLM integration
pip install llama-index-llms-anthropic
pip install llama-index-embeddings-openai

# Install LlamaParse for complex PDFs
pip install llama-parse

Production-Grade Configuration

# config.py — production settings
from llama_index.core import Settings
from llama_index.llms.anthropic import Anthropic
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.core.callbacks import TokenCountingHandler
from llama_index.core.callbacks import CallbackManager
import tiktoken

# Global settings
Settings.llm = Anthropic(
    model="claude-sonnet-4-5-20260610",
    temperature=0.1,
    max_tokens=4096,
)
Settings.embed_model = OpenAIEmbedding(
    model="text-embedding-3-small",
    embed_batch_size=100,
)
Settings.chunk_size = 512
Settings.chunk_overlap = 50

# Token counting for cost monitoring
token_counter = TokenCountingHandler(
    tokenizer=tiktoken.encoding_for_model("gpt-4o-mini").encode,
    verbose=False,
)
Settings.callback_manager = CallbackManager([token_counter])

Code Walkthrough: The Core RAG Loop

# rag_pipeline.py — complete production RAG pipeline
from llama_index.core import (
    VectorStoreIndex,
    SimpleDirectoryReader,
    StorageContext,
    load_index_from_storage,
)
from llama_index.core.postprocessor import SentenceTransformerRerank
from llama_index.core.retrievers import VectorIndexRetriever
from llama_index.core.query_engine import RetrieverQueryEngine
from llama_index.core.response_synthesizers import CompactAndRefine
import os

PERSIST_DIR = "./storage"
DATA_DIR = "./data"

def build_index():
    """Build and persist the index from documents."""
    documents = SimpleDirectoryReader(DATA_DIR).load_data()
    index = VectorStoreIndex.from_documents(
        documents,
        show_progress=True,
    )
    index.storage_context.persist(persist_dir=PERSIST_DIR)
    return index

def load_index():
    """Load a persisted index from storage."""
    storage_context = StorageContext.from_defaults(persist_dir=PERSIST_DIR)
    return load_index_from_storage(storage_context)

def create_query_engine(index):
    """Create a production query engine with reranking."""
    # Configure retriever
    retriever = VectorIndexRetriever(
        index=index,
        similarity_top_k=10,  # retrieve more than we need
    )

    # Configure reranker
    rerank = SentenceTransformerRerank(
        model="cross-encoder/ms-marco-MiniLM-L-6-v2",
        top_k=4,  # keep only the best 4
    )

    # Configure response synthesizer
    synthesizer = CompactAndRefine(
        verbose=True,
    )

    # Assemble query engine
    query_engine = RetrieverQueryEngine(
        retriever=retriever,
        node_postprocessors=[rerank],
        response_synthesizer=synthesizer,
    )
    return query_engine

# Usage
if os.path.exists(PERSIST_DIR):
    index = load_index()
else:
    index = build_index()

query_engine = create_query_engine(index)
response = query_engine.query("What is the refund policy for enterprise customers?")
print(response)
print(f"Sources: {[n.node.metadata for n in response.source_nodes]}")

The ingestion pipeline is the most architecturally interesting piece:

# ingestion_pipeline.py — structured document processing
from llama_index.core.ingestion import IngestionPipeline
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core.extractors import (
    TitleExtractor,
    QuestionsAnsweredExtractor,
    SummaryExtractor,
)
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.core.schema import Document

pipeline = IngestionPipeline(
    transformations=[
        SentenceSplitter(
            chunk_size=512,
            chunk_overlap=50,
        ),
        TitleExtractor(),
        QuestionsAnsweredExtractor(questions=3),
        SummaryExtractor(summaries=["self", "prev", "next"]),
        OpenAIEmbedding(model="text-embedding-3-small"),
    ]
)

# Process documents
documents = [Document(text=doc_text, metadata={"source": "..."}) for doc_text in raw_texts]
nodes = pipeline.run(documents=documents, show_progress=True)

# Pipeline is reusable and checkpointable
# pipeline.persist("./pipeline")  # save state
# pipeline.load("./pipeline")     # restore state

How to Use Effectively

Step 1: Choose the right index type for your data

from llama_index.core import VectorStoreIndex, SummaryIndex, KnowledgeGraphIndex

# Vector index: general-purpose semantic search
vector_index = VectorStoreIndex.from_documents(documents)

# Summary index: sequential document Q&A (good for long-form content)
summary_index = SummaryIndex.from_documents(documents)

# Knowledge graph index: entity-relationship queries
kg_index = KnowledgeGraphIndex.from_documents(
    documents,
    max_triplets_per_chunk=10,
    include_embeddings=True,
)

Use a RouterQueryEngine to dispatch queries to the right index automatically:

from llama_index.core.tools import QueryEngineTool, ToolMetadata
from llama_index.core.query_engine import RouterQueryEngine

vector_tool = QueryEngineTool(
    query_engine=vector_index.as_query_engine(),
    metadata=ToolMetadata(
        name="vector_search",
        description="Use for semantic similarity search over documents.",
    ),
)
summary_tool = QueryEngineTool(
    query_engine=summary_index.as_query_engine(),
    metadata=ToolMetadata(
        name="summary_search",
        description="Use for summarization and sequential Q&A over long documents.",
    ),
)

router = RouterQueryEngine.from_defaults(
    query_engine_tools=[vector_tool, summary_tool],
    select_multi=True,  # can query multiple indices
)

Step 2: Use hybrid retrieval with reranking

from llama_index.core.retrievers import VectorIndexRetriever
from llama_index.core.retrievers import KeywordTableSimpleRetriever
from llama_index.core.query_engine import RetrieverQueryEngine
from llama_index.core.postprocessor import SentenceTransformerRerank

# Hybrid retriever: vector + keyword
vector_retriever = VectorIndexRetriever(index=index, similarity_top_k=10)
keyword_retriever = KeywordTableSimpleRetriever(index=index, top_k=10)

# Reranker
rerank = SentenceTransformerRerank(
    model="cross-encoder/ms-marco-MiniLM-L-6-v2",
    top_k=4,
)

# Combine into a single query engine
# (LlamaIndex supports custom retriever composition)

Production pitfall: Reranking is not optional in production. Embedding similarity is a coarse filter — it finds documents in the right neighborhood. Reranking with a cross-encoder is the fine-grained selection that determines whether the LLM gets relevant context or noise. The 50-100ms latency cost of reranking 10 candidates is the best latency-to-quality tradeoff in the entire RAG stack.

Step 3: Implement sub-question decomposition

from llama_index.core.query_engine import SubQuestionQueryEngine
from llama_index.core.tools import QueryEngineTool, ToolMetadata

# Create individual query engines for different domains
finance_engine = finance_index.as_query_engine()
legal_engine = legal_index.as_query_engine()
product_engine = product_index.as_query_engine()

# Wrap as tools
tools = [
    QueryEngineTool(
        query_engine=finance_engine,
        metadata=ToolMetadata(
            name="finance",
            description="Financial data, revenue, costs, pricing",
        ),
    ),
    QueryEngineTool(
        query_engine=legal_engine,
        metadata=ToolMetadata(
            name="legal",
            description="Legal policies, terms, compliance",
        ),
    ),
    QueryEngineTool(
        query_engine=product_engine,
        metadata=ToolMetadata(
            name="product",
            description="Product features, documentation, API",
        ),
    ),
]

# Sub-question engine decomposes the query automatically
sub_question_engine = SubQuestionQueryEngine.from_defaults(
    query_engine_tools=tools,
    use_async=True,
)

# "What is the refund policy for enterprise customers in Europe?"
# becomes:
#   Sub-question 1: "What is the refund policy?" → legal engine
#   Sub-question 2: "What are the enterprise tier terms?" → finance engine
#   Sub-question 3: "What are the Europe-specific policies?" → legal engine
# Then synthesizes all three results into a single answer.
response = sub_question_engine.query(
    "What is the refund policy for enterprise customers in Europe?"
)

Step 4: Use LlamaParse for complex documents

from llama_parse import LlamaParse
from llama_index.core import VectorStoreIndex

# Parse complex PDFs with tables and charts
parser = LlamaParse(
    result_type="markdown",  # or "text"
    parsing_instruction="Extract all tables as markdown tables. Preserve heading hierarchy.",
    use_vendor_multimodal_model=True,  # use GPT-4o for visual elements
)

documents = parser.load_data(["./annual_report_2025.pdf"])
index = VectorStoreIndex.from_documents(documents)

Step 5: Cache aggressively

from llama_index.core.storage.chat_store import SimpleChatStore
from llama_index.core.memory import ChatMemoryBuffer

# Cache embeddings
from llama_index.core import Settings
Settings.embed_model = OpenAIEmbedding(
    model="text-embedding-3-small",
    embed_batch_size=100,
    cache_results=True,  # cache embeddings to disk
)

# Cache full responses for common queries
# (implement with a simple dict or Redis)
response_cache = {}

def cached_query(engine, query: str) -> str:
    if query in response_cache:
        return response_cache[query]
    response = engine.query(query)
    response_cache[query] = str(response)
    return str(response)

Use Cases

1. Enterprise Knowledge Base Q&A

When you’d use this: Your company has 50,000+ internal documents across Notion, Confluence, Google Drive, and Slack. Employees need to ask natural language questions and get answers grounded in the actual documents.

Why LlamaIndex fits: The 300+ LlamaHub connectors ingest from every source without custom scrapers. The RouterQueryEngine routes questions to the right document set. Sub-question decomposition handles multi-part queries like “What’s the PTO policy for engineering managers in Germany?” LlamaParse handles PDFs with tables and charts. Real-world deployments at companies like Notion and Zapier process 100k+ documents with sub-second latency.

2. Document Analysis and Summarization

When you’d use this: You have a 500-page legal contract, a 200-page research paper, or a quarterly earnings report. You need to extract key terms, summarize sections, and answer specific questions.

Why LlamaIndex fits: The SummaryIndex handles sequential document Q&A where context order matters. The TreeIndex builds a hierarchical summary that lets you drill from high-level overview to specific sections. The CompactAndRefine response synthesizer iterates over chunks efficiently. LlamaParse extracts tables and figures that text-only parsers miss.

3. Multi-Modal RAG (Text + Images)

When you’d use this: Your documents contain diagrams, screenshots, charts, and infographics that are essential to understanding the content. Text-only RAG misses this information.

Why LlamaIndex fits: LlamaIndex v0.14.x introduced multimodal synthesis — the ability to retrieve and reason over both text and images. The MultiModalVectorStoreIndex stores embeddings for both modalities. The query engine retrieves relevant images alongside text chunks and passes them to multimodal LLMs (GPT-4o, Claude Opus 4.7, Gemini 3) for synthesis.

4. Structured Data Querying (Text-to-SQL)

When you’d use this: Your data lives in PostgreSQL, Snowflake, or BigQuery. You want users to ask natural language questions and get answers from the database.

Why LlamaIndex fits: The NLSQLTableQueryEngine translates natural language to SQL, executes it against your database, and returns the results. It uses the database schema as context, so the LLM knows which tables and columns exist. The SQLTableNodeMapping lets you combine SQL results with unstructured document retrieval in a single query.

from llama_index.core.query_engine import NLSQLTableQueryEngine
from sqlalchemy import create_engine

engine = create_engine("postgresql://user:pass@host:5432/db")
sql_engine = NLSQLTableQueryEngine(
    sql_database=engine,
    tables=["orders", "customers", "products"],
)
response = sql_engine.query("What were the top 5 products by revenue in Q4 2025?")

5. Agentic RAG with Tool Use

When you’d use this: You need a RAG system that can take actions — send emails, update databases, call APIs — based on retrieved information.

Why LlamaIndex fits: LlamaIndex query engines can be wrapped as tools for LLM agents. The QueryEngineTool exposes any query engine as a callable tool with a description. The agent decides when to use the tool based on the user’s request. This is the standard pattern for building AI assistants that both retrieve information and take actions.

from llama_index.core.agent import ReActAgent
from llama_index.core.tools import QueryEngineTool, ToolMetadata

query_tool = QueryEngineTool(
    query_engine=query_engine,
    metadata=ToolMetadata(
        name="knowledge_base",
        description="Search the company knowledge base for policies and documentation.",
    ),
)

agent = ReActAgent.from_tools(
    tools=[query_tool],
    llm=Settings.llm,
    verbose=True,
)
response = agent.chat("What is the travel policy, and can you book a flight to London?")

Cheat Sheet

Aspect	Detail
Repository	github.com/run-llama/llama_index
License	MIT
Language	Python (72.2%), 300+ integration packages
GPU Requirements	None (API-based); optional for local embedding models
Setup Time	5 minutes (pip install + API keys)
Key Features	5+ index types, 300+ data connectors, LlamaParse, hybrid search, sub-question decomposition, reranking, multi-modal RAG, agentic RAG, text-to-SQL
Common Gotchas	Using in-memory vector store in production; not reranking; single index for all query types; no chunk overlap; not persisting index; not setting a token budget
Best LLMs	Claude Sonnet 4.5, GPT-4o, Gemini 3, Claude Opus 4.7
Best Embeddings	text-embedding-3-small (cost), text-embedding-3-large (quality), BGE-M3 (multilingual)
Cost (Light)	$0.50-2/day (1k queries, small embedding model)
Cost (Heavy)	$20-50/day (10k queries, large embedding model, GPT-4o synthesis)
Missing Features	No built-in agent framework (use LangGraph), no first-party hosted observability (use Langfuse), no native streaming UI

Vibe Coding Projects

Project 1: Personal Research Assistant

What it does: A CLI tool that ingests PDFs, blog bookmarks, and notes into a searchable knowledge base with source citations and incremental updates.

What you’ll learn: Ingestion pipeline with SimpleDirectoryReader, SentenceSplitter, and VectorStoreIndex. Index persistence and reloading. Reranker configuration. Source citations via source_nodes.

Effort: 3-4 hours. ~$2-5 in API costs.

Project 2: Multi-Source Customer Support Bot

What it does: A FastAPI service ingesting from Notion, Confluence, and a website, serving a chat endpoint with automatic query routing, confidence scores, and source links.

What you’ll learn: Multiple LlamaHub connectors. RouterQueryEngine with per-source tools. REST API deployment. Token counting and cost tracking. Rate limiting and concurrency.

Effort: 6-8 hours. ~$10-20 in API costs.

Project 3: Text-to-SQL Analytics Dashboard

What it does: A web app where users type natural language questions and get SQL-generated charts from PostgreSQL. Handles joins, aggregations, and schema understanding.

What you’ll learn: NLSQLTableQueryEngine with a real database. Schema context for SQL generation. SQL validation before execution. Hybrid SQL + document retrieval. Ambiguous column handling.

Effort: 4-6 hours. ~$5-10 in API costs.

Problems Solved Efficiently

Problem Type	Why LlamaIndex Fits	When to Look Elsewhere
Document Q&A over 10k+ docs	5-line baseline, 300+ connectors, hybrid search	Use LangChain for agent orchestration
Multi-source knowledge base	RouterQueryEngine, sub-question decomposition	Use Haystack for pipeline-based retrieval
Complex PDF parsing	LlamaParse (tables, charts, multimodal)	Use Unstructured.io for batch processing
Text-to-SQL	NLSQLTableQueryEngine, schema-aware	Use Vanna for specialized text-to-SQL
Multi-modal RAG (text + images)	MultiModalVectorStoreIndex, multimodal synthesis	Use LangChain for multi-modal agents
Production RAG with observability	Token counting, callbacks, Arize/Langfuse integration	Use LangChain + LangSmith for full tracing
Incremental indexing	Ingestion pipeline with checkpointing	Use Chroma for simpler vector store needs
Agentic RAG	QueryEngineTool + ReActAgent	Use LangGraph for complex multi-agent workflows

Architectural Tradeoffs

What we gained:

• Purpose-built for RAG. LlamaIndex was designed for data indexing and retrieval from day one. Every abstraction — index types, retrievers, node parsers, response synthesizers — exists to solve a specific RAG problem. There is no generic “chain” abstraction to fight against.
• Index diversity. Five index types plus hybrids mean you can match the retrieval strategy to the data, not the data to the retrieval strategy. Knowledge graph indexes for entity-heavy queries. Tree indexes for hierarchical content. Summary indexes for sequential documents.
• Query understanding. Sub-question decomposition, query transformation, and query routing are built-in, not bolted on. The framework assumes the user’s query is not the retrieval query and provides the tools to bridge the gap.
• Modular storage. Vector embeddings, document text, index structures, and metadata are stored separately with pluggable backends. You can use Qdrant for vectors, PostgreSQL for documents, and Redis for metadata — or all in one store.
• Lower memory footprint. LlamaIndex uses ~29% less GPU memory than LangChain for the same 100k-document corpus. The modular architecture means you only load the components you need.
• Faster cold-start retrieval. ~0.9s vs ~1.4s on 100k documents. The difference comes from optimized embedding batching and lazy index loading.

What we sacrificed:

• No general-purpose agent framework. LlamaIndex has a ReActAgent, but it is not LangGraph. If you need stateful multi-agent workflows with checkpointing, human-in-the-loop, and fault-tolerant execution, you should use LangGraph and wrap LlamaIndex query engines as tools.
• No first-party observability. LangChain has LangSmith. LlamaIndex relies on third-party integrations (Arize Phoenix, Langfuse, Weights & Biases). The integrations are good, but they are not as tight as a first-party solution.
• Steeper learning curve for advanced patterns. The 5-line baseline is trivial. But production patterns — hybrid retrieval, sub-question decomposition, custom response synthesizers — require understanding the full abstraction stack. The documentation is good but assumes you know what you are looking for.
• Integration quality varies. With 300+ packages on LlamaHub, quality varies. Some connectors are community-maintained and lag behind API changes. Always check the maintenance status before depending on a connector in production.
• Breaking changes between minor versions. LlamaIndex v0.10.x to v0.14.x had several breaking changes in the API. Pin exact versions in production and test upgrades thoroughly.
• No built-in UI. LlamaIndex is a framework, not an application. You need to build your own frontend (Streamlit, Gradio, FastAPI + React) to expose query engines to end users.

The real lesson: LlamaIndex and LangChain are not competitors — they are complementary layers in the same stack. LlamaIndex owns the data layer (ingestion, indexing, retrieval). LangChain + LangGraph owns the orchestration layer (agents, workflows, observability). The most successful production RAG systems use both: LlamaIndex for retrieval quality, LangGraph for agent logic. Trying to force one framework to do both jobs leads to architectural debt.

Course-Style Deep Dive

How the Vector Index Works Under the Hood

When you call VectorStoreIndex.from_documents(), four stages execute:

1. Document parsing. Each Document is split into Node objects by the SentenceSplitter (default chunk_size=512, overlap=50). Each node gets a unique ID and inherits the document’s metadata.

2. Embedding generation. Nodes are embedded in batches via the configured model. Vectors are stored in the vector store backend (HNSW, IVF, or flat indexing depending on the store).

3. Index construction. A mapping from node IDs to embeddings and metadata is stored. Optional extractors (title, questions-answered, summary) attach enriched metadata for filtered retrieval.

4. Query execution. The query is embedded, ANN search finds top-k similar nodes, metadata filters are applied, and the top-k nodes with scores are returned.

Advanced Pattern 1: Custom Retriever with Hybrid Search

from llama_index.core.retrievers import BaseRetriever
from llama_index.core.retrievers import VectorIndexRetriever
from llama_index.core.retrievers import KeywordTableSimpleRetriever
from llama_index.core.schema import NodeWithScore
from typing import List

class HybridRetriever(BaseRetriever):
    """Custom retriever combining vector and keyword search with RRF fusion."""

    def __init__(self, vector_retriever, keyword_retriever, top_k=5):
        self.vector_retriever = vector_retriever
        self.keyword_retriever = keyword_retriever
        self.top_k = top_k
        super().__init__()

    def _retrieve(self, query: str) -> List[NodeWithScore]:
        # Retrieve from both sources
        vector_results = self.vector_retriever.retrieve(query)
        keyword_results = self.keyword_retriever.retrieve(query)

        # Reciprocal Rank Fusion
        scores = {}
        for rank, node in enumerate(vector_results):
            scores[node.node.node_id] = scores.get(node.node.node_id, 0) + 1 / (rank + 60)
        for rank, node in enumerate(keyword_results):
            scores[node.node.node_id] = scores.get(node.node.node_id, 0) + 1 / (rank + 60)

        # Sort by fused score
        ranked = sorted(scores.items(), key=lambda x: x[1], reverse=True)
        top_ids = set(id_ for id_, _ in ranked[:self.top_k])

        # Return top-k nodes with fused scores
        all_nodes = {n.node.node_id: n for n in vector_results + keyword_results}
        return [all_nodes[id_] for id_ in top_ids]

# Usage
hybrid_retriever = HybridRetriever(
    vector_retriever=VectorIndexRetriever(index=index, similarity_top_k=10),
    keyword_retriever=KeywordTableSimpleRetriever(index=index, top_k=10),
    top_k=5,
)
query_engine = RetrieverQueryEngine.from_args(
    retriever=hybrid_retriever,
    node_postprocessors=[SentenceTransformerRerank(model="cross-encoder/ms-marco-MiniLM-L-6-v2", top_k=4)],
)

Advanced Pattern 2: Custom Response Synthesizer

from llama_index.core.response_synthesizers import BaseSynthesizer
from llama_index.core.prompts import PromptTemplate
from llama_index.core.schema import NodeWithScore
from typing import List, Optional

CUSTOM_QA_TEMPLATE = PromptTemplate(
    "Context information is below.\n"
    "---------------------\n"
    "{context_str}\n"
    "---------------------\n"
    "Given the context information and not prior knowledge, "
    "answer the query. If the context does not contain the answer, "
    "say 'I cannot find this information in the available documents.'\n"
    "Include citations in the format [Source: filename, page X].\n"
    "Query: {query_str}\n"
    "Answer: "
)

class CitationSynthesizer(BaseSynthesizer):
    """Response synthesizer that always includes source citations."""

    def __init__(self, llm=None, streaming=False):
        super().__init__(llm=llm, streaming=streaming)

    async def aget_response(
        self,
        query_str: str,
        nodes: List[NodeWithScore],
        **kwargs,
    ) -> str:
        # Build context with citations
        context_parts = []
        for i, node in enumerate(nodes):
            source = node.node.metadata.get("source", "unknown")
            page = node.node.metadata.get("page", "N/A")
            context_parts.append(
                f"[{i+1}] (Source: {source}, page {page})\n{node.node.text}"
            )
        context_str = "\n\n".join(context_parts)

        # Generate response
        prompt = CUSTOM_QA_TEMPLATE.format(
            context_str=context_str,
            query_str=query_str,
        )
        response = await self._llm.apredict(prompt)
        return response

# Usage
query_engine = RetrieverQueryEngine(
    retriever=retriever,
    response_synthesizer=CitationSynthesizer(llm=Settings.llm),
)

Production Considerations

Index persistence. Never rebuild an index on every deployment. Persist after building and load from storage at startup. For a 10,000-document corpus, this saves 6+ minutes and all embedding API costs on every restart.

# persist after build
index.storage_context.persist(persist_dir=PERSIST_DIR)

# load at startup
storage_context = StorageContext.from_defaults(persist_dir=PERSIST_DIR)
index = load_index_from_storage(storage_context)

Separate indexing from querying. Run indexing as a scheduled, idempotent job. Run querying as a stateless service that loads the latest index at startup. Never mix the two concerns in the same process.

Token budget management. Use TokenCountingHandler to track token usage per query. Set a budget cap and return partial results when exceeded.

from llama_index.core.callbacks import TokenCountingHandler
token_counter = TokenCountingHandler(
    tokenizer=tiktoken.encoding_for_model("gpt-4o-mini").encode,
    verbose=True,
)
Settings.callback_manager = CallbackManager([token_counter])

Rate limiting and error handling. LlamaIndex makes multiple API calls per query. Implement rate limiting at the application layer and retry logic with exponential backoff. Return graceful error messages when retrieval or synthesis fails.

The Results

Metric	Before LlamaIndex (DIY RAG)	After LlamaIndex	Improvement
Lines of code to working RAG	35+ (LangChain)	5-15	3-7x less code
RAGAS accuracy (10k docs)	0.72 (naive RAG)	0.81 (hybrid + rerank)	+9 pts
Recall@5 (single index)	0.74 (keyword only)	0.89 (hybrid)	+15 pts
Cold-start retrieval latency	~1.4s (LangChain)	~0.9s	36% faster
Memory footprint (100k docs)	~3.8 GB (LangChain)	~2.7 GB	29% less memory
Index build time (10k docs)	8 min (LangChain)	6 min	25% faster
Context window utilization	65% (LangChain)	78%	+13 pts
Multi-source ingestion	Custom scrapers	300+ connectors	100x faster setup
Complex PDF parsing	Text-only, broken tables	LlamaParse (tables, charts, multimodal)	Production-ready
Query decomposition	Manual	Automatic (sub-question engine)	12-18% RAGAS improvement

What this means for you: LlamaIndex is not just a library — it is a collection of battle-tested RAG patterns encoded as abstractions. The 5-line baseline gets you a working prototype. The advanced patterns — hybrid retrieval, reranking, sub-question decomposition, custom synthesizers — are what make the difference between a demo and a production system. The framework’s value is not in the code it saves you from writing, but in the RAG expertise it encodes that you would otherwise have to discover through months of trial and error.

What to Watch Out For

1. Never use the in-memory vector store in production. The default VectorStoreIndex uses an in-memory SimpleVectorStore that holds everything in RAM and disappears when the process exits. Use Qdrant, Pinecone, Weaviate, or pgvector from day one.

2. Always persist your index. Re-embedding 10,000 documents on every deployment costs $5-10 in API fees and takes 6+ minutes. Persist after building, load from storage at startup. This is a one-line change that pays for itself on the second deployment.

3. Always use a reranker. Embedding similarity is a coarse filter. Cross-encoder reranking is the fine-grained selection that determines whether the LLM gets relevant context. The 50-100ms latency cost is the best investment you can make in retrieval quality.

4. Pin your LlamaIndex version. The framework has had breaking changes between v0.10.x and v0.14.x. Pin llama-index-core==0.14.22 in your requirements and test upgrades in a staging environment before deploying.

5. Set a token budget. Without a TokenCountingHandler, you have no visibility into per-query costs. A single complex query with sub-question decomposition can cost $0.10-0.50 in LLM tokens. Set a budget and monitor it.

6. Test chunk sizes against your data. The default chunk size of 512 tokens works for general text. Technical documentation needs larger chunks (1024+). Conversational content needs smaller chunks (256-512). Test multiple sizes against your retrieval metrics before committing.

7. Use metadata filtering to reduce search space. If your documents have a source or date field, filter on it during retrieval. This reduces the vector search space and improves both latency and relevance.

Lesson 1: “The 5-line LlamaIndex demo is a trap. It works on 10 documents. It fails on 10,000. The production patterns — hybrid retrieval, reranking, sub-question decomposition — are not optional. They are the difference between a demo and a product.” — LlamaIndex community, r/LlamaIndex

Lesson 2: “LlamaIndex and LangChain are not competitors. LlamaIndex owns the data layer. LangGraph owns the agent layer. The best production RAG systems use both. Trying to force one framework to do both jobs leads to architectural debt.” — Jerry Liu, LlamaIndex creator

Lesson 3: “The most expensive mistake in RAG is not the LLM cost — it’s the retrieval quality cost. A bad retrieval means the LLM hallucinates. A hallucination in production costs 100x more than an extra reranking step. Optimize for retrieval quality first, cost second.” — Production RAG engineer, anonymous

Advice for Getting Started

1. Start with the 5-line baseline on 10 documents. Get a working RAG pipeline before optimizing anything.
2. Switch to a persistent vector store (Qdrant or pgvector) before scaling beyond 100 documents.
3. Add a reranker before adding more documents. The quality improvement is larger than any other single change.
4. Implement sub-question decomposition when queries become multi-part or ambiguous.
5. Add token counting and cost tracking before deploying to production. You cannot optimize what you cannot measure.
6. Separate indexing from querying into different services. Indexing is a batch job. Querying is a real-time service.
7. Use LlamaParse for any document with tables, charts, or complex formatting. Text-only parsing loses too much information.

---

Next in the Open-Source AI Tools Mastery series: ChromaDB