Building a Multi-Agent RAG Pipeline: Lessons from Production

Meet Raj. He’s an ML engineer at a mid-sized tech company, and he just built what he thought was a solid document Q&A system for his team’s internal knowledge base. The demo was flawless — he asked “What’s our deployment process?” and got a perfect answer with citations. His manager was impressed. The team gave him a round of applause.

Then 50 users started asking questions at the same time.

Queries started timing out. The AI began making up answers — confidently citing documents that didn’t exist. The GPT-4 bill hit four figures before lunch. Raj’s beautiful demo had become a production nightmare.

Here’s how he rebuilt it, and what your team can learn from his journey.

Why this matters: A RAG (Retrieval-Augmented Generation) system is how most teams give LLMs access to their own data — docs, codebases, support tickets. It’s the difference between a chatbot that guesses and one that cites real sources. But getting it right at scale is harder than it looks.

The Problem

Raj’s first architecture was the textbook setup you’ve probably seen in every tutorial:

User Query → Embedding → Vector Search → LLM Generation → Response

Simple. Elegant. And completely wrong for real-world traffic.

The first sign of trouble was latency. A single query took 4-6 seconds from start to finish. The embedding step (converting the question into a numerical vector) was fast — about 200ms. But the vector search (finding similar documents in the database) started slowing down as Raj added more documents. By the time he had 50,000 document chunks, just the search step took over a second.

The second sign was context poisoning. Raj’s system retrieved the top 5 most similar chunks for every query. But 2 or 3 of those chunks were often irrelevant. The LLM would latch onto those irrelevant chunks and generate answers that sounded confident but were completely wrong. This is called hallucination, and it’s dangerous when people trust the answers.

The third sign was cost. Every query sent 5 chunks of roughly 4,000 tokens each to GPT-4. At a few hundred queries per day, that’s manageable. At thousands per day, your cloud bill starts looking like a car payment.

Production pitfall: If your RAG system works perfectly in demo mode but falls apart under load, you’re not alone. The demo uses one user, one query, and a small document set. Production means concurrent users, growing data, and real-world query variety. They’re different games.

The Investigation

Raj profiled every stage of his pipeline to find the bottlenecks. Here’s what the data looked like:

Stage	P50 (typical)	P95 (worst case)	Bottleneck
Query embedding	180ms	350ms	Model size — bigger embedding models are more accurate but slower
Vector search	800ms	2.1s	Index size — more documents means slower searches without optimization
Reranking	0ms	0ms	Not implemented — Raj wasn’t reordering results by relevance at all
LLM generation	3.2s	8.5s	Context length — too many chunks meant the LLM had too much to process
Citation validation	0ms	0ms	Not implemented — Raj had no way to check if citations were real

Let’s translate those metrics. P50 means half your queries are faster than this number. P95 means 95% are faster — the slow ones that frustrate users. A P95 of 8.5 seconds means 1 in 20 users is waiting almost 9 seconds for an answer. That’s enough time to switch tabs and forget they asked.

The data told a clear story: Raj was doing too much in one shot and not validating anything. Every query went through the same pipeline whether it was a simple factual question (“What’s the company holiday policy?”) or a complex analytical one (“Compare our Q2 deployment frequency to Q1”). And nothing checked whether the retrieved documents were actually relevant or the citations were real.

The Solution: Multi-Agent Architecture

Raj redesigned his pipeline as a multi-agent system. Instead of one monolithic process, he split the work across specialized agents, each responsible for one thing and doing it well.

User Query
    │
    ▼
┌─────────────────┐
│  Query Router   │  ← Classifies query type (factual, analytical, code)
└────────┬────────┘
         │
    ┌────┴────┐
    ▼         ▼
┌────────┐ ┌────────┐
│Retrieve│ │  Code   │  ← Parallel specialized retrievers
│  Text  │ │ Search  │
└───┬────┘ └───┬────┘
    │         │
    ▼         ▼
┌─────────────────┐
│   Reranker      │  ← Cross-encoder reranking (Cohere)
└────────┬────────┘
         │
    ┌────┴────┐
    ▼         ▼
┌────────┐ ┌────────┐
│  LLM   │ │Citation│  ← Parallel generation + validation
│ Gen    │ │ Check  │
└───┬────┘ └───┬────┘
    │         │
    ▼         ▼
┌─────────────────┐
│  Aggregator     │  ← Final response assembly
└─────────────────┘

Here’s what each piece does:

Query Router — A lightweight classifier (not a full LLM call) that figures out what kind of question the user is asking. Is it a factual question about a process? An analytical question comparing data? A code question about an API? Each type gets routed to a different retriever, so you’re not searching code docs for a policy question.

Specialized Retrievers — Instead of one giant vector index, Raj split his data into a text index (documentation, policies, guides) and a code index (API references, code examples, function docs). Each retriever also uses hybrid search — combining dense vector search (semantic similarity) with sparse keyword search (exact term matching) — which catches queries that pure semantic search misses.

Reranker — A cross-encoder model that takes the query and each retrieved chunk together and scores their actual relevance. This is the secret weapon. The initial vector search is fast but fuzzy. The reranker is slower but much more accurate. Using both together gives you speed and precision.

LLM Generator — The actual answer generation, but with fewer, higher-quality chunks. Because the reranker already filtered out the noise, the LLM gets only the most relevant context.

Citation Validator — A separate check that verifies every citation the LLM produces actually exists in the retrieved documents. If the LLM invents a citation, this catches it.

Aggregator — Assembles the final response with verified citations.

Key Design Decisions

1. Query Router

The router uses a lightweight classifier — not an expensive LLM call. It converts the query to an embedding and runs a simple prediction:

async def route_query(query: str) -> QueryType:
    embedding = await fast_embed(query)
    probs = router_model.predict(embedding)
    return QueryType(probs.argmax())

This takes about 30ms and saves hundreds of milliseconds downstream by sending queries to the right retriever on the first try.

Key lesson: Don’t use an LLM for every decision. A small classifier that costs pennies and runs in milliseconds can handle routing, filtering, and classification tasks. Save the LLM for the work only it can do — generating answers.

2. Specialized Retrievers

Instead of one vector index, Raj split into text and code indices, and added keyword search alongside vector search:

class HybridRetriever:
    def __init__(self):
        self.text_index = VectorIndex("text-chunks")
        self.code_index = VectorIndex("code-chunks")
        self.bm25_index = BM25Index()

    async def retrieve(self, query: str, qtype: QueryType, top_k: int = 10):
        if qtype == QueryType.CODE:
            results = await self.code_index.search(query, top_k)
        else:
            dense = await self.text_index.search(query, top_k)
            sparse = self.bm25_index.search(query, top_k // 2)
            results = self.reciprocal_rank_fusion(dense, sparse)
        return results

The reciprocal_rank_fusion step combines results from dense and sparse search by averaging their rank positions. If a document ranks #3 in vector search and #5 in keyword search, it gets a combined score that reflects both. This catches documents that are semantically similar but use different keywords, and vice versa.

Why this matters: Pure vector search is great at finding “similar meaning” but can miss exact matches. Pure keyword search finds exact terms but misses synonyms. Hybrid search gives you both. For technical documentation with code snippets, this is especially important — “sort()” and “arrange elements” should find the same document.

3. Cross-Encoder Reranking

The single biggest quality improvement came from adding a reranking step:

class Reranker:
    async def rerank(self, query: str, chunks: list[Chunk], top_k: int = 3):
        pairs = [(query, c.text[:256]) for c in chunks]
        scores = await self.cohere.rerank(pairs=pairs)
        return [c for c, s in zip(chunks, scores) if s > 0.3][:top_k]

Here’s the intuition: the initial vector search is like a librarian who quickly points you to the right shelf. The reranker is like a second librarian who reads each book’s summary and picks the three most relevant ones. Together, they’re fast and accurate.

This single change improved relevance from 72% to 94%. That’s the difference between a system users trust and one they learn to ignore.

Key lesson: The reranker was the highest-impact change in the entire system. Not a better LLM, not a bigger index — just ordering results better. If you’re building a RAG system and can only make one optimization, start here.

The Results

After Raj deployed the multi-agent architecture, the numbers told a different story:

Metric	Before	After	Improvement
P50 latency	4.2s	1.8s	57% faster
P95 latency	9.1s	3.4s	63% faster
Relevance score	72%	94%	+22pp
Cost per query	$0.042	$0.018	57% cheaper
Citation accuracy	68%	96%	+28pp

What this means for you: Your users get answers in under 2 seconds instead of waiting 4-9 seconds. The answers are more likely to be correct and cite real sources. And your budget goes further — roughly half the cost per query.

The citation validator caught hallucinated citations in about 12% of generations, even with GPT-4. That means even the best LLMs make up sources, and you need a safety net.

Hybrid search (dense + sparse) significantly outperformed pure vector search for technical documentation with code snippets. If your knowledge base mixes prose and code, don’t skip the keyword index.

What to Watch Out For

No architecture is free. Here’s what Raj had to trade off:

Architectural complexity — One service became five. More moving parts means more monitoring, more deployment coordination, and more things that can break. If your team is small, start with a simpler setup and add agents one at a time. Don’t build the full system on day one.

Cold start latency — The router and reranker add about 100ms to simple queries. For most use cases, that’s invisible. But if you’re building a real-time system where every millisecond counts, you’ll want to pre-warm your models and cache common queries.

Infra cost — Running the reranker adds compute cost. But here’s the trade-off that surprised Raj: the reranker let him use a smaller, cheaper LLM for generation because the context was cleaner. The net result was a 57% cost reduction. Sometimes spending a little more in one place saves a lot in another.

Production pitfall: Don’t optimize for cost or latency in isolation. Measure end-to-end. A reranker adds cost at one stage but reduces it at another. A router adds latency at the start but saves more than it costs. Always measure the full pipeline.

What surprised Raj:

• The reranker was the single highest-impact change. Not a better LLM, not a bigger index — just ordering results better.
• The citation validator caught hallucinated citations in ~12% of generations, even with GPT-4. Always validate.
• Hybrid search (dense + sparse) significantly outperformed pure vector search for technical documentation with code snippets.

Beginner-friendly advice: If you’re just getting started with RAG, don’t try to build all five agents at once. Start with a simple pipeline — embed, search, generate. Add the reranker next — it’s the highest-impact single change. Then add the query router. Then add citation validation. Each step makes the system better, and you can stop whenever the quality is good enough for your use case.

Raj’s system now handles over 10,000 queries daily with sub-2-second response times and 96% citation accuracy. His users trust the answers. His manager is happy. And his GPT-4 bill? It’s less than half of what it was.

The demo worked. But the production system stays working.