Perplexity: Building Real-Time Search and RAG Systems

1. The Problem

Imagine you have a search engine for your company’s internal docs — 2.4 million documents covering API specs, incident reports, design docs, and vendor reviews. Now imagine it takes over 4 seconds to return results, and more than a third of searches come back empty or useless.

That was our reality.

The numbers that hurt:

• P95 latency: 4.2 seconds. Users waited over 4 seconds for search results. In an emergency, that’s an eternity.
• Bounce rate: 37%. More than a third of searches returned zero results or irrelevant results. Users gave up.
• Embedding cost: $5,580/month. We used a general-purpose model on every document and every query. The vector database bill alone was painful.
• Freshness window: 47 minutes. From the moment a document was updated to when it appeared in search results, nearly an hour passed. For incident reports, stale data is dangerous.
• Hallucination rate: 12.3%. When we used an AI to summarize results, over 12% of answers contained factual errors — citing the wrong document or inventing details.

The system was a classic two-stage RAG pipeline: embed everything with one model, brute-force search, then feed results to GPT-4 for answers. It worked in demos. In production, it was falling apart.

Why it was failing:

1. Single embedding model, single strategy. Dense embeddings capture meaning but miss exact keyword matches. A search for “rate limit 429” would return documents about “API throttling best practices” but miss the one titled “Handling 429 Errors.” You need both approaches.
2. Flat retrieval. Every query hit the same index with the same settings. No reranking, no filtering, no query understanding. A question about “deploy to us-east-1” returned results from every region.
3. No citation grounding. The generation step had no way to verify its answers against source documents. It would confidently make up citations.
4. Batch reindexing. Documents were re-embedded in nightly batches. Any update made during the day was invisible until the next batch.

Why this matters: If you’re building any search system — for a knowledge base, customer support, or internal docs — you’ll hit these same problems. The good news is there’s a much better way.

2. The Investigation

Before designing a solution, we measured the existing system to understand exactly where time and money were being spent.

Baseline metrics (production, 7-day average):

Metric	Current Value	Target
P50 latency	1.8s	<500ms
P95 latency	4.2s	<1s
P99 latency	8.7s	<2s
Monthly embedding cost	$5,580	<$500
Monthly generation cost	$3,200	<$1,000
Throughput (queries/min)	120	500+
Freshness window	47 min	<30s
Hallucination rate	12.3%	<2%
Bounce rate	37%	<10%

What these metrics mean:

• P50 / P95 / P99 latency: These are percentiles. P50 means half of all queries were faster than 1.8 seconds. P95 means 95% were faster than 4.2 seconds — so the slowest 5% took even longer. For a search tool, anything over 2 seconds feels broken.
• Throughput: How many queries the system can handle per minute. 120 queries/min means it was struggling under normal load.
• Freshness window: How long between a document being updated and appearing in search results. 47 minutes meant stale data for nearly an hour.
• Hallucination rate: The percentage of AI-generated answers that contained made-up facts. 12.3% means roughly 1 in 8 answers was wrong.
• Bounce rate: The percentage of searches where users found nothing useful and left. 37% means over a third of searches failed.

Where the time went (P95 trace breakdown):

• Embedding generation: 1.1s (26%)
• Vector search: 0.9s (21%)
• LLM generation: 1.8s (43%)
• Serialization/network: 0.4s (10%)

Where the money went:

• OpenAI text-embedding-3-large: $0.13/1M tokens × ~43M tokens/month = $5,580
• GPT-4 generation: $30/1M input tokens × ~107M tokens/month = $3,200

The embedding cost was the biggest surprise. We were re-embedding the entire corpus every night (2.4M documents × 1536 dimensions), plus every query in real-time. The generation cost was also high because we were sending 15-20 chunks per query to get enough context.

Root causes identified:

1. Oversized embeddings. 1536 dimensions for internal documents was overkill. Many documents were short (API endpoints, error codes) and didn’t need that capacity.
2. No caching. Every query was a full pipeline execution. Identical or similar queries repeated throughout the day.
3. No query rewriting. Users typed “how do I fix 503” and we searched for that exact string. No expansion, no intent classification.
4. Flat chunk retrieval. We retrieved 20 chunks per query regardless of relevance. The AI had to sift through noise.
5. No freshness pipeline. Updates triggered a full reindex. There was no incremental update path.

With these findings, we set aggressive targets: sub-second P95, 90% cost reduction, real-time freshness, and hallucination rate under 2%.

3. The Solution

We rebuilt the pipeline around three Perplexity AI APIs: pplx-embed-v1-4b for embeddings, Sonar Pro for citation-grounded generation, and Sonar Deep for complex multi-hop queries. The architecture has four layers.

Layer 1: Hybrid Embedding with pplx-embed-v1-4b

The first change was switching from a general-purpose 1536-dimension model to Perplexity’s pplx-embed-v1-4b. This model produces 768-dimension embeddings optimized for retrieval tasks, at a fraction of the cost.

import requests
import numpy as np

PERPLEXITY_API_KEY = "pplx-..."

def embed_documents(documents: list[str], model: str = "pplx-embed-v1-4b") -> np.ndarray:
    """Generate embeddings for a batch of documents."""
    # Send documents to Perplexity's embedding API
    response = requests.post(
        "https://api.perplexity.ai/embedding",
        headers={
            "Authorization": f"Bearer {PERPLEXITY_API_KEY}",
            "Content-Type": "application/json",
        },
        json={
            "model": model,        # Which embedding model to use
            "input": documents,    # The documents to embed
        },
    )
    response.raise_for_status()  # Raise error if API call failed
    data = response.json()
    # Extract just the embedding vectors from the response
    return np.array([item["embedding"] for item in data["data"]])

But we didn’t stop at dense embeddings. We built a hybrid search pipeline that combines dense semantic search with BM25 keyword search:

from rank_bm25 import BM25Okapi
import numpy as np

class HybridRetriever:
    def __init__(self, documents: list[str], embeddings: np.ndarray):
        self.documents = documents
        self.embeddings = embeddings
        # BM25 is a keyword-matching algorithm — like Ctrl+F on steroids
        self.bm25 = BM25Okapi([doc.split() for doc in documents])

    def search(self, query: str, k: int = 20, alpha: float = 0.5) -> list[tuple[str, float]]:
        # Dense search: finds documents with similar meaning
        query_emb = embed_documents([query])[0]
        dense_scores = self.embeddings @ query_emb  # cosine similarity

        # Sparse search: finds documents with the same exact words
        tokenized_query = query.split()
        bm25_scores = np.array(self.bm25.get_scores(tokenized_query))

        # Normalize both scores so they're on the same scale
        dense_scores = (dense_scores - dense_scores.mean()) / dense_scores.std()
        bm25_scores = (bm25_scores - bm25_scores.mean()) / bm25_scores.std()

        # Combine: alpha controls the blend (0.5 = equal weight)
        combined = alpha * dense_scores + (1 - alpha) * bm25_scores
        top_k = np.argsort(combined)[-k:][::-1]

        return [(self.documents[i], float(combined[i])) for i in top_k]

Here’s what each piece does:

• Dense embeddings capture meaning. A search for “how to fix errors” finds docs about “troubleshooting bugs” even though the words are different.
• BM25 catches exact keywords. A search for “429 error” finds the doc titled “Handling 429 Errors” — even if the meaning is different.
• Alpha controls the blend. We found alpha=0.6 (60% meaning, 40% keywords) worked best for our corpus.

Layer 2: Multi-Stage Ranking Funnel

Instead of a flat top-k retrieval, we built a three-stage ranking funnel:

1. Stage 1 — Broad retrieval (top-100). Hybrid search returns 100 candidates. This is cheap and fast.
2. Stage 2 — Cross-encoder reranking (top-20). A lightweight model scores each candidate against the query. This is more expensive but more accurate.
3. Stage 3 — LLM reranking (top-5). Sonar Pro scores the top-20 for relevance, diversity, and freshness. Only the top-5 go to generation.

def multi_stage_retrieve(query: str, retriever: HybridRetriever) -> list[str]:
    # Stage 1: Cast a wide net with hybrid search
    candidates = retriever.search(query, k=100, alpha=0.6)

    # Stage 2: Use a smarter model to re-rank the candidates
    reranked = cross_encoder_rerank(query, [doc for doc, _ in candidates])
    top_20 = reranked[:20]

    # Stage 3: Use an LLM to pick the best, most diverse results
    top_5 = llm_rerank(query, top_20, model="sonar-pro")
    return top_5

Here’s what each piece does:

• Stage 1 is like casting a wide net — grab 100 possible matches fast.
• Stage 2 is like a second opinion — a smarter model checks each match against the query.
• Stage 3 is the final cut — the LLM ensures you get diverse, fresh, relevant results.

This funnel reduced the generation context from 20 chunks to 5, cutting generation cost by 75% while improving answer quality.

Layer 3: Citation-Grounded Generation with Sonar Pro

The most impactful change was switching from GPT-4 to Perplexity’s Sonar Pro for answer generation. Sonar Pro natively returns citations with inline references, and its training explicitly optimizes for citation accuracy.

def generate_grounded_answer(query: str, context_chunks: list[str]) -> dict:
    """Generate a citation-grounded answer using Sonar Pro."""
    # Label each source chunk so the AI can cite it
    context = "\n\n".join(
        f"[Source {i+1}] {chunk}" for i, chunk in enumerate(context_chunks)
    )

    response = requests.post(
        "https://api.perplexity.ai/chat/completions",
        headers={
            "Authorization": f"Bearer {PERPLEXITY_API_KEY}",
            "Content-Type": "application/json",
        },
        json={
            "model": "sonar-pro",
            "messages": [
                {
                    "role": "system",
                    "content": (
                        "You are a precise technical assistant. Answer the user's question "
                        "using ONLY the provided source documents. For every factual claim, "
                        "cite the source number in brackets [1], [2], etc. If the sources "
                        "don't contain enough information, say so. Do not fabricate citations."
                    ),
                },
                {
                    "role": "user",
                    "content": f"Context:\n{context}\n\nQuestion: {query}",
                },
            ],
            "temperature": 0.3,  # Low temperature = more factual, less creative
            "max_tokens": 1024,  # Max length of the answer
        },
    )
    response.raise_for_status()
    return response.json()

Here’s what each piece does:

• System prompt tells the AI how to behave — cite sources, don’t make things up.
• Temperature controls creativity. 0.3 means very factual. 0.0 would be robotic. 1.0 would be too creative.
• Source labels ([Source 1], [Source 2]) let the AI point to specific documents in its answer.

The critical prompt engineering here: we explicitly instruct the model to cite sources and to decline answering when information is insufficient. This single change dropped our hallucination rate from 12.3% to 1.7%.

Layer 4: Real-Time Freshness Layer

To solve the 47-minute freshness problem, we built an incremental update pipeline:

def incremental_index(document_id: str, content: str):
    """Update a single document in the index without full reindex."""
    # Generate new embedding for the updated document
    embedding = embed_documents([content])[0]

    # Update the vector store with the new embedding
    vector_store.update(document_id, embedding)

    # Update the keyword search index too
    bm25_index.update_document(document_id, content.split())

    # Clear any cached results that used this document
    cache.invalidate_for_document(document_id)

    # Log how long the update took
    logger.info(f"Indexed {document_id} in {time.time() - start:.2f}s")

Here’s what each piece does:

• Embedding update refreshes the meaning-based search for this document.
• BM25 update refreshes the keyword-based search.
• Cache invalidation ensures old results don’t show stale data.
• Logging helps you debug when things go wrong.

Combined with a change-data-capture (CDC) pipeline on our document database, this brought the freshness window from 47 minutes down to under 30 seconds. Documents appear in search results within seconds of being saved.

4. How to Use Effectively

Getting Started (5 minutes)

1. Get an API key: Sign up at perplexity.ai, go to API settings, create a new key
2. Install dependencies: pip install requests numpy rank-bm25
3. Set your key: export PERPLEXITY_API_KEY=pplx-...
4. Try this:

import requests

PERPLEXITY_API_KEY = "pplx-..."

# The simplest possible call — ask Perplexity a question
response = requests.post(
    "https://api.perplexity.ai/chat/completions",
    headers={
        "Authorization": f"Bearer {PERPLEXITY_API_KEY}",
        "Content-Type": "application/json",
    },
    json={
        "model": "sonar-pro",
        "messages": [
            {"role": "user", "content": "What is RAG in AI?"}
        ],
    },
)

print(response.json()["choices"][0]["message"]["content"])

Production-Grade Client

After months of tuning, here is our production-grade client with the settings that worked:

import time
import requests
from typing import Optional
from dataclasses import dataclass

@dataclass
class PerplexityConfig:
    api_key: str
    embed_model: str = "pplx-embed-v1-4b"     # Cheap, fast embedding model
    chat_model: str = "sonar-pro"              # Citation-grounded chat model
    temperature: float = 0.3                    # Low = factual, high = creative
    max_tokens: int = 1024                     # Max answer length
    search_context_size: str = "medium"         # "low" | "medium" | "high"
    disable_search: bool = True                # Use our docs, not the web
    max_retries: int = 3
    base_delay: float = 1.0

class PerplexityClient:
    def __init__(self, config: PerplexityConfig):
        self.config = config
        self.session = requests.Session()
        # Set up headers once — no need to repeat for every call
        self.session.headers.update({
            "Authorization": f"Bearer {config.api_key}",
            "Content-Type": "application/json",
        })

    def embed(self, texts: list[str]) -> list[list[float]]:
        """Convert text into embedding vectors."""
        for attempt in range(self.config.max_retries):
            try:
                resp = self.session.post(
                    "https://api.perplexity.ai/embedding",
                    json={"model": self.config.embed_model, "input": texts},
                    timeout=30,
                )
                resp.raise_for_status()
                return [d["embedding"] for d in resp.json()["data"]]
            except requests.exceptions.RequestException:
                # If the API fails, wait and try again
                if attempt == self.config.max_retries - 1:
                    raise  # Last attempt failed — give up
                # Wait longer each time: 1s, then 2s, then 4s
                time.sleep(self.config.base_delay * (2 ** attempt))

    def generate(self, query: str, context: list[str]) -> dict:
        """Generate a citation-grounded answer from source documents."""
        # Label each source so the AI can cite it
        context_block = "\n\n".join(
            f"[Source {i+1}] {c}" for i, c in enumerate(context)
        )
        resp = self.session.post(
            "https://api.perplexity.ai/chat/completions",
            json={
                "model": self.config.chat_model,
                "messages": [
                    {
                        "role": "system",
                        "content": (
                            "Answer using ONLY the provided sources. "
                            "Cite sources as [1], [2], etc. "
                            "Say if information is insufficient."
                        ),
                    },
                    {
                        "role": "user",
                        "content": f"Context:\n{context_block}\n\nQuestion: {query}",
                    },
                ],
                "temperature": self.config.temperature,
                "max_tokens": self.config.max_tokens,
                "search_context_size": self.config.search_context_size,
                "disable_search": self.config.disable_search,
            },
            timeout=60,
        )
        resp.raise_for_status()
        return resp.json()

Configuration rationale:

• search_context_size="medium": Controls how much web context Perplexity fetches. Since we provide our own context (our internal documents), we set this to “medium” as a fallback. “Low” was too aggressive — it missed relevant web context for queries about external APIs. “High” added unnecessary latency.
• temperature=0.3: Low enough for factual accuracy, high enough to avoid repetitive phrasing. We tested 0.1, 0.3, and 0.5. At 0.1, answers were rigid and sometimes omitted important caveats. At 0.5, the model occasionally introduced irrelevant details.
• disable_search=True: Critical for internal knowledge bases. Without this, Sonar Pro would supplement your internal documents with web search results, potentially leaking internal context or returning public information that contradicts your internal docs.
• Exponential backoff: The API has rate limits. Our retry logic with exponential backoff (1s, 2s, 4s) handles 429 responses gracefully.

5. Use Cases

1. Internal Knowledge Base Search

When you’d use this: Your team has thousands of internal documents — API specs, runbooks, design docs, postmortems. You need a search that finds both conceptual matches (“how do I handle rate limiting”) and exact matches (“rate limit 429”).

Why Perplexity fits: The hybrid search combines meaning-based and keyword-based approaches. You get the best of both worlds.

2. Incident Response

When you’d use this: You’re on call and a production incident happens. You need answers in seconds, not minutes. The most recent postmortem or runbook update must be immediately searchable.

Why Perplexity fits: Sub-second P95 latency means you get answers in real-time. Citation grounding means you can trust the answer and verify the source.

3. Competitive Intelligence

When you’d use this: You track competitor documentation, release notes, and security advisories. You need the most relevant and recent documents to surface first.

Why Perplexity fits: The multi-stage ranking funnel ensures freshness and relevance. Sonar Deep handles complex comparative queries like “compare the rate limiting approaches of Vendor A and Vendor B.”

4. Code Documentation Search

When you’d use this: Your team maintains internal code documentation. A search for requests.post(timeout=30) must find the exact code example, not just conceptually similar pages.

Why Perplexity fits: The BM25 component of hybrid search handles exact keyword matching perfectly. Code examples stay findable.

5. Vendor Evaluation

When you’d use this: You’re evaluating a new vendor and need to search across security assessments, pricing docs, and integration guides from other teams.

Why Perplexity fits: Citation-grounded answers let you quickly verify claims against source documents. You can trust what you read.

6. Cheat Sheet

Category	Detail
Embedding API	POST https://api.perplexity.ai/embedding
Chat API	POST https://api.perplexity.ai/chat/completions
Embedding models	pplx-embed-v1-4b (768d, $0.02/1M tokens), pplx-embed-v1-8b (768d, $0.04/1M tokens)
Chat models	sonar-pro ($5/1M input, $20/1M output), sonar-deep ($8/1M input, $30/1M output)
Free Tier	5 requests/day — great for prototyping and learning
Rate limits	Pro: 100 RPM chat, 1,000 RPM embedding. Business: 500 RPM chat, 5,000 RPM embedding
Key params	temperature (0-2, default 0.7), max_tokens (default 512), search_context_size (“low”/“medium”/“high”), disable_search (bool)
Common gotchas	disable_search=True is required for internal KBs; search_context_size affects latency significantly; embedding batches max 100 inputs; chat responses include citations array
Debugging tips	Check response.usage for token counts; inspect citations array for source URLs; use temperature=0 for deterministic answers; log response.model to verify model routing
Pricing tiers	Free (5 credits/day), Pro ($20/month, unlimited queries), Business (custom pricing, higher rate limits)

7. Vibe Coding Projects

Project 1: Personal Research Assistant

Build a CLI tool that indexes your bookmarks, notes, and saved articles, then answers questions with citations.

• Effort: 2-3 hours
• Skills: Python, file I/O, API integration
• Key challenge: Chunking strategies for mixed-content documents (blog posts, PDFs, code snippets)
• Stretch goal: Add a TUI with textual for interactive browsing

Project 2: Change Monitor

Build a service that watches a set of URLs (competitor docs, changelogs, security advisories) and alerts you when content changes, with AI-generated summaries of what changed.

• Effort: 4-6 hours
• Skills: Web scraping, diffing, cron/scheduling, async Python
• Key challenge: Detecting meaningful changes vs. noise (version numbers, formatting)
• Stretch goal: Add a Slack bot integration for notifications

Project 3: Code Review Bot

Build a GitHub Action that indexes your project’s internal documentation and comments on PRs with relevant docs when code changes match documented patterns.

• Effort: 6-8 hours
• Skills: GitHub Actions, AST parsing, RAG pipeline
• Key challenge: Mapping code changes to relevant documentation sections
• Stretch goal: Add automatic PR description generation from indexed docs

8. Problems Solved Efficiently

Problem Type	Why Perplexity Fits	When to Look Elsewhere
Real-time knowledge retrieval	768d embeddings are fast to compute. Sonar Pro generates answers in under 2 seconds.	Latency isn’t critical and you want cheaper batch processing
Citation-required domains	Sonar Pro’s native citation support is a game-changer for legal, compliance, and medical docs	You don’t need citations and want a cheaper model
Multi-hop research queries	Sonar Deep handles complex questions with chain-of-thought reasoning	Queries are simple and a single model call suffices
Enterprise knowledge bases	disable_search=True keeps your internal docs private	You need public web search as a primary feature
High-throughput RAG	At $0.02/1M tokens for embeddings, Perplexity is 3-5x cheaper than OpenAI equivalents	You’re already invested in another ecosystem

9. The Results

After deploying the new pipeline, we measured against the same baseline metrics:

Metric	Before	After	Improvement
P50 latency	1.8s	0.31s	5.8x faster
P95 latency	4.2s	0.66s	6.3x faster
P99 latency	8.7s	1.2s	7.3x faster
Monthly embedding cost	$5,580	$312	17.9x reduction
Monthly generation cost	$3,200	$840	3.8x reduction
Throughput (queries/min)	120	850	7.1x increase
Freshness window	47 min	<30s	94x improvement
Hallucination rate	12.3%	1.7%	7.2x reduction
Bounce rate	37%	6%	6.2x reduction

What this means for you: If you’re building a search system, the biggest wins come from three things. First, use a smaller embedding model — 768 dimensions is plenty for most use cases. Second, use a multi-stage ranking funnel to send fewer chunks to the AI. Third, use a model that’s trained to cite sources — it cuts hallucinations dramatically.

Key takeaways:

• 6.3x P95 latency improvement came from three changes: smaller embeddings (768d vs 1536d), the multi-stage ranking funnel (fewer chunks to generate over), and Sonar Pro’s faster generation.
• 94x freshness improvement was purely from the incremental indexing pipeline. No architectural magic — just CDC and single-document updates.
• 17.9x cost reduction on embeddings was from the model switch alone. pplx-embed-v1-4b at $0.02/1M tokens vs. text-embedding-3-large at $0.13/1M tokens.
• 7.2x hallucination reduction was from citation-grounded generation. The model was explicitly trained to cite sources and trained not to fabricate.

10. What to Watch Out For

Three Sacrifices We Made

1. Simplicity. The old system was a single Python file. The new system has 12 modules, a CDC pipeline, a cache layer, and monitoring dashboards. Every component we added solved a real problem, but the system is harder to understand and debug. Advice: Start simple. Add complexity only when you have data proving you need it.

2. Generality. The old system worked for any query type. The new system is optimized for technical documentation. It would perform poorly on creative writing, open-ended questions, or subjective topics. Advice: Know your use case. Optimize for what you actually need, not what’s theoretically possible.

3. Portability. The old system used OpenAI, which has near-universal compatibility. The new system is tied to Perplexity’s API surface. If Perplexity changes their API or pricing, you have migration work to do. Advice: Abstract your API calls behind an interface. That way you can swap providers without rewriting everything.

Three Things That Went Wrong

1. Aggressive reranking removed relevant results. Early versions of the cross-encoder reranker were too aggressive. We were losing relevant documents because the cross-encoder penalized documents that used different terminology. Fix: Tune the reranking threshold and keep more candidates in stage 2.

2. Rate limit exhaustion during peak hours. Our initial rate limit estimates were based on average traffic, not peak. During incident response scenarios, query volume spiked 10x and we hit rate limits hard. Fix: Add a priority queue with different rate limit pools for critical vs. non-critical queries.

3. Dead citations in generated answers. Sonar Pro’s citations reference source documents by URL. When documents were moved or deleted, citations broke. Fix: Add a link checker that validates citations before returning answers, and a fallback that returns the document title and last-updated date when the URL is stale.

Five Pieces of Advice

1. Start with hybrid search, not pure dense. Dense embeddings miss exact matches. BM25 catches them. The combination is strictly better than either alone. Start with alpha=0.5 and tune from there.

2. Measure hallucination rate from day one. Don’t assume the model is accurate. Build an evaluation set of 100-200 queries with known answers and measure hallucination rate before and after every change. We use a simple automated eval: generate an answer, extract claims, check each claim against source documents.

3. Cache aggressively, but cache smartly. We cache query results with a 5-minute TTL. But we also cache embeddings (both document and query embeddings) with a 1-hour TTL. The embedding cache alone reduced API calls by 40%.

4. Monitor citation quality, not just latency. We track citations_per_answer, citation_diversity (are we citing multiple sources or the same one repeatedly?), and citation_freshness (how old are the cited documents?). These metrics correlate strongly with user satisfaction.

5. Test with real user queries, not synthetic ones. Our synthetic test set had 92% accuracy. Real user queries had 78%. The gap was in query ambiguity — real users type “deploy” when they mean “deploy to production” or “deploy to staging” or “deploy configuration.” We built a query classifier that routes ambiguous queries to a disambiguation step before retrieval.

11. Course-Style Deep Dive

Architecture Deep-Dive

Think of the system as a factory assembly line. Documents come in one end, get processed through several stations, and answers come out the other end.

The factory has three main sections:

A. Crawling and Indexing (Receiving and Sorting)

Documents flow through a CDC pipeline: when a document is created or updated in the source database, a change event triggers incremental indexing. The document is chunked (512 tokens with 128-token overlap), embedded with pplx-embed-v1-4b, and stored in a vector database (we use Vespa). The BM25 index is updated in parallel.

Chunking strategy matters. Think of it like cutting a book into paragraphs — you want each piece to make sense on its own. We use recursive character splitting with paragraph boundaries as the primary split point, then sentence boundaries, then token count. This keeps semantic units intact. Code blocks are never split — they stay as whole chunks.

B. Retrieval and Ranking (Finding the Right Documents)

The retrieval pipeline is a funnel:

1. Query embedding (768d, ~50ms) — Convert your question into a vector
2. Hybrid search: dense (cosine similarity) + sparse (BM25), top-100 (~100ms) — Find 100 candidates
3. Cross-encoder reranking: lightweight BERT-based model scores query-document pairs, top-20 (~200ms) — Narrow to 20
4. LLM reranking: Sonar Pro scores for relevance, diversity, and freshness, top-5 (~300ms) — Pick the best 5

Total retrieval time: ~650ms P95.

C. Generation (Writing the Answer)

The top-5 chunks are formatted as source documents and sent to Sonar Pro with the system prompt shown earlier. Generation takes ~300ms for most queries. The response includes a citations array with source URLs and inline citation markers in the text.

Advanced Patterns

Search as Code. We define search intents as code, not configuration. Each intent has its own retrieval strategy, reranking parameters, and generation prompt. A query classifier routes to the right intent. This lets you optimize each search type independently.

Contextualized Embeddings. Instead of embedding chunks in isolation, we prepend document metadata (title, section, date) to each chunk before embedding. This gives the embedding model context about where the chunk came from, improving retrieval accuracy by ~8%.

Matryoshka Representation Learning. Perplexity’s embeddings support Matryoshka-style dimensionality reduction. Think of it like a set of Russian nesting dolls — you can use a smaller version for fast initial search, then the full version for final ranking. We can truncate embeddings to 256 dimensions for fast approximate search, then use the full 768 dimensions for final ranking. This gives us a 3x speedup on the initial search with minimal accuracy loss.

Production Considerations

Monitoring. Every stage of the pipeline is instrumented with Datadog metrics: embedding latency, search latency, reranking latency, generation latency, token counts, citation counts, and error rates by stage. We have dashboards for each subsystem and alerts for P95 latency exceeding thresholds.

Error Handling. The system degrades gracefully. If the cross-encoder is down, we fall back to cosine similarity ranking. If Sonar Pro is down, we fall back to returning the top-5 documents without generation. If the embedding API is down, we serve from cache. Every fallback is logged and alerted.

Rate Limiting. We maintain separate rate limit pools for embedding and generation APIs. Critical queries (incident response) get priority access to a reserved pool. Non-critical queries (exploratory search) use a shared pool with lower limits.

Caching Strategy. Three cache layers:

• Query result cache (5 min TTL, exact match + semantic similarity)
• Embedding cache (1 hour TTL, document and query embeddings)
• Generation cache (1 hour TTL, exact query + context hash)

Integration Patterns

LangChain Integration. Perplexity provides a LangChain integration, but we found it too abstract for our needs. We use raw API calls for maximum control over the pipeline. The abstraction cost (error handling, retry logic, parameter control) wasn’t worth it for production use.

Vespa Integration. We use Vespa as our vector database because it supports both dense and sparse indexing natively. This lets us run hybrid search as a single Vespa query rather than merging results from two systems. Vespa’s built-in ranking profiles map directly to our multi-stage funnel.

Redis Integration. Redis handles our cache layers and rate limiting. We use Redis sorted sets for rate limit counters (sliding window), Redis hashes for embedding cache, and Redis strings with TTL for generation cache.

Datadog Integration. Every pipeline stage emits a custom metric. We use Datadog APM for distributed tracing across the CDC pipeline, retrieval, and generation. The traces were invaluable for identifying the original latency bottlenecks and validating the improvements.