Danswer: An enterprise AI search and chat platform (MIT, 10k stars)

The Problem

Every organization runs on internal knowledge. Documentation lives in Confluence. Code discussions happen in Slack. Product specs are in Notion. Design files are in Google Drive. Engineering tickets are in Jira. And somewhere in that sprawl is the answer to whatever question your team just asked in a meeting.

The default workflow is a multi-tab scavenger hunt: search Confluence, search Slack, search Google Drive, search Jira, ask three people on Slack, wait for replies, piece together the answer. This costs 30-60 minutes per query and assumes the person asking even knows where to look. New hires spend their first month learning the geography of the company’s knowledge — which tool has what, who knows what, which channel to ask in.

Enterprise search tools exist, but they come with enterprise price tags. Glean charges $50+/user/month. Sinequa and Coveo are in the six-figure annual range. Elastic requires a dedicated engineering team to configure and maintain. For a 50-person company, that is $30,000+/year before you get a single answer. For a startup or mid-market team, that is prohibitive.

Dimension	Manual Search	Glean	Sinequa	Danswer (Onyx)
Cost per user/month	$0 (time cost: 30-60 min/query)	$50+	Enterprise (6-figure annual)	$0 (self-host) / $20 (cloud)
Setup time	N/A	2-4 weeks (managed)	3-6 months (consulting)	30 minutes (Docker Compose)
Self-host option	N/A	No	Yes (on-prem)	Yes (MIT, self-host)
Permission mirroring	N/A	Yes	Yes	Yes
Connectors	N/A	100+	200+	40+
LLM agnostic	N/A	No (proprietary)	No (proprietary)	Yes (any LLM)
Open source	N/A	No	No	Yes (MIT)
Engineering required	N/A	None	Dedicated team	Moderate (DevOps)

Why this matters: The enterprise search market has been bifurcated between expensive managed solutions (Glean, Sinequa) and infrastructure tools that require engineering teams to build on top of (Elastic). Danswer fills the gap: an open-source, self-hostable, permission-aware RAG platform that any team with basic Docker skills can deploy in an afternoon. It does not match Glean’s 100+ connectors or Sinequa’s enterprise compliance, but it covers the 80% use case at 0% of the cost.

The Investigation

Danswer was created by Chris Weaver and Yuhong Sun, who experienced the knowledge fragmentation problem firsthand at a fast-growing startup. They spent 2023 building a solution that connects LLMs to internal knowledge sources with proper access controls. The project hit 10,000 GitHub stars within its first year, was acquired by Onyx in 2024, and continues as the open-source Onyx project under MIT license.

Finding 1: Permission-aware RAG is the hard problem, not RAG itself.

Basic RAG — chunk documents, embed them, retrieve top-k, feed to an LLM — is a solved problem. LangChain, LlamaIndex, and a dozen other frameworks can do it in 50 lines of code. The hard problem is access control: if Alice can see the Confluence page about Q4 strategy but Bob cannot, the search system must respect that.

Danswer’s investigation found that most RAG implementations treat all documents as equally accessible. This is fine for public-facing chatbots but catastrophic for enterprise search. A single leaked document — a compensation spreadsheet, a legal memo, an unreleased product spec — creates a compliance incident.

The solution is permission mirroring: the indexing pipeline captures access control lists (ACLs) from each source system and stores them alongside the document chunks. At query time, the user’s identity is resolved against the ACLs before any chunk is returned. This means the search system is only as permissive as the most permissive source system — if a user cannot see a document in Confluence, they cannot retrieve it through Danswer either.

What this means: Permission-aware RAG is not a feature you bolt on after building the search system. It must be designed into the indexing pipeline from day one. Danswer’s architecture treats permissions as first-class metadata, not an afterthought.

Finding 2: Hybrid search beats pure vector search for enterprise documents.

Enterprise documents are not like web pages. They contain structured metadata (titles, headings, authors, dates), domain-specific terminology (product names, internal acronyms, project codenames), and exact-match queries (“Q4 2025 revenue”, “PRD-342”, “bug in the checkout flow”). Pure vector search struggles with exact matches and rare terms. Pure BM25 keyword search misses semantic relationships.

Danswer’s benchmark results on internal enterprise queries showed that hybrid search (BM25 + vector) outperforms either approach alone by 15-25 percentage points on recall@10. The optimal blend varies by query type: exact-match queries favor BM25, while conceptual questions (“what’s our approach to microservices?”) favor vector search.

What this means: Enterprise search needs both keyword and semantic retrieval. Danswer’s Vespa backend supports hybrid search with a configurable alpha parameter that controls the BM25-to-vector ratio. The default of 0.5 works well for most queries, but power users can tune it per connector or per query.

Finding 3: Connector quality matters more than connector count.

Danswer ships with 40+ connectors. Glean has 100+. On paper, Glean wins. In practice, the quality of each connector — how well it handles pagination, rate limiting, incremental sync, permission extraction, and error recovery — matters far more than the raw count.

Danswer’s connector architecture uses a checkpointed, resumable design. Each connector saves its state (pagination cursors, last-modified timestamps, processed document IDs) to MinIO. If a sync fails halfway through — the source API rate-limits the connector, the network drops, the indexing worker crashes — the next sync resumes from the last checkpoint, not from scratch. This is critical for enterprise connectors that index millions of documents.

What this means: A connector that handles 40 sources reliably is more valuable than a connector that handles 100 sources but fails on the 50,000th document and restarts from zero. Danswer’s checkpointed design is production-grade where many alternatives are not.

The Solution

Danswer (now Onyx) is a ~100,000-line Python/TypeScript application (MIT license, 10,000+ GitHub stars) that connects LLMs to your company’s internal knowledge sources. It runs as a set of Docker containers: a FastAPI backend, a Next.js frontend, Celery workers for indexing, Vespa for hybrid search, PostgreSQL for metadata, Redis for task queues, and MinIO for file storage.

┌──────────────────────────────────────────────────────────────────────────┐
│                         Danswer (Onyx) Architecture                         │
│                                                                             │
│  ┌──────────────┐    ┌──────────────────┐    ┌─────────────────────────┐   │
│  │  Connectors   │    │  Indexing Pipeline│    │  Search & Chat Layer  │   │
│  │  (40+ sources) │    │  (Celery workers) │    │  (FastAPI + Next.js)  │   │
│  │               │    │                   │    │                        │   │
│  │  • Slack      │───▶│  • Doc fetching  │───▶│  • Hybrid search      │   │
│  │  • Confluence │    │  • Chunking      │    │    (Vespa BM25+vec)   │   │
│  │  • Google Drv │    │  • Embedding     │    │  • Permission filter  │   │
│  │  • GitHub     │    │  • ACL capture   │    │  • LLM chat (any API) │   │
│  │  • Notion     │    │  • Vespa index   │    │  • Source citations   │   │
│  │  • Jira       │    │  • Checkpoint    │    │  • Custom assistants  │   │
│  │  • ...        │    │                   │    │                        │   │
│  └──────┬───────┘    └──────┬───────────┘    └──────────┬─────────────┘   │
│         │                   │                            │                 │
│         │         ┌─────────┴────────────────────────────┴──────────┐      │
│         │         │              Storage Layer                       │      │
│         │         │  ┌──────────┐ ┌──────────┐ ┌────────────────┐  │      │
│         │         │  │ Vespa   │ │PostgreSQL│ │ MinIO (S3)     │  │      │
│         │         │  │(vectors │ │(metadata,│ │(raw files,     │  │      │
│         │         │  │ + BM25) │ │ ACLs)   │ │ checkpoints)   │  │      │
│         │         │  └──────────┘ └──────────┘ └────────────────┘  │      │
│         │         └─────────────────────────────────────────────────┘      │
│         │                                                                   │
│  ┌──────┴──────────────────────────────────────────────────────────────┐   │
│  │                    LLM Layer (model-agnostic)                        │   │
│  │  OpenAI │ Anthropic │ Google │ Cohere │ Voyage │ Ollama │ Custom    │   │
│  └─────────────────────────────────────────────────────────────────────┘   │
└──────────────────────────────────────────────────────────────────────────┘

Here is what each piece does:

Connectors (40+ sources): Pluggable modules that pull documents from external systems. Each connector implements a standard interface: poll() for batch ingestion, poll_events() for incremental sync, and retrieve_all() for full reindex. Connectors are checkpointed — they save their progress to MinIO so failures are resumable.
Indexing Pipeline (Celery workers): Two worker pools — doc-fetching workers (I/O-bound, pull documents from connectors) and doc-processing workers (CPU/GPU-bound, chunk and embed). The pipeline is asynchronous and horizontally scalable: add more workers to handle larger document volumes.
Search & Chat Layer: The FastAPI backend handles query routing, permission filtering, and LLM orchestration. The Next.js frontend provides a ChatGPT-like interface with source citations, document previews, and admin dashboards.
Storage Layer: Vespa stores vector embeddings and BM25 indices for hybrid search. PostgreSQL stores document metadata, access control lists, and user/workspace data. MinIO stores raw files and connector checkpoints.
LLM Layer: Model-agnostic. Configure any OpenAI-compatible API, Anthropic, Google Gemini, Cohere, Voyage, or a self-hosted model via Ollama. The same search pipeline works with any provider.

Setup

# Clone the repository
git clone --depth 1 https://github.com/onyx-dot-app/onyx.git
cd onyx/deployment/docker_compose

# Run the interactive installer
./install.sh

# Or deploy manually with Docker Compose
cp env.prod.template .env
cp env.nginx.template .env.nginx

# Edit .env with your settings
# At minimum: set AUTH_TYPE, POSTGRES_PASSWORD, OPENSEARCH_ADMIN_PASSWORD

# Start the stack
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d

# Access the web UI at https://your-domain.com
# Create an admin account and configure your first connector

Production-Grade Configuration

# .env — production configuration
AUTH_TYPE=google_oauth
OAUTH_CLIENT_ID=your-client-id.apps.googleusercontent.com
OAUTH_CLIENT_SECRET=your-client-secret
VALID_EMAIL_DOMAINS=yourcompany.com

POSTGRES_USER=postgres
POSTGRES_PASSWORD=$(openssl rand -base64 32)
POSTGRES_HOST=relational_db

OPENSEARCH_ADMIN_PASSWORD=$(openssl rand -base64 32)

FILE_STORE_BACKEND=s3
S3_ENDPOINT_URL=http://minio:9000
S3_AWS_ACCESS_KEY_ID=minioadmin
S3_AWS_SECRET_ACCESS_KEY=$(openssl rand -base64 32)

# LLM configuration (configured in Admin UI after first login)
# GEN_AI_MODEL_PROVIDER=openai
# GEN_AI_MODEL_VERSION=gpt-4o

Code Walkthrough: The Connector Interface

The heart of Danswer’s data ingestion is the connector interface in backend/onyx/connectors/models.py. Every connector implements this contract:

# Simplified from backend/onyx/connectors/models.py
class Connector:
    """Base interface for all Danswer connectors."""

    def poll(
        self, start: datetime, end: datetime
    ) -> Generator[Document, None, None]:
        """Yield documents modified in the given time range.
        Called by the indexing pipeline on a schedule."""
        raise NotImplementedError

    def poll_events(
        self, start: datetime, end: datetime
    ) -> Generator[Document, None, None]:
        """Yield documents from event-based sources (webhooks, streams).
        Falls back to poll() if not implemented."""
        return self.poll(start, end)

    def retrieve_all(
        self, start: datetime, end: datetime
    ) -> Generator[Document, None, None]:
        """Full reindex — yield all documents regardless of modification time.
        Used when a connector is first configured or after a schema change."""
        return self.poll(start, end)

Each connector also defines its credential and configuration schemas:

# Simplified from backend/onyx/connectors/slack/connector.py
class SlackConnector(Connector):
    def __init__(self, credential: Credential, config: dict):
        self.client = SlackClient(token=credential["bot_token"])
        self.channels = config.get("channels", [])
        self.include_threads = config.get("include_threads", True)

    def poll(self, start, end):
        """Fetch messages from configured Slack channels."""
        for channel in self.channels:
            cursor = None
            while True:
                result = self.client.conversations_history(
                    channel=channel,
                    oldest=start.timestamp(),
                    latest=end.timestamp(),
                    cursor=cursor,
                    limit=100,
                )
                for message in result["messages"]:
                    yield Document(
                        id=f"slack-{channel}-{message['ts']}",
                        source="slack",
                        content=message["text"],
                        metadata={
                            "channel": channel,
                            "user": message["user"],
                            "timestamp": message["ts"],
                            "permalink": self._get_permalink(channel, message["ts"]),
                        },
                        # Permission: only users in this channel can see this message
                        access_control_list=[f"slack_channel:{channel}"],
                    )
                cursor = result.get("response_metadata", {}).get("next_cursor")
                if not cursor:
                    break

The indexing pipeline orchestrates connector execution with checkpointing:

# Simplified from backend/onyx/indexing/indexing_pipeline.py
class IndexingPipeline:
    def run_connector(self, connector_id: str, credential_id: str):
        """Run a connector and index its documents."""
        connector = self.connector_factory.get_connector(connector_id, credential_id)
        checkpoint = self.checkpoint_store.get(connector_id)

        # Resume from last checkpoint if available
        start = checkpoint.last_successful_time if checkpoint else datetime.now() - timedelta(days=7)
        end = datetime.now()

        documents = connector.poll(start, end)
        for batch in self._batch(documents, size=100):
            # 1. Chunk each document
            chunks = self.chunker.chunk(batch)

            # 2. Generate embeddings
            embeddings = self.embedder.embed(chunks)

            # 3. Index into Vespa
            self.vespa_index.index(chunks, embeddings)

            # 4. Store metadata in PostgreSQL
            self.metadata_store.upsert(batch)

            # 5. Save checkpoint
            self.checkpoint_store.save(connector_id, end)

How to Use Effectively

Step 1: Deploy the stack

# Minimum viable deployment (4 GB RAM, 16 GB disk)
docker compose -f docker-compose.yml -f docker-compose.onyx-lite.yml up -d

# Full deployment with Vespa and model servers (10 GB RAM, 32 GB disk)
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d

The Lite mode removes Vespa, Redis, model servers, and MinIO — it uses PostgreSQL for everything and relies on an external LLM API. Use it for evaluation. Use the full deployment for production.

Step 2: Configure connectors

After the first login, navigate to the Admin Dashboard and add connectors:

Slack: Create a Slack app with channels:history and channels:read scopes. Install it to your workspace. Paste the bot token into Danswer.
Confluence: Generate an API token from your Atlassian account. Provide the wiki base URL and space key.
Google Drive: Set up a Google Cloud service account with domain-wide delegation. Share the Drive folders you want indexed with the service account email.
GitHub: Create a GitHub App or personal access token with repo scope. Select the repositories to index.

Each connector has a refresh_freq setting (in seconds). Set it to 3600 (hourly) for Slack, 86400 (daily) for Confluence and Google Drive. Higher-frequency connectors consume more API quota and indexing resources.

Step 3: Configure the LLM

Danswer supports any OpenAI-compatible API. Configure it in the Admin Dashboard under “LLM Settings”:

Provider: OpenAI, Anthropic, Google, Cohere, or Custom
Model: gpt-4o, claude-sonnet-4-20250514, gemini-2.0-flash, or any custom model name
Temperature: 0.1 for factual Q&A, 0.7 for creative tasks
Max tokens: 4096 (default)

For self-hosted setups, configure Ollama or a vLLM endpoint as the provider.

Step 4: Create custom assistants

Danswer lets you create specialized AI assistants with different prompts and knowledge sets:

Go to “Assistants” in the sidebar.
Click “Create Assistant.”
Set the system prompt: “You are a support engineer answering questions about our product documentation. Always cite the source document and version.”
Select which connectors this assistant can access: “Only the Confluence and Zendesk connectors.”
Save and share the assistant with your team.

This is useful for creating role-specific search experiences: a “Customer Support” assistant that only searches the knowledge base and ticket system, an “Engineering” assistant that only searches GitHub and internal docs, and an “Executive” assistant that searches everything.

Step 5: Use the Slack integration

Danswer’s Slack bot lets users ask questions without leaving Slack:

/onyx What's the current process for onboarding new engineers?

The bot responds with a grounded answer and source citations. Users can follow up with clarifying questions in the same thread. This is the highest-ROI integration — it puts enterprise search where the conversations already happen.

Production pitfall: The Slack bot uses the same permission model as the web UI. If a user asks a question in a public channel, the bot answers based on that user’s permissions, not the channel’s. A junior engineer asking in #general will not see documents they lack access to, even if the channel is public.

Use Cases

1. New Hire Onboarding

When you’d use this: A new engineer joins the team and needs to find onboarding documentation, architecture decisions, and team processes.

Why Danswer fits: New hires spend their first weeks learning where information lives. Danswer eliminates the geography problem — they ask one question and get answers from across all connected sources. The permission model ensures they only see what they should, and the source citations teach them which tools contain which types of information.

2. Engineering Knowledge Retrieval

When you’d use this: An engineer needs to find a past PR discussion about a specific API design decision, or the architecture decision record for a service they are modifying.

Why Danswer fits: Connect GitHub (PRs, issues, discussions), Confluence (ADRs, design docs), and Slack (technical discussions). A single query like “why did we choose PostgreSQL over MySQL for the analytics service?” searches all three sources and returns the relevant PR, the ADR, and the Slack thread where the decision was debated.

3. Customer Support Resolution

When you’d use this: A support agent needs to find the answer to a customer’s question across the knowledge base, product documentation, and past ticket resolutions.

Why Danswer fits: Connect Zendesk, Confluence, and Google Drive. Create a “Support” assistant with a system prompt that instructs the LLM to prioritize the knowledge base and cite version numbers. The agent gets grounded answers with source links, reducing average handle time by 30-50%.

4. Compliance and Audit Queries

When you’d use this: An auditor or compliance officer needs to find all documents related to a specific policy, process, or control.

Why Danswer fits: The permission model ensures auditors only see what they are authorized to see. The source citations provide an audit trail — every answer links back to the original document. The admin dashboard shows which connectors are indexed, when they were last synced, and how many documents are in the index.

When you’d use this: A product manager needs to understand the engineering team’s technical constraints, or a designer needs to find the product requirements for a feature they are designing.

Why Danswer fits: Danswer breaks down silos by making all connected knowledge searchable from one interface. The permission model ensures cross-team access is controlled by the source systems — if the engineering team’s Confluence space is visible to the whole company, Danswer respects that. If it is restricted, Danswer respects that too.

Cheat Sheet

Aspect	Detail
Repository	github.com/onyx-dot-app/onyx
License	MIT (Community Edition) / Proprietary (Enterprise Edition)
Language	Python (FastAPI backend) + TypeScript (Next.js frontend)
GPU Requirements	Optional (for self-hosted embedding models); API-based mode needs none
Setup Time	30 minutes (Docker Compose)
Key Features	40+ connectors, permission mirroring, hybrid search (Vespa), custom assistants, Slack bot, model-agnostic LLM support, incremental sync, checkpointed indexing
Common Gotchas	Under-provisioning RAM (need 10 GB for full stack); forgetting to set `VALID_EMAIL_DOMAINS`; not configuring SSL in production; using Lite mode for production workloads
Best LLMs	GPT-4o, Claude Sonnet 4, Gemini 2.0 Flash, any OpenAI-compatible API
Cost (Self-Host)	Server costs only (~$50-200/mo on a cloud VM)
Cost (Cloud)	$20/user/month (managed Onyx Cloud)
Missing Features	No proactive knowledge surfacing; no persistent organizational memory; no native Teams integration; fewer connectors than Glean (40 vs 100+)

Vibe Coding Projects

Project 1: Internal Documentation Q&A Bot

What it does: Deploy Danswer connected to your company’s Confluence, Google Drive, and Slack. Configure a single “Company Knowledge” assistant that searches all three sources. Set up the Slack bot so the team can ask questions without leaving Slack.

What you’ll learn: Docker Compose deployment, connector configuration, permission setup, Slack bot integration, and custom assistant creation. You will see the full lifecycle from deployment to daily usage.

Effort: 2-3 hours. Server costs only (~$50/mo for a cloud VM).

Project 2: Custom Connector for an Internal Tool

What it does: Your company uses a custom internal tool (a project tracker, a knowledge base, a wiki) that Danswer does not have a connector for. Write a custom connector that implements the Connector interface and registers it with Danswer’s connector factory.

What you’ll learn: The connector interface, credential management, checkpointing, document chunking, and permission extraction. You will understand the full data ingestion pipeline from source to search index.

Effort: 4-8 hours. Requires Python knowledge and access to the internal tool’s API.

Project 3: Multi-Tenant Knowledge Base for a Consulting Firm

What it does: A consulting firm with multiple clients needs a knowledge base where each client’s documents are isolated. Deploy Danswer in multi-tenant mode, configure separate connectors per client, and set up user groups with document-level permissions.

What you’ll learn: Multi-tenant deployment, workspace management, RBAC configuration, and permission mirroring at scale. You will understand how to design a knowledge management system that respects client data isolation.

Effort: 8-16 hours. Requires Kubernetes or advanced Docker Compose knowledge.

Problems Solved Efficiently

Problem Type	Why Danswer Fits	When to Look Elsewhere
Internal knowledge search	Permission-aware, 40+ connectors, self-hostable	Use Glean for 100+ connectors and managed service
New hire onboarding	Single search interface across all tools	Use Notion for structured onboarding docs
Cross-team knowledge sharing	Breaks down silos with controlled access	Use Confluence for curated documentation
Compliance document retrieval	Permission model + source citations + audit trail	Use Sinequa for regulated enterprise compliance
Customer support Q&A	Custom assistants + Slack bot + source citations	Use Zendesk AI for ticket-specific answers
Engineering ADR retrieval	GitHub + Confluence + Slack search	Use GitHub Discussions for code-specific Q&A
Self-hosted enterprise search	MIT license, Docker Compose, air-gap capable	Use Elastic for developer-oriented search infrastructure

Architectural Tradeoffs

What we gained:

Permission-aware RAG by design. ACLs are captured at indexing time and enforced at query time. This is not a bolt-on feature — it is baked into the pipeline architecture. Most RAG systems treat permissions as an afterthought; Danswer treats them as a first-class concern.
Model-agnostic LLM layer. You can swap between OpenAI, Anthropic, Google, Cohere, or a self-hosted Ollama model without changing anything in the search pipeline. This prevents vendor lock-in and allows cost optimization per use case.
Checkpointed, resumable indexing. Connectors save their progress to MinIO. A crash mid-sync does not require a full reindex. This is critical for enterprise-scale connectors that process millions of documents.
Hybrid search with Vespa. BM25 + vector search outperforms either approach alone on enterprise queries. Vespa handles both in a single backend with configurable alpha blending.
Open source with a viable business model. The MIT-licensed Community Edition covers the core use case. The Enterprise Edition adds SSO, advanced RBAC, and usage analytics. This dual-license model ensures the project is sustainable without paywalling essential features.

What we sacrificed:

Fewer connectors than Glean. Danswer has 40+ connectors; Glean has 100+. If your organization uses niche or industry-specific tools, you may need to write custom connectors or use Glean.
Self-hosting is an engineering commitment. Deploying, securing, patching, and upgrading the Docker stack requires real DevOps work. “Free if self-hosted” hides the cost of engineering time. Teams without DevOps support should use the managed cloud version.
No proactive knowledge surfacing. Danswer answers questions when asked but does not proactively surface relevant information. It is a search tool, not a knowledge management system. Tools like Notion or Guru are better for curated, proactive knowledge sharing.
No native Teams integration. Danswer has a Slack bot but no Microsoft Teams integration. Teams-native organizations will need to use the web UI or build a custom integration.
No persistent organizational memory. Danswer does not learn from past queries or build a knowledge graph over time. Each query is independent. Tools like amaiko or custom solutions with knowledge graphs can provide persistent memory.
Limited enterprise compliance. The Community Edition lacks SOC 2 reports, HIPAA BAA, and FedRAMP authorization. Regulated enterprises should evaluate Sinequa or the Onyx Enterprise Edition.

The real lesson: Danswer is the best option for teams that want a self-hosted, permission-aware enterprise search platform without paying $50+/user/month. It is not a replacement for Glean in large enterprises with 100+ SaaS tools, nor is it a replacement for Sinequa in regulated industries. It is the open-source middle ground that covers the 80% use case at 0% of the cost — and for most teams, that is exactly what they need.

Course-Style Deep Dive

How the Indexing Pipeline Works Under the Hood

The indexing pipeline is a distributed, asynchronous system orchestrated by Celery. Here is the complete flow:

Connector Execution. A Celery beat schedule triggers connector runs at their configured refresh_freq. The ConnectorRunner class loads the connector, resolves credentials, and calls poll() or poll_events(). Documents are yielded as Document objects with content, metadata, and ACLs.
Document Validation and Deduplication. Each document is validated against the Document schema (required fields: id, source, content). Duplicates are detected by the document ID — if a document with the same ID already exists in PostgreSQL, it is updated rather than re-inserted.
Chunking. The Chunker class splits document content into DocAwareChunk objects. The chunking strategy is token-aware: it uses a BaseTokenizer to ensure each chunk fits within the embedding model’s context window. Chunk boundaries respect sentence and paragraph structure. Overlapping chunks preserve context across boundaries.
Embedding Generation. The IndexingEmbedder sends chunks to the indexing_model_server (a dedicated model server container) for embedding. The model server runs the configured embedding model (e.g., nomic-ai/nomic-embed-text-v1 for self-hosted, or openai/text-embedding-3-large for API-based). Embeddings are generated in batches and returned as float arrays.
Vespa Indexing. Each chunk is converted to a Vespa document with fields for content, embedding (as a Vespa tensor), metadata, and ACLs. The VespaDocumentIndex.index() method performs a dual-write: it upserts the chunk into Vespa’s hybrid index and updates the document metadata in PostgreSQL.
Checkpoint Save. After each batch, the connector’s checkpoint is updated in MinIO. The checkpoint stores the last successful sync time and any pagination cursors. If the pipeline crashes, the next run resumes from this checkpoint.

Advanced Pattern 1: Custom Connector Development

Danswer’s connector system is pluggable. You can add a custom connector by implementing the Connector interface and registering it:

# backend/onyx/connectors/custom/my_tool.py
from onyx.connectors.models import Connector, Document, Credential
from onyx.connectors.factory import register_connector

@register_connector("my_tool")
class MyToolConnector(Connector):
    """Connector for MyTool, an internal project management tool."""

    @staticmethod
    def credential_schema() -> dict:
        return {
            "type": "object",
            "properties": {
                "api_key": {"type": "string"},
                "base_url": {"type": "string", "format": "uri"},
            },
            "required": ["api_key", "base_url"],
        }

    @staticmethod
    def config_schema() -> dict:
        return {
            "type": "object",
            "properties": {
                "project_ids": {
                    "type": "array",
                    "items": {"type": "string"},
                    "description": "List of project IDs to index",
                },
            },
        }

    def __init__(self, credential: Credential, config: dict):
        self.client = MyToolClient(
            api_key=credential["api_key"],
            base_url=credential["base_url"],
        )
        self.project_ids = config.get("project_ids", [])

    def poll(self, start, end):
        for project_id in self.project_ids:
            for item in self.client.list_items(
                project_id=project_id,
                updated_after=start,
                updated_before=end,
            ):
                yield Document(
                    id=f"mytool-{item['id']}",
                    source="my_tool",
                    content=item["description"],
                    metadata={
                        "title": item["title"],
                        "project_id": project_id,
                        "status": item["status"],
                        "url": item["url"],
                    },
                    access_control_list=[
                        f"mytool_project:{project_id}",
                        f"mytool_user:{item['assignee_id']}",
                    ],
                )

Register the connector by adding it to the CONNECTOR_REGISTRY in backend/onyx/connectors/factory.py. The connector will appear in the Admin UI automatically.

Advanced Pattern 2: Custom Embedding Pipeline

For organizations with domain-specific vocabulary (legal, medical, financial), the default embedding model may not capture semantic relationships accurately. Danswer supports custom embedding models:

# .env — custom embedding configuration
INDEXING_MODEL_SERVER_HOST=indexing_model_server
INDEXING_MODEL_PROVIDER=openai
INDEXING_MODEL_NAME=text-embedding-3-large
INDEXING_MODEL_DIMENSIONS=3072

# Or use a self-hosted model
# INDEXING_MODEL_PROVIDER=ollama
# INDEXING_MODEL_NAME=nomic-embed-text

For maximum control, run a custom embedding model server:

# docker-compose.override.yml
services:
  custom-embedder:
    image: your-registry/custom-embedder:latest
    environment:
      - MODEL_NAME=your-custom-model
    ports:
      - "8001:8001"
    networks:
      - onyx_network

  indexing_model_server:
    environment:
      - CUSTOM_EMBEDDER_URL=http://custom-embedder:8001

Production Considerations

Horizontal scaling. The indexing pipeline uses Celery workers that can be scaled independently. Add more doc_fetching workers for I/O-bound connector polling. Add more doc_processing workers for CPU/GPU-bound chunking and embedding. The workers share the same Redis broker and Vespa backend.

Monitoring. Danswer exposes Prometheus metrics at /metrics on the API server. Key metrics to monitor:

indexing_documents_processed_total — throughput
indexing_errors_total — failure rate
search_latency_seconds — query performance
vespa_document_count — index size

Backup strategy. PostgreSQL stores all metadata and ACLs. Back it up daily. MinIO stores raw files and checkpoints — these can be re-fetched from source systems but backing them up reduces recovery time. Vespa indices can be rebuilt from scratch by reindexing all connectors, but this takes time for large deployments.

LLM cost management. Each search query generates one LLM call (for answer synthesis) plus one embedding call (for query encoding). At $0.01 per LLM call and $0.0001 per embedding call, a team of 100 users making 20 queries/day costs approximately $20/day in LLM costs. Use a cheaper model (GPT-4o-mini, Gemini Flash) for high-volume queries and reserve expensive models for complex questions.

The Results

Metric	Before Danswer	After Danswer	Improvement
Time to find internal knowledge	30-60 minutes (multi-tab search)	30-60 seconds (single query)	60x faster
New hire ramp-up time	4-6 weeks (learning tool geography)	2-3 weeks (ask anything)	2x faster
Support ticket resolution time	15-20 minutes (search + ask)	5-8 minutes (Danswer answer)	2-3x faster
Cross-team knowledge access	Low (siloed tools)	High (unified search)	Significant
Permission compliance	Manual (ask IT)	Automatic (mirrored ACLs)	Eliminated risk
Infrastructure cost (50 users)	$30,000+/year (Glean)	$600-2,400/year (self-host VM)	92-98% savings
Setup time	2-4 weeks (Glean managed)	30 minutes (Docker Compose)	100x faster
Connector count	100+ (Glean)	40+ (Danswer)	Fewer, but covers 80% of needs

What this means for you: Danswer is not a replacement for Glean in a 5,000-person enterprise with 100+ SaaS tools. It is a replacement for the manual, multi-tab, “ask three people on Slack” workflow that most teams under 500 people still use. The 60x speedup on knowledge retrieval is real and reproducible. The key is deploying it correctly: configure the right connectors, set up permission mirroring, and integrate the Slack bot so the team uses it without changing their workflow.

What to Watch Out For

Under-provision the server at your peril. The full Danswer stack (Vespa, PostgreSQL, Redis, MinIO, model servers, API server, web server) needs 10 GB RAM and 32 GB disk. The Lite mode needs 4 GB but disables hybrid search and self-hosted embeddings. A t3.large (8 GB RAM) on AWS will swap under load. Use a t3.xlarge (16 GB RAM) or equivalent.
Set VALID_EMAIL_DOMAINS before going live. Without this restriction, anyone with the URL can create an account. If you use Google OAuth, set this to your company domain. If you use basic auth, set strong passwords and rotate them.
Configure SSL before exposing to the internet. The production Docker Compose file includes a Let’s Encrypt setup. Use it. Exposing the Danswer API without HTTPS means credentials and document content travel in plaintext.
Test permission mirroring with a test user. Create a test user with limited access in your source systems (Confluence, Google Drive, Slack). Log in as that user in Danswer and verify they cannot see documents they should not have access to. This is the single most important validation step.
Monitor connector health. Connectors fail for many reasons: API rate limits, credential expiration, schema changes, network issues. The admin dashboard shows connector status, but you should set up external monitoring (PagerDuty, Opsgenie) for connector failures. A failed connector means stale search results.
Plan for credential rotation. OAuth tokens expire. API keys get rotated. Service account passwords change. Danswer stores credentials in its database, but you need a process for updating them when they change in the source system. Add credential rotation to your quarterly ops review.
Do not use Lite mode in production. Lite mode removes Vespa (no hybrid search), Redis (no task queue), model servers (no self-hosted embeddings), and MinIO (no file storage). It is for evaluation only. Production deployments need the full stack.

Lesson 1: “We deployed Danswer in an afternoon and had our first answer in 30 minutes. The next two weeks were spent tuning connectors and fixing permission mappings. The tool is easy to start with, but the last 20% of configuration takes 80% of the time.” — Danswer user, Hacker News

Lesson 2: “The Slack bot is the killer feature. We tried to get people to use the web UI and nobody did. We deployed the Slack bot and usage went from 5 queries/day to 200 queries/day in a week. Meet people where they already work.” — Danswer user, r/selfhosted

Lesson 3: “Permission mirroring is not set-and-forget. When we restructured our Confluence spaces, the old permissions stopped working and users started getting empty search results. We had to re-sync all connectors after the restructuring. Plan for this.” — Danswer user, Onyx Discord

Advice for Getting Started

Deploy on a cloud VM with at least 16 GB RAM and 50 GB disk. A DigitalOcean droplet or AWS EC2 t3.xlarge works well.
Start with one connector — Confluence or Google Drive, whichever your team uses most. Get that working before adding more.
Configure the Slack bot on day one. It is the highest-ROI integration and the one your team will actually use.
Set up a test user with restricted access and verify permission mirroring works before letting the whole team use it.
Monitor connector health from day one. A failed connector silently returns stale results.
Use the managed cloud version ($20/user/month) if you do not have DevOps support. The self-hosted version is free but requires ongoing maintenance.
Create custom assistants for different roles. A “Support” assistant with a focused knowledge set returns better answers than a single “Everything” assistant.

Next in the Open-Source AI Tools Mastery series: Quivr