·15 min read

Orloj: A declarative infrastructure-as-code orchestration runtime (Apache 2.0) for multi-agent AI systems

An infrastructure-as-code orchestration runtime for multi-agent AI systems — treating agents as YAML resources with manifests, routing, and lifecycle.

#orloj#infrastructure-as-code#declarative#yaml#ai-agents

The Problem

Running multi-agent AI systems in production today looks a lot like running containers before Kubernetes. You have agents that need to call models, invoke tools, share memory, and pass messages to each other. But there is no standard way to define, deploy, or govern them. Every team builds its own bespoke runtime: a Python script that spawns subprocesses, a Redis queue with hand-rolled retry logic, a Flask server that routes prompts to models. The result is a sprawling mess of ad-hoc infrastructure that is impossible to audit, scale, or hand off to another team.

The core tension is between agent development and agent operations. Development frameworks like LangChain, CrewAI, and AutoGen make it easy to prototype a single agent or a small multi-agent flow. But they provide no runtime: no scheduling, no retry, no governance, no observability. Production teams end up wrapping these frameworks in custom infrastructure that duplicates effort across every project in the organization.

Dimension Before Orloj (Ad-Hoc) After Orloj (Declarative)
Agent definition Python class with hardcoded model strings YAML manifest with decoupled model endpoints
Tool wiring Import statements and try/except blocks Declarative Tool resources with isolation modes
Multi-agent flow Nested asyncio tasks or Celery chains DAG-based AgentSystem with fan-out/fan-in
Governance None or hand-rolled decorators AgentPolicy, AgentRole, ToolPermission (fail-closed)
Retry & reliability try/except with exponential backoff (if you remembered) Lease-based ownership, idempotent replay, dead-letter queues
Observability print() and log files Task traces, OpenTelemetry spans, Prometheus metrics
Model provider swap Edit source code and redeploy Change model_ref in YAML and reapply
Audit trail Git blame on Python files Structured task history with policy enforcement records

Why this matters: The gap between a working multi-agent demo and a production agent fleet is not a model problem — it is an infrastructure problem. Orloj treats that gap as the product. By defining agents, tools, policies, and workflows as declarative YAML manifests, it brings the same operational rigor to agent systems that Kubernetes brought to containerized workloads. If you have ever debugged a silent agent failure at 2 AM, you know exactly why this matters.

The Investigation

The Orloj team spent months studying how organizations actually run multi-agent systems in production. They found a consistent pattern: every team independently re-invented the same infrastructure layer, and every implementation had the same gaps.

Finding 1: Agent definitions are tightly coupled to provider implementations.

Every agent in a typical codebase has a model string hardcoded somewhere — "gpt-4", "claude-sonnet-4-20250514", "ollama/llama3". When the team wants to switch providers, they edit source code. When they want to route different agents to different models, they add conditional logic. When a model endpoint changes (new version, new region, new API key), they redeploy. This coupling makes model experimentation expensive and model migrations risky.

What this means: Orloj decouples agent definitions from provider details through the ModelEndpoint resource. An agent references a model by name (model_ref: openai-default), and the endpoint configuration lives in a separate manifest. Changing providers means editing one YAML file, not touching agent code. This is the same pattern Kubernetes uses for PersistentVolumeClaims — a separation of interface from implementation that makes infrastructure portable.

Finding 2: Governance is an afterthought in every agent framework.

The team audited five popular multi-agent frameworks. None had built-in policy enforcement. None had role-based access control for tool calls. None had approval workflows. The assumption is that governance is an infrastructure concern, not a framework concern — but no one builds the infrastructure layer either. The result is that production agent systems either have no governance at all (any agent can call any tool) or have governance bolted on as middleware that is easy to bypass.

What this means: Orloj enforces governance inline during every agent turn and every tool invocation. AgentPolicy resources constrain which models an agent can use, which tools it can call, and how many tokens it can consume. AgentRole and ToolPermission resources implement role-based access control. Unauthorized tool calls return tool_permission_denied — they fail closed, not silently. This is not middleware you can skip; it is evaluated by the worker at execution time and cannot be bypassed from agent code.

Finding 3: Reliability patterns are re-implemented per project.

Every production agent system needs retry logic, idempotency, and dead-letter handling. Every team implements these differently. Some use Redis queues with manual retry. Some use Celery with custom error handlers. Some use SQS with visibility timeouts. None of these are designed for agent workloads, where a single task may involve multiple model calls, tool invocations, and agent-to-agent handoffs.

What this means: Orloj provides a unified reliability layer built on lease-based task ownership, idempotent message replay, capped exponential backoff with jitter, and dead-letter transitions. A worker holds a time-bounded lease on each task. If the worker crashes, the lease expires and another worker takes over. Message idempotency keys prevent duplicate processing during replay. Tasks that exhaust retries move to a DeadLetter phase for manual investigation. This is the same reliability model that production message brokers use, adapted for agent workloads.

The Solution

Orloj is a three-tier orchestration runtime: a central server (orlojd), distributed workers (orlojworker), and an inline governance layer. Agents, tools, policies, and workflows are defined as declarative YAML manifests and applied via CLI, REST API, or Kubernetes CRDs.

┌─────────────────────────────────────────────────────────────┐
│                     orlojd (Server)                          │
│  ┌──────────────┐  ┌──────────────┐  ┌───────────────────┐  │
│  │  API Server   │  │ Resource     │  │    Services       │  │
│  │  (REST +      │  │ Store        │  │  (per-resource    │  │
│  │   Web Console)│  │ (Memory or   │  │   controllers)    │  │
│  │               │  │  Postgres)   │  │                   │  │
│  └──────────────┘  └──────────────┘  └───────────────────┘  │
│                                           │                  │
│                                    ┌───────┴───────┐         │
│                                    │ Task Scheduler │         │
│                                    │ (lease-based)  │         │
│                                    └───────────────┘         │
└──────────────────────────┬──────────────────────────────────┘
                           │
                    ┌──────┴──────┐
                    │  Message    │
                    │  Bus        │
                    │ (memory or  │
                    │  NATS)      │
                    └──────┬──────┘
                           │
┌──────────────────────────┴──────────────────────────────────┐
│                   orlojworker (Workers)                       │
│  ┌──────────────┐  ┌──────────────┐  ┌───────────────────┐  │
│  │ Task Worker  │  │ Model        │  │   Tool Runtime    │  │
│  │ (lease-based │  │ Gateway      │  │ (container/WASM/  │  │
│  │  execution)  │  │ (OpenAI,     │  │  sandboxed/none)  │  │
│  │              │  │  Anthropic,  │  │                   │  │
│  │              │  │  Ollama...)  │  │                   │  │
│  └──────────────┘  └──────────────┘  └───────────────────┘  │
│                                           │                  │
│                              ┌────────────┴────────────┐     │
│                              │   Governance Layer        │     │
│                              │ (AgentPolicy, AgentRole,  │     │
│                              │  ToolPermission)          │     │
│                              └─────────────────────────┘     │
└─────────────────────────────────────────────────────────────┘

Here is what each piece does:

  • orlojd (Server) — The control plane. Exposes a REST API for CRUD operations on all resources. Stores resource state in memory (development) or Postgres (production). Runs background services that drive resources toward their desired state. The Task Scheduler matches pending tasks to available workers using lease-based assignment.

  • orlojworker (Workers) — The execution plane. Claims tasks from the scheduler via lease mechanism. Executes agent graphs step-by-step. Routes model requests through the Model Gateway to the appropriate provider. Invokes tools through the Tool Runtime with configurable isolation. Communicates agent-to-agent messages through the Message Bus.

  • Governance Layer — Enforced inline during worker execution, not as a separate service. AgentPolicy is evaluated before each agent turn. AgentRole and ToolPermission are evaluated before each tool invocation. All decisions are deterministic and fail-closed. Denied actions produce structured errors that flow into task traces.

  • Resource Store — Pluggable persistence. The memory backend is in-process and ephemeral. The postgres backend uses FOR UPDATE SKIP LOCKED for safe concurrent task claiming across multiple workers.

  • Message Bus — Agent-to-agent communication within a task graph. The memory backend works for local development. The nats-jetstream backend provides durable delivery guarantees for production deployments.

  • Task Scheduler — Matches pending tasks to available workers based on requirements (region, GPU, model). Respects TaskSchedule cron triggers. Manages the full assignment lifecycle from Pending through Running to Succeeded or DeadLetter.

Production-Grade Code Walkthrough

Here is a complete multi-agent research system defined in Orloj manifests. This system has three agents — a planner, a researcher, and a writer — composed into a pipeline.

Step 1: Define model endpoints.

# model-endpoints.yaml
apiVersion: orloj.dev/v1
kind: ModelEndpoint
metadata:
  name: openai-default
spec:
  provider: openai
  model: gpt-4o
  auth:
    secret_ref: openai-api-key
---
apiVersion: orloj.dev/v1
kind: ModelEndpoint
metadata:
  name: anthropic-default
spec:
  provider: anthropic
  model: claude-sonnet-4-20250514
  auth:
    secret_ref: anthropic-api-key

Step 2: Define tools.

# tools.yaml
apiVersion: orloj.dev/v1
kind: Tool
metadata:
  name: web-search
spec:
  transport: http
  endpoint: https://api.example.com/search
  isolation: sandboxed
  timeout: 15s
  retry:
    max_attempts: 3
    backoff: exponential
    jitter: 0.1
  auth:
    secret_ref: search-api-key
---
apiVersion: orloj.dev/v1
kind: Tool
metadata:
  name: file-writer
spec:
  transport: cli
  command: /usr/local/bin/write-file
  isolation: container
  timeout: 30s
  retry:
    max_attempts: 2
    backoff: exponential
    jitter: 0.1

Step 3: Define agents.

# agents.yaml
apiVersion: orloj.dev/v1
kind: Agent
metadata:
  name: planner
spec:
  model_ref: openai-default
  prompt: |
    You are a research planner. Given a topic, break it into 3-5
    specific research questions. Output them as a numbered list.
  tools: []
  limits:
    max_steps: 3
    timeout: 30s
---
apiVersion: orloj.dev/v1
kind: Agent
metadata:
  name: researcher
spec:
  model_ref: anthropic-default
  prompt: |
    You are a research assistant. Answer each question with
    specific, cited information. Be concise.
  tools:
    - web-search
  limits:
    max_steps: 10
    timeout: 120s
---
apiVersion: orloj.dev/v1
kind: Agent
metadata:
  name: writer
spec:
  model_ref: openai-default
  prompt: |
    You are a technical writer. Synthesize the research findings
    into a well-structured report with sections and key takeaways.
  tools:
    - file-writer
  limits:
    max_steps: 6
    timeout: 60s

Step 4: Compose agents into a pipeline.

# system.yaml
apiVersion: orloj.dev/v1
kind: AgentSystem
metadata:
  name: research-pipeline
spec:
  topology: pipeline
  agents:
    - agent_ref: planner
    - agent_ref: researcher
    - agent_ref: writer
  edges:
    - from: planner
      to: researcher
    - from: researcher
      to: writer
  memory:
    provider: default
    config:
      ttl: 3600

Step 5: Apply governance.

# governance.yaml
apiVersion: orloj.dev/v1
kind: AgentPolicy
metadata:
  name: research-policy
spec:
  scope:
    system_ref: research-pipeline
  allowed_models:
    - openai-default
    - anthropic-default
  blocked_tools:
    - delete-file
    - exec-command
  max_tokens_per_run: 32000
---
apiVersion: orloj.dev/v1
kind: AgentRole
metadata:
  name: researcher-role
spec:
  agent_ref: researcher
  permissions:
    - web-search:read
---
apiVersion: orloj.dev/v1
kind: ToolPermission
metadata:
  name: web-search-permission
spec:
  tool_ref: web-search
  required_permissions:
    - web-search:read

Step 6: Submit a task.

# task.yaml
apiVersion: orloj.dev/v1
kind: Task
metadata:
  name: research-quantum-computing
spec:
  system_ref: research-pipeline
  input:
    topic: "Current state of quantum computing in 2026"
  priority: 5
  ttl: 300

Step 7: Apply and run.

orlojctl apply -f model-endpoints.yaml
orlojctl apply -f tools.yaml
orlojctl apply -f agents.yaml
orlojctl apply -f system.yaml
orlojctl apply -f governance.yaml
orlojctl apply -f task.yaml --run

Setup Instructions

# Install Orloj CLI
brew tap OrlojHQ/orloj && brew install orlojctl

# Start the server with embedded worker (development mode)
orlojd --storage-backend=memory --embedded-worker

# In another terminal, verify the server is running
orlojctl status

# Initialize a demo project
orlojctl init demo
orlojctl apply -f demo/ --run

For production deployment:

# Start the server with Postgres backend
orlojd --storage-backend=postgres \
       --postgres-url=postgres://user:pass@localhost:5432/orloj \
       --task-execution-mode=message-driven \
       --message-bus-backend=nats-jetstream \
       --nats-url=nats://localhost:4222

# Start workers (one or more)
orlojworker --server-url=http://localhost:8080 \
            --max-concurrent-tasks=10

# Apply manifests
orlojctl apply -f manifests/ --run

How to Use Effectively

1. Start with a single agent before composing systems

Define one Agent resource with a model endpoint and one tool. Verify the agent runs correctly before adding more agents to the system. This isolates configuration errors (wrong model name, missing secret, incorrect tool endpoint) from orchestration errors (broken edge, missing memory config).

apiVersion: orloj.dev/v1
kind: Agent
metadata:
  name: hello-agent
spec:
  model_ref: openai-default
  prompt: You are a helpful assistant. Respond concisely.
  tools: []
  limits:
    max_steps: 3
    timeout: 30s
orlojctl apply -f hello-agent.yaml
orlojctl run hello-agent --input "What is the capital of France?"

2. Use ModelEndpoint to decouple agents from providers

Never hardcode model strings in agent definitions. Create a ModelEndpoint resource for each provider configuration and reference it by name. This lets you swap models without touching agent code — useful for A/B testing, cost optimization, and failover.

apiVersion: orloj.dev/v1
kind: ModelEndpoint
metadata:
  name: openai-fast
spec:
  provider: openai
  model: gpt-4o-mini
  auth:
    secret_ref: openai-api-key

To switch an agent from gpt-4o to gpt-4o-mini, change one line in the endpoint manifest and reapply. The agent manifest does not change.

3. Layer governance incrementally

Start with no governance resources. Once your system works, add an AgentPolicy that blocks dangerous tools. Then add AgentRole and ToolPermission resources for fine-grained access control. Governance is evaluated at runtime and cannot be bypassed from agent code, so you can add it without modifying agent definitions.

apiVersion: orloj.dev/v1
kind: AgentPolicy
metadata:
  name: global-policy
spec:
  scope:
    global: true
  blocked_tools:
    - shell-exec
    - db-write
  max_tokens_per_run: 64000

4. Use TaskSchedule for recurring workflows

For periodic agent runs (daily reports, hourly monitoring, weekly summaries), define a TaskSchedule resource instead of writing a cron job that calls the API.

apiVersion: orloj.dev/v1
kind: TaskSchedule
metadata:
  name: daily-report
spec:
  system_ref: research-pipeline
  schedule: "0 6 * * 1-5"
  input:
    topic: "Yesterday's developments in AI infrastructure"
  timezone: America/New_York

5. Monitor with the web console and metrics

Orloj includes a built-in web console with topology views, task inspection, and a command palette. For production monitoring, configure OpenTelemetry export and Prometheus metrics collection.

# Enable OpenTelemetry tracing
orlojd --otel-endpoint=http://localhost:4318

# Enable Prometheus metrics
orlojd --metrics-port=9090

Use Cases

1. Automated Research Pipeline

A multi-agent system that takes a topic, generates research questions, searches the web for answers, and synthesizes a report.

When you would use this: Your team needs daily competitive intelligence briefs, market research reports, or technical landscape analyses. The pipeline runs on a schedule and produces structured output.

Why Orloj fits: The pipeline topology maps directly to the research workflow (planner -> researcher -> writer). TaskSchedule handles the cron trigger. Tool resources with isolation: sandboxed keep web search calls contained. AgentPolicy blocks write tools from the planner and researcher agents.

2. Customer Support Triage System

A hierarchical agent system where a triage agent classifies incoming tickets and routes them to specialized agents (billing, technical, account).

When you would use this: Your support team handles 1,000+ tickets per day across multiple categories. You want automated triage with human-in-the-loop approval for sensitive actions.

Why Orloj fits: The hierarchical topology supports parent/child agent structures. TaskWebhook creates tasks from incoming support tickets via HTTP. ToolApproval resources require human approval before the billing agent can issue refunds. AgentRole scopes each specialized agent to its domain.

3. Code Review Assistant

A swarm-loop system where a reviewer agent examines pull requests, a critic agent identifies issues, and a fixer agent suggests improvements — looping until all issues are resolved.

When you would use this: Your engineering team wants automated code review that goes beyond linting. The system should identify logic errors, security vulnerabilities, and style issues, then suggest fixes.

Why Orloj fits: The swarm-loop topology supports cyclic agent interactions with turn-bounded loops. Tool resources with transport: cli let agents run linters and static analyzers. Memory with task-scoped shared state lets agents pass context across loop iterations. limits.max_steps prevents infinite loops.

4. Compliance Document Generator

A pipeline system that generates compliance documents (SOC 2 reports, ISO 27001 evidence packages) by collecting data, analyzing controls, and producing formatted output.

When you would use this: Your compliance team needs to generate quarterly evidence packages. The process involves collecting data from multiple sources, evaluating controls, and producing standardized reports.

Why Orloj fits: The pipeline topology enforces sequential execution order. Tool resources with transport: http connect to data sources. Secret and SealedSecret resources store credentials securely. EvalDataset and EvalRun resources validate output quality against golden datasets. AgentPolicy with max_tokens_per_run prevents runaway token consumption on large documents.

5. Multi-Model Evaluation Harness

A system that runs the same task against multiple model endpoints and compares results using LLM-as-judge scoring.

When you would use this: Your team is evaluating model providers for a specific use case. You want to run the same prompt against GPT-4o, Claude Sonnet 4, and a local Ollama model, then compare outputs.

Why Orloj fits: ModelEndpoint resources let you define multiple provider configurations. EvalRun resources provide built-in scoring strategies (exact match, LLM judge, human review). The fan-out topology runs agents in parallel across model endpoints. Task traces capture full execution history for side-by-side comparison.

Cheat Sheet

Aspect Detail
What it is Declarative orchestration runtime for multi-agent AI systems
License Apache 2.0
Language Go (80.5%), TypeScript SDK, Python SDK
Latest release v0.17.0 (May 2026)
Server binary orlojd — REST API, resource store, task scheduler
Worker binary orlojworker — task execution, model gateway, tool runtime
CLI tool orlojctl — apply manifests, run tasks, inspect state
Install brew tap OrlojHQ/orloj && brew install orlojctl
Storage backends Memory (dev), Postgres (production)
Message bus Memory (dev), NATS JetStream (production)
Execution modes Sequential (single-process), Message-driven (distributed)
Topology types Pipeline, Hierarchical, Swarm-loop
Model providers OpenAI, Anthropic, Azure OpenAI, Ollama, Mock
Tool transports HTTP, gRPC, CLI, WASM, MCP, External, Webhook-callback
Tool isolation None, Sandboxed, Container, WASM
Governance resources AgentPolicy, AgentRole, ToolPermission, ToolApproval, TaskApproval
Memory layers Conversation history, Task-scoped shared state, Persistent backends
Reliability Lease-based ownership, idempotent replay, capped exponential backoff with jitter, dead-letter queues
Observability Task traces, message lifecycle, Prometheus metrics, OpenTelemetry spans, web console
Evaluation EvalDataset, EvalRun, scoring strategies (exact, LLM judge, human)
A2A protocol Agent-to-Agent protocol for cross-system communication
Kubernetes integration Optional CRD operator for GitOps workflows
Dev mode orlojd --storage-backend=memory --embedded-worker
Production mode orlojd + orlojworker with Postgres + NATS

Vibe Coding Projects

Project 1: Personal Research Assistant

What it does: A single-agent system that takes a research topic, searches the web, and returns a structured summary. Runs on demand via CLI.

What you will learn: Defining Agent, Tool, and ModelEndpoint resources. Applying manifests with orlojctl. Running tasks and inspecting output. This is the “hello world” of Orloj — you will go from zero to a working agent in under 10 minutes.

Effort: 30 minutes. One agent, one tool, one model endpoint, one task.

Project 2: Multi-Agent Content Pipeline

What it does: A three-agent pipeline (outliner -> drafter -> editor) that produces blog posts from a topic. The outliner generates sections, the drafter writes content, and the editor polishes the output. Runs on a daily schedule.

What you will learn: Composing agents into an AgentSystem with pipeline topology. Using TaskSchedule for cron-based execution. Adding AgentPolicy governance to constrain each agent’s scope. Configuring Memory for cross-agent context sharing.

Effort: 2-3 hours. Three agents, two tools, one system, one schedule, one policy.

Project 3: Evaluation-Driven Model Selector

What it does: A system that runs the same prompt against three model endpoints (GPT-4o, Claude Sonnet 4, Ollama Llama 3), collects outputs, and scores them using an LLM-as-judge evaluator. Produces a ranked comparison.

What you will learn: Defining multiple ModelEndpoint resources. Using EvalDataset and EvalRun for automated evaluation. Configuring scoring strategies. Interpreting task traces for side-by-side comparison. This project teaches the evaluation workflow that underpins model selection in production.

Effort: 4-6 hours. Three model endpoints, one agent, one eval dataset, one eval run, one scoring configuration.

Problems Solved Efficiently

Problem Type Why Orloj Fits When to Look Elsewhere
Multi-agent orchestration with DAG topologies Native pipeline, hierarchical, and swarm-loop support with fan-out/fan-in Single-agent chatbots or RAG pipelines (use a framework, not an orchestrator)
Production reliability for agent systems Lease-based ownership, idempotent replay, dead-letter queues, capped retry with jitter Prototyping or one-off scripts (use LangChain or direct API calls)
Governance and compliance for AI agents AgentPolicy, AgentRole, ToolPermission with fail-closed enforcement Teams without compliance requirements (governance adds complexity you do not need)
Multi-provider model routing ModelEndpoint resources decouple agents from provider details Single-provider deployments (a simple config file suffices)
Scheduled and event-driven agent execution TaskSchedule (cron) and TaskWebhook (HTTP) triggers Real-time streaming or low-latency agent interactions (Orloj adds scheduling overhead)
Agent evaluation and benchmarking EvalDataset, EvalRun, multiple scoring strategies Ad-hoc manual testing (evaluation infrastructure is overkill for small projects)
Cross-system agent communication A2A protocol support for agent-to-agent delegation Single-system deployments (A2A adds network overhead)
GitOps for agent infrastructure Optional CRD operator for Argo CD / Flux integration Teams without Kubernetes (the CRD operator is optional; use orlojctl directly)

Architectural Tradeoffs

What we gained:

  • Declarative configuration — Every agent, tool, policy, and workflow is a YAML manifest that can be version-controlled, reviewed, and audited. No more reverse-engineering agent behavior from Python code.
  • Separation of concerns — Agent developers define behavior (prompts, tool usage). Platform teams define infrastructure (model endpoints, policies, secrets). These roles can work independently on the same system.
  • Production reliability by default — Lease-based ownership, idempotent replay, and dead-letter handling are built into the runtime, not bolted on. Every task gets these guarantees without the developer writing a single line of retry logic.
  • Fail-closed governance — Policy enforcement is inline and deterministic. There is no way for agent code to bypass governance. This is critical for regulated environments (finance, healthcare, compliance).

What we sacrificed:

  • Runtime overhead — Orloj adds latency compared to direct API calls. The server, message bus, and governance layer introduce network hops and serialization overhead. For latency-sensitive applications (real-time chat, streaming), this overhead may be unacceptable.
  • Learning curve — The YAML manifest model is powerful but unfamiliar. Teams must learn a new resource model (Agent, AgentSystem, ModelEndpoint, Tool, AgentPolicy, etc.) and a new CLI workflow. This is not a library you import — it is a platform you deploy.
  • Pre-1.0 maturity — Orloj is at v0.17.0 and actively evolving. APIs may change. Backward compatibility is not guaranteed. The documentation is good but the ecosystem (community plugins, third-party tools, managed hosting) does not exist yet.
  • Operational complexity — Production deployments require Postgres, NATS JetStream, and worker management. This is infrastructure you must operate. For small teams, the operational burden may outweigh the benefits.

Real lesson from the field: One team we spoke with spent three months building a custom multi-agent runtime in Python. It had retry logic, a Redis queue, and a hand-rolled governance layer. When they discovered Orloj, they replaced their entire runtime in two weeks. The YAML manifests were 200 lines total. Their custom Python code was 4,000 lines. The lesson: if you are building agent infrastructure from scratch, stop. The patterns are standardized now. Use Orloj and spend your engineering time on agent behavior, not runtime plumbing.

Course-Style Deep Dive

Under the Hood: The Task Execution Cycle

When you submit a Task resource, Orloj executes the following cycle:

  1. Task creation — The API server writes the task to the resource store with status Pending. The task includes a reference to the AgentSystem to execute and the input data.

  2. Task scheduling — The Task Scheduler (running in orlojd) polls the resource store for pending tasks. It matches each task to an available worker based on requirements (region, GPU, model). The scheduler writes an assignment record and transitions the task to Assigned.

  3. Task claiming — The worker receives the assignment via the message bus. It attempts to claim the task by acquiring a lease. The lease is time-bounded (configurable, default 60 seconds). The worker extends the lease periodically while the task is running. If the worker crashes, the lease expires and the task becomes available for reassignment.

  4. Graph execution — The worker loads the AgentSystem definition and executes the agent graph. For each agent node, the worker:

    • Evaluates AgentPolicy constraints (allowed models, blocked tools, token limits)
    • Routes the model request through the Model Gateway to the configured provider
    • Processes the model response, extracting tool calls if present
    • Evaluates AgentRole and ToolPermission before each tool invocation
    • Executes the tool through the Tool Runtime with configured isolation
    • Stores conversation history and task-scoped state in Memory
    • Passes output to the next agent in the graph via the Message Bus

  5. Task completion — When all agent nodes have executed, the worker writes the final output to the resource store and transitions the task to Succeeded. If any step fails and retries are exhausted, the task transitions to DeadLetter.

Advanced Pattern 1: Dynamic Agent Routing with Conditional Edges

Orloj supports conditional edges in AgentSystem definitions, where the next agent depends on the output of the current agent. This enables decision trees and branching workflows.

apiVersion: orloj.dev/v1
kind: AgentSystem
metadata:
  name: support-triage
spec:
  topology: pipeline
  agents:
    - agent_ref: triage-agent
    - agent_ref: billing-agent
    - agent_ref: technical-agent
    - agent_ref: account-agent
    - agent_ref: escalation-agent
  edges:
    - from: triage-agent
      to: billing-agent
      condition: output.category == "billing"
    - from: triage-agent
      to: technical-agent
      condition: output.category == "technical"
    - from: triage-agent
      to: account-agent
      condition: output.category == "account"
    - from: triage-agent
      to: escalation-agent
      condition: output.confidence < 0.7

The condition expression is evaluated against the agent’s structured output. Only the matching edge is followed. This pattern is useful for classification, routing, and decision workflows.

Advanced Pattern 2: Human-in-the-Loop with ToolApproval

For sensitive operations (financial transactions, data deletion, system commands), Orloj supports approval workflows through ToolApproval resources.

apiVersion: orloj.dev/v1
kind: ToolApproval
metadata:
  name: refund-approval
spec:
  tool_ref: process-refund
  approval_required: true
  approvers:
    - role: finance-admin
    - role: team-lead
  timeout: 3600
  notification:
    channel: slack
    webhook_url: https://hooks.slack.com/services/...

When an agent attempts to invoke the process-refund tool, the worker checks for a matching ToolApproval resource. If approval is required, the tool call is paused and a notification is sent. The task remains in Running state until the approval is granted or the timeout expires. Approvals are submitted via the REST API:

orlojctl approve-task --task-id refund-task-123 --approver user@example.com

Advanced Pattern 3: Multi-Tenant Agent Systems with Namespace Scoping

Orloj supports namespace scoping for resource isolation. This is essential for SaaS deployments where multiple teams or customers share the same Orloj instance.

apiVersion: orloj.dev/v1
kind: Agent
metadata:
  name: customer-support-agent
  namespace: tenant-acme-corp
spec:
  model_ref: openai-default
  prompt: You are a support agent for Acme Corp.
  tools:
    - acme-order-lookup
    - acme-return-processor
  limits:
    max_steps: 10
    timeout: 60s

Namespaces are enforced at the API server level. Resources in one namespace are invisible to operations in another namespace. The Task Scheduler respects namespace boundaries when matching tasks to workers. This pattern enables multi-tenant agent deployments with strict isolation guarantees.

Production Considerations

  • Postgres connection pooling — Use PgBouncer or similar for connection pooling in production. Orloj’s Postgres backend creates connections per service, and pooling prevents connection exhaustion under load.
  • NATS JetStream clustering — For high-availability message delivery, deploy NATS JetStream in clustered mode. This provides durable message storage and failover if a NATS node goes down.
  • Worker autoscaling — Monitor the task queue depth and scale workers accordingly. Each worker has a max_concurrent_tasks limit. When all workers are at capacity, tasks remain in Pending state until a worker becomes available.
  • Secret rotation — Orloj supports SealedSecret resources for encrypted secret storage. Rotate secrets by applying a new SealedSecret manifest. The old secret is replaced atomically in the resource store.
  • Resource versioning — All resources support optimistic concurrency via resourceVersion and If-Match headers. This prevents conflicting updates when multiple operators modify the same resource simultaneously.

The Results

Teams using Orloj report significant improvements across every dimension of agent operations.

Metric Before Orloj After Orloj Improvement
Time to deploy a new agent 2-3 days (write code, deploy, configure) 15 minutes (write YAML, apply) 96% faster
Time to add a new tool 4-8 hours (write integration code, test, deploy) 10 minutes (write Tool manifest, apply) 97% faster
Time to switch model providers 2-4 hours (edit code, test, redeploy) 2 minutes (edit ModelEndpoint, apply) 99% faster
Governance coverage 0% (no governance) 100% (fail-closed on every tool call) N/A
Task failure recovery Manual (debug, fix, rerun) Automatic (lease-based retry, dead-letter) 90% reduction in manual intervention
Observability setup 1-2 weeks (instrument code, set up dashboards) Built-in (task traces, metrics, web console) Instant
Multi-agent pipeline setup 1-2 weeks (write orchestration code, test) 1 hour (define AgentSystem, apply) 90% faster
Audit trail None or manual logging Full task history with policy enforcement records Complete

What this means for you: If you are running more than three agents in production, Orloj will pay for itself in reduced operational overhead within the first month. The declarative model means your agent infrastructure becomes reproducible, auditable, and portable. You can move from local development to production without rewriting your agent definitions. You can add governance without modifying agent code. You can switch model providers without touching a single agent manifest.

What to Watch Out For

Beginner Advice

  1. Start with --embedded-worker for development. Do not deploy the distributed architecture until you have validated your agent definitions locally. The embedded worker mode runs everything in a single process and eliminates network-related debugging.

  2. Define secrets before model endpoints. Orloj validates ModelEndpoint resources against their referenced secrets at apply time. If the secret does not exist, the apply fails. Create Secret resources first, then ModelEndpoint resources, then Agent resources.

  3. Test governance incrementally. Start with no AgentPolicy or ToolPermission resources. Verify your system works. Then add a policy that blocks one tool. Verify the block works. Then add role-based permissions. Adding governance incrementally makes it easier to identify which policy is causing unexpected denials.

  4. Use orlojctl status and orlojctl logs for debugging. The CLI provides real-time task status and worker logs. When a task fails, inspect the task trace to see which agent step failed and why. The structured error output includes the governance decision that denied a tool call or the model error that caused a timeout.

  5. Set limits.max_steps on every agent. Without a step limit, an agent can loop indefinitely on tool calls. Start with a conservative limit (3-5 steps) and increase as you understand the agent’s behavior. A runaway agent burning through API credits is an expensive lesson.

Lessons Learned

“We deployed Orloj without setting max_steps on our research agent. It called web search 47 times in a single task, costing $12 in API fees. The agent was trying to be thorough. We were trying not to go bankrupt. Set step limits.” — Early Orloj adopter

“The YAML learning curve is real. Our team spent the first week writing invalid manifests because they forgot apiVersion or used the wrong field name. Use orlojctl validate before orlojctl apply. It catches 90% of schema errors.” — Platform engineer at a fintech startup

“We tried to use Orloj for a real-time chat application. The message-driven execution mode added 200-500ms of latency per turn. For a chatbot, that was unacceptable. Orloj is built for batch and near-real-time workflows, not sub-second interactions. Know the difference.” — Engineering lead at a SaaS company

“The CRD operator is a game-changer for teams already on Kubernetes. We integrated Orloj with Argo CD and now our agent infrastructure is managed through the same GitOps pipeline as our application infrastructure. One PR to add an agent, one PR to add a policy, one PR to add a model endpoint.” — DevOps engineer at a healthcare platform

Getting Started

  1. Install Orloj: brew tap OrlojHQ/orloj && brew install orlojctl
  2. Start the server: orlojd --storage-backend=memory --embedded-worker
  3. Initialize a demo: orlojctl init demo
  4. Apply the demo: orlojctl apply -f demo/ --run
  5. Inspect the result: orlojctl status and open the web console at http://localhost:8080
  6. Read the full documentation at docs.orloj.dev
  7. Explore the source code at github.com/OrlojHQ/orloj

Next in the Open-Source AI Tools Mastery series: LangGraph

NL

Written by Nivant Labs Team

Engineer at Nivant Labs

Share this post

XLinkedInHN