Webwright: Terminal-native web agent achieving 86.7% on Online-Mind2Web

The Problem

Every web agent framework built in the last two years shares the same fundamental design: a persistent browser session where the LLM predicts one primitive action at a time — click element 47, type “hello” into element 12, scroll down 300 pixels. The agent holds a fragile pointer into the DOM, the browser state is the only state, and if the page re-renders mid-task the entire trajectory is invalidated.

This approach has three systemic failure modes:

1. Action granularity mismatch. A single human intent (“book the cheapest flight on June 15”) requires 30-80 discrete DOM actions. Each action is a separate LLM call. Each call is an opportunity for the model to mis-predict a coordinate or select the wrong element ID. Error compounds linearly.

2. No durable artifacts. The agent’s work lives in the browser session. When the session ends, the knowledge dies with it. You cannot replay, audit, or share what the agent did. Every task starts from zero.

3. Context window saturation. Each observation returns the full accessibility tree or DOM snapshot. After 10-15 steps, the model is drowning in page state and losing track of the original goal.

Dimension	Conventional Web Agent	Webwright
Action space	Single click/type/scroll per LLM call	Free-form Python (Playwright scripts)
State persistence	Browser session only	Local workspace (code, logs, screenshots)
Reusability	None (trajectory is ephemeral)	Reusable, parameterized Python scripts
Error recovery	Re-plan from current DOM state	Edit code, re-run in fresh browser
Context efficiency	Full DOM/AX tree every step	Screenshots + logs on demand
Step count per task	30-80+	15-30 (code composes actions)

Why this matters: The conventional web agent paradigm treats the browser as a persistent puppet that the LLM controls one string at a time. Webwright treats the browser as a disposable resource the agent launches, inspects, and discards while writing a program. This is not an incremental improvement — it is a fundamentally different decomposition of the problem that yields 15-35% absolute accuracy gains on every published benchmark.

The Investigation

Microsoft Research’s AI Frontiers lab (in collaboration with the University of Hong Kong) ran a systematic investigation into why conventional web agents fail. Their diagnosis: the problem is not the model — it is the harness.

Finding 1: Action space is the bottleneck, not the model.

The same GPT-5.4 model achieves 33.5% accuracy on Online-Mind2Web when predicting xy-coordinate actions in a persistent browser. When given a terminal and asked to write Playwright code, the same model achieves 86.7%. That is a 53.2 percentage point improvement from changing the action space alone.

What this means: The model knows how to navigate the web. The harness was preventing it from expressing that knowledge efficiently. Code lets the model compose 10-20 interactions into a single script, reducing the number of LLM calls by 3-5x and eliminating the compounding error of per-step prediction.

Finding 2: Persistent browser state creates a single point of failure.

Conventional agents maintain a single browser session across all steps. If the page navigates unexpectedly, a modal dialog appears, or a session cookie expires, the agent’s mental model of the page state diverges from reality. Recovery requires the model to notice the divergence and re-plan — something models do poorly because they cannot “see” the page state directly.

Webwright’s agent spawns a fresh browser for each script execution. If a script fails, the agent edits the code and re-runs. The browser is stateless from the agent’s perspective — it is a compute resource, not a stateful partner.

What this means: Disposable browsers eliminate the state synchronization problem entirely. The agent never needs to “recover” from a bad page state because it never commits to one.

Finding 3: Self-reflection gates prevent premature success claims.

The team found that agents routinely claim success after producing a script that looks correct but fails on execution. Webwright’s self-reflection gate requires the agent to: (1) write a final script, (2) run it in a fresh directory, (3) capture logs and screenshots, (4) run an LLM-based self-reflection judge against the output, and (5) only emit done: true if the judge returns predicted_label == 1.

On the Odysseys benchmark, this gate alone prevented 12-18% of false-positive completions across model backends.

What this means: An agent that cannot verify its own work will confidently report success on tasks it failed. The self-reflection gate is a cheap (one extra LLM call) but effective guardrail that should be standard in every production web agent deployment.

Benchmark Data:

Benchmark	Metric	Webwright (GPT-5.4)	Prior SOTA	Delta
Online-Mind2Web (300 tasks, 136 sites)	Accuracy (N=100)	86.7%	~71% (best open-source)	+15.7 pts
Online-Mind2Web — Easy split	Accuracy	96.2%	—	—
Online-Mind2Web — Medium split	Accuracy	88.1%	—	—
Online-Mind2Web — Hard split	Accuracy	76.6%	—	—
Odysseys (200 long-horizon tasks)	Success rate	60.1%	44.5% (Opus 4.6)	+15.6 pts
Odysseys — GPT-5.4 baseline (xy-coord)	Success rate	33.5%	—	+26.6 pts (relative)
Online-Mind2Web — Claude Opus 4.7	Accuracy	84.7%	—	—
Online-Mind2Web Hard — Claude Opus 4.7	Accuracy	80.5%	76.6% (GPT-5.4)	+3.9 pts
Online-Mind2Web — Qwen-3.5-9B (with tools)	Accuracy	66.2%	—	—

Cost analysis: GPT-5.4 averages $2.37/task on Online-Mind2Web. Claude Opus 4.7 averages $6.09/task — 2.6x more expensive despite being more step-efficient (21.9 vs 26.3 mean steps). The first 50 steps deliver ~82% accuracy; the next 50 add only 3-4 additional points, suggesting a steep diminishing returns curve.

The Solution

Webwright is a ~1,000 line Python harness (MIT license, 5,500+ GitHub stars) that replaces the persistent-browser paradigm with a terminal-native agent loop. The entire system fits in three modules:

┌─────────────────────────────────────────────────────────────┐
│                    Webwright Agent Loop                       │
│                                                               │
│  ┌──────────┐    ┌──────────────┐    ┌──────────────────┐   │
│  │  Runner   │───▶│  Model       │───▶│  Environment     │   │
│  │ (~150 LoC)│    │  Endpoint    │    │  (~300 LoC)      │   │
│  │           │    │  (~550 LoC)  │    │                  │   │
│  │  • Manage │    │  • OpenAI    │    │  • Local workspace│   │
│  │    loop   │    │  • Anthropic │    │  • Execute bash   │   │
│  │  • Render │    │  • OpenRouter│    │  • Playwright     │   │
│  │    prompts│    │  • Parse     │    │  • Capture output │   │
│  │  • Compact│    │    actions   │    │  • Screenshots    │   │
│  │    history│    │  • Format    │    │  • Error handling │   │
│  └────┬─────┘    └──────┬───────┘    └────────┬─────────┘   │
│       │                 │                      │             │
│       └─────────────────┴──────────────────────┘             │
│                           │                                   │
│                           ▼                                   │
│              ┌─────────────────────────┐                      │
│              │  Local Workspace        │                      │
│              │  ┌───────────────────┐  │                      │
│              │  │ plan.md           │  │                      │
│              │  │ final_script.py   │  │                      │
│              │  │ final_runs/       │  │                      │
│              │  │ screenshots/      │  │                      │
│              │  │ trajectory.json   │  │                      │
│              │  │ runtime_errors.log │  │                      │
│              │  └───────────────────┘  │                      │
│              └─────────────────────────┘                      │
└─────────────────────────────────────────────────────────────┘

Here’s what each piece does:

• Runner (agents/default.py, ~450 LoC): The core agent loop. Renders system and instance templates from Jinja2, sends the message history to the model, parses the response into actions (bash commands or Python code), feeds observations back, and enforces the self-reflection gate before accepting done: true. Handles history compaction every N steps by asking the model to summarize the conversation so far.

• Model Endpoint (models/): Thin wrappers around OpenAI, Anthropic, and OpenRouter APIs. Each is ~150-200 lines. Handles message formatting, token counting, streaming, and error retries. The format_message and format_observation_messages methods normalize the model-specific response format into a uniform action dict.

• Environment (environments/, ~570 LoC): Manages a local workspace directory and executes shell commands. Each command runs in a subprocess with a 60-second timeout. Captures stdout, stderr, exit codes, and screenshots. The workspace persists across steps — scripts, logs, and screenshots accumulate as durable artifacts.

• CLI (run/cli.py, ~175 LoC): Typer-based command-line interface. Merges YAML config files, resolves output paths, wires model + environment + agent together, and handles cleanup in a finally block. Supports --debug mode for headed browser inspection.

The agent loop in pseudocode:

while not done:
    1. Render system prompt + task + conversation history
    2. Query LLM → returns {thought, bash_command or python_code, done?}
    3. If done: check self-reflection gate → accept or reject
    4. Execute command in environment → {stdout, stderr, screenshots, exit_code}
    5. Feed observation back into message history
    6. If step_count % N == 0: compact history via LLM summary

Production-grade code walkthrough — the core step loop:

# From webwright/agents/default.py — the heart of the agent
def step(self) -> list[dict[str, Any]]:
    return self.execute_actions(self.query())

def query(self) -> dict[str, Any]:
    if 0 < self.config.step_limit <= self.n_calls:
        raise LimitsExceeded(
            self.model.format_message(
                role="exit",
                content="Step limit exceeded.",
                extra={"exit_status": "LimitsExceeded", "submission": ""},
            )
        )
    message = self.model.query(self.messages)
    self.n_calls += 1
    self.add_messages(message)
    return message

def execute_actions(self, message: dict[str, Any]) -> list[dict[str, Any]]:
    extra = message.get("extra", {})
    if extra.get("done"):
        gate_error = self._self_reflection_gate_error()
        if gate_error is not None:
            extra["done"] = False
            return self.add_messages(
                self.model.format_message(
                    role="user",
                    content=gate_error,
                    extra={"interrupt_type": "SelfReflectionGate"},
                )
            )
        # ... emit exit message
    outputs = [self.env.execute(action) for action in extra.get("actions", [])]
    observation_messages = self.model.format_observation_messages(
        message, outputs, self.get_template_vars()
    )
    return self.add_messages(*observation_messages)

The self-reflection gate is the most important production pattern in this codebase. Here is the exact logic that prevents false-positive completions:

def _self_reflection_gate_error(self) -> str | None:
    if not self.config.require_self_reflection_success:
        return None
    # Check that final_runs/run_<id>/self_reflect_result.json exists
    # and contains predicted_label == 1
    judge_path = latest_run_dir / "self_reflect_result.json"
    judge_data = json.loads(judge_path.read_text(encoding="utf-8"))
    predicted_label = judge_data.get("predicted_label")
    if predicted_label != 1:
        return (
            f"Completion blocked: predicted_label={predicted_label!r} "
            f"(expected 1). Diagnose the failure, fix final_script.py, "
            f"re-run in a new folder, and re-run self_reflection."
        )
    return None

Setup instructions:

# Prerequisites: Python 3.10+, Chromium, API key
git clone https://github.com/microsoft/Webwright.git
cd Webwright
pip install -e .
playwright install chromium

# Export your API key
export OPENAI_API_KEY="sk-..."  # or ANTHROPIC_API_KEY

# Run a task
python -m webwright.run.cli \
    -c base.yaml -c model_openai.yaml \
    -t "Search for flights from SEA to JFK on 2026-08-15 to 2026-08-20" \
    --start-url https://www.google.com/flights \
    --task-id demo_flights \
    -o outputs/default

# Validate your setup
python -m webwright.run.cli doctor

How to Use Effectively

Step 1: Configure the model and environment

Webwright uses stackable YAML configs. Start with base.yaml (shared settings) and layer a model-specific config on top:

# config/my_setup.yaml
model:
  name: gpt-5.4
  temperature: 0.0
  max_tokens: 8192

environment:
  headless: true
  slow_mo_ms: 100
  output_dir: ./outputs

agent:
  step_limit: 30
  require_self_reflection_success: true
  summary_every_n_steps: 20
  keep_last_n_observations: 5

Run with: python -m webwright.run.cli -c base.yaml -c my_setup.yaml -t "..."

Step 2: Write effective task descriptions

The task prompt is the single most important input. Be specific about constraints, output format, and success criteria:

# Good
"Find the cheapest used 8-cylinder BMW made between 2005-2015, priced $25k-$50k, 
 mileage under 50k miles. Output the VIN, price, mileage, and dealer URL as a CSV row."

# Bad
"Find a cheap BMW."

The model uses the task description to write its plan.md and final_script.py. Ambiguous tasks produce ambiguous scripts.

Step 3: Use the debug mode for development

When building a new task script, run with --debug to see the browser in action:

python -m webwright.run.cli \
    -c base.yaml -c model_openai.yaml \
    -t "..." \
    --start-url "..." \
    --debug

This launches a headed Chromium instance with devtools open, 250ms slow-mo, and the browser stays open after the script finishes. You can inspect selectors, watch the agent’s actions in real time, and diagnose failures by examining the browser console.

Step 4: Export reusable scripts with the task showcase

Once a task works, add the task showcase config to generate a repeatable, parameterized script:

python -m webwright.run.cli \
    -c base.yaml -c model_openai.yaml -c task_showcase.yaml \
    -t "<repeatable web task>" \
    --task-id my_repeatable_task \
    -o outputs/default

This generates a report.json with structured output. Render the dashboard:

python assets/task_showcase/app.py \
    --tasks-dir outputs/default/<run>/task_showcase/tasks

Step 5: Integrate with Claude Code or Codex

Webwright ships plugin manifests for Claude Code, OpenAI Codex, OpenClaw, and Hermes Agent:

# Claude Code
/plugin marketplace add microsoft/Webwright
/plugin install webwright@webwright

# OpenAI Codex
codex plugin marketplace add microsoft/Webwright
# Then restart Codex and use @webwright in a new thread

# OpenClaw
openclaw plugins install /absolute/path/to/Webwright
openclaw gateway restart

In Claude Code, two slash commands become available:

• /webwright:run — one-shot final_script.py for literal task values
• /webwright:craft — reusable CLI tool with parameterized function, argparse wrapper, and Google-style docstring

Use Cases

1. Automated flight and hotel price monitoring

When you’d use this: You need to check prices across multiple travel sites daily and get a structured report.

Why Webwright fits: The agent writes a Playwright script that navigates Google Flights, Kayak, or Expedia, extracts prices, and outputs a CSV. The script is reusable — parameterize the dates and destinations, run it on a cron job, and get price alerts without maintaining fragile selectors by hand.

2. E-commerce competitor analysis

When you’d use this: Your team needs weekly pricing data from 5-10 competitor product pages.

Why Webwright fits: Each competitor site gets its own script. The agent handles site-specific navigation (login walls, cookie consent, pagination) in code. Scripts are version-controlled and auditable. When a site redesign breaks a script, the agent re-explores and fixes it in minutes rather than hours of manual selector debugging.

3. Form automation for government and enterprise portals

When you’d use this: You need to submit forms on portals that have no API — tax filings, permit applications, vendor registration.

Why Webwright fits: These workflows are high-stakes (errors cost money) and the pages are complex (multi-step forms with file uploads, CAPTCHAs, session timeouts). The agent writes a script that handles each step, captures screenshots at every stage for audit, and the self-reflection gate verifies the submission succeeded before reporting done.

4. Data extraction from JavaScript-heavy SPAs

When you’d use this: A single-page application renders data dynamically and the underlying API is not documented.

Why Webwright fits: The agent can open browser devtools, inspect network requests, and write a script that either extracts data from the DOM or intercepts the XHR/fetch calls. The script captures the data as JSON, and the agent can verify the output matches expected patterns before finishing.

5. Accessibility compliance auditing

When you’d use this: You need to audit a web application for WCAG compliance issues across 50+ pages.

Why Webwright fits: The agent writes Playwright scripts that navigate each page, capture accessibility trees, check for missing ARIA labels, contrast ratios, and keyboard navigation issues. The output is a structured report with screenshots of violations. The self-reflection gate verifies each page was fully loaded before the audit ran. The research paper explicitly notes that web-agent infrastructure built on accessibility trees has a responsibility to give back to the accessibility community — Webwright’s audit scripts are a direct application of that principle.

6. CI/CD pipeline integration for end-to-end testing

When you’d use this: Your CI pipeline needs to verify that a deployment works correctly by walking through critical user journeys on the staging environment.

Why Webwright fits: The agent writes Playwright scripts that function as end-to-end tests. Unlike traditional E2E test suites that require manual selector maintenance, Webwright’s agent can re-explore and fix broken selectors automatically when the UI changes. The self-reflection gate acts as a test assertion — if the judge determines the journey failed, the CI pipeline gets a non-zero exit code. The trajectory.json output provides a full audit trail for debugging failures.

7. Research data collection from academic portals

When you’d use this: You need to collect publication metadata from 20+ conference websites, each with a different layout and navigation pattern.

Why Webwright fits: Each conference site gets its own Playwright script. The agent handles site-specific quirks (pop-up modals, cookie walls, paginated paper lists) in code. The output is a normalized JSON dataset. When a conference updates its website, the agent re-explores and fixes the script in minutes. The reusable script library means next year’s data collection starts from last year’s working scripts, not from scratch.

Cheat Sheet

Aspect	Detail
Repository	github.com/microsoft/Webwright
License	MIT
Language	Python 3.10+ (58% Python, rest JS/HTML/CSS/Shell)
GPU requirements	None (uses API-based LLMs)
Setup time	5 minutes (clone + pip install + playwright install chromium)
Key features	Terminal-native agent loop, self-reflection gate, history compaction, reusable script output, plugin integrations for Claude Code/Codex/OpenClaw/Hermes
Model backends	OpenAI, Anthropic, OpenRouter
Dependencies	httpx, pydantic, playwright, typer, jinja2
Code footprint	~1,000 lines across 3 core modules
Benchmark SOTA	86.7% Online-Mind2Web, 60.1% Odysseys
Cost per task	$2.37 (GPT-5.4), $6.09 (Claude Opus 4.7)
Common gotchas	Self-reflection gate blocks completion if require_self_reflection_success: true but no judge result exists; history compaction can lose nuance on long tasks; Playwright scripts may break on site redesigns; headed mode requires display server on Linux

Vibe Coding Projects

Project 1: Personal price tracker for concert tickets

What it does: A script that checks Ticketmaster or StubHub daily for specific events and sends a Slack notification when prices drop below a threshold.

What you’ll learn: Parameterizing Playwright scripts with argparse, handling cookie consent and login flows, scheduling with cron, integrating with Slack webhooks.

Effort: 2-3 hours. The agent writes 90% of the script; you add the Slack integration and cron wrapper.

Project 2: Multi-site job application auto-filler

What it does: A CLI tool that takes a resume PDF and cover letter template, navigates through 5-10 job application portals (Greenhouse, Lever, Workday), fills in fields, and uploads documents.

What you’ll learn: Handling iframe-based form fields, file upload automation, multi-step form navigation, error recovery when fields are missing or renamed.

Effort: 4-6 hours. Each portal requires a separate script. The agent handles the first portal; you generalize the pattern.

Project 3: SaaS onboarding flow tester

What it does: A regression test suite that walks through your SaaS product’s signup, onboarding, and first-paywall flow, capturing screenshots at each step and flagging visual regressions.

What you’ll learn: Using Webwright as a testing harness, integrating with pixel-diff tools, parameterizing across user roles (admin, viewer, trial), running in CI pipelines. You will also learn how to structure the self-reflection judge to act as a test assertion — the judge checks that each page loaded correctly, that error states are absent, and that the flow completed as expected.

Effort: 3-4 hours. The agent writes the navigation scripts; you add the visual diff and CI integration. The hardest part is defining the success criteria for the self-reflection judge — be explicit about what “success” looks like for each step of the flow.

Project 4: Automated social media content scheduler

What it does: A script that logs into LinkedIn, Twitter/X, or a CMS dashboard, drafts posts from a content calendar CSV, schedules them, and captures confirmation screenshots for audit.

What you’ll learn: Handling OAuth login flows, multi-step form interactions with rich text editors, file uploads for media attachments, and scheduling with timezone-aware datetime parameters. The reusable script pattern means you can share the scheduler across your team without sharing browser session state.

Effort: 3-5 hours. The login flow is the trickiest part — most social platforms have anti-bot measures that require careful handling of session cookies and rate limits. The agent handles the navigation; you handle the credential management and scheduling logic.

Project 5: API documentation scraper with version diffing

What it does: A weekly cron job that visits 5-10 API documentation sites (Stripe, Twilio, OpenAI, etc.), extracts endpoint definitions, parameters, and response schemas, and diffs them against the previous week’s snapshot to detect breaking changes.

What you’ll learn: Navigating documentation sites with varying layouts (sidebar nav, search-based, single-page), extracting structured data from HTML tables and code blocks, storing versioned snapshots in git, and generating human-readable changelogs from diffs.

Effort: 4-6 hours. Each documentation site requires a separate script. The diffing logic is the most complex part — you need to normalize the extracted data before comparing, because documentation sites often reorder sections or reformat code examples without changing the actual API.

Problems Solved Efficiently

Problem Type	Why Webwright Fits	When to Look Elsewhere
Multi-step form submission	Code composes 10-20 actions per script; self-reflection verifies success	Single-click tasks (use a bookmarklet or simple Puppeteer script)
Data extraction from dynamic pages	Playwright handles JS-rendered content; agent inspects network requests	Static HTML scraping (use BeautifulSoup or cheerio)
Cross-site workflow automation	Agent spawns multiple browser sessions; scripts are composable	API-first integrations (use the actual API if available)
Repetitive QA regression testing	Scripts are version-controlled and CI-runnable; screenshots provide audit trail	Exploratory testing (use a human or a dedicated testing framework)
Price monitoring and alerts	Reusable parameterized scripts; cron-schedulable	Real-time streaming data (use WebSocket listeners or webhook integrations)
Accessibility auditing	Agent navigates all pages; captures AX trees and screenshots	Deep WCAG analysis (use axe-core or Lighthouse CI for comprehensive reports)

Architectural Tradeoffs

What we gained:

• 3-5x reduction in LLM calls per task. Code composes actions that would otherwise require individual model predictions. The Odysseys benchmark shows Webwright completing tasks in 76.1 mean steps vs. 200+ for conventional agents on the same tasks.

• Durable, auditable artifacts. Every run produces a workspace with plan.md, scripts, logs, screenshots, and trajectory.json. You can replay, debug, and share the agent’s work. This is the difference between “the agent did something” and “here is exactly what the agent did, with evidence.”

• Model-agnostic performance. The terminal-native paradigm works across model families. GPT-5.4, Claude Opus 4.7, and even Qwen-3.5-9B all show significant gains over their respective baselines. The improvement comes from the harness, not the model.

• Reusable script library. Common patterns — filling a form, picking a date, making a reservation — become cached scripts. The project demonstrates that even a 9B parameter model reaches 66.2% on the hard split of Online-Mind2Web when augmented with 5+ reusable tools.

What we sacrificed:

• No real-time interactivity. The agent writes code and executes it. There is no “watch the agent browse” experience. If you need a human-in-the-loop to approve each action, this is not the right paradigm.

• Script brittleness. Playwright scripts depend on DOM structure. A site redesign can break a working script. The agent can re-explore and fix it, but that takes additional LLM calls and time. Low-level action spaces (click element, type text) are more resilient to layout changes because they target semantic elements rather than CSS selectors.

• Higher latency per step. Executing a Playwright script takes 5-30 seconds. A conventional agent’s single action takes 1-3 seconds. The total wall-clock time per task is comparable (fewer steps but slower steps), but the perceived latency is worse.

• No support for non-browser interfaces. The terminal-native paradigm assumes the web is the interface. If your task involves native mobile apps, desktop applications, or API calls, Webwright is not the right tool.

The real lesson: The web agent community has been optimizing the wrong thing. We spent two years building better action predictors, better DOM parsers, and better planning hierarchies. Microsoft Research spent a weekend giving the model a terminal and letting it write code. The terminal won by 15-35 points. The lesson is not that code is better than clicks — it is that the model’s ability to plan and compose is the bottleneck, not its ability to point at elements.

Course-Style Deep Dive

How the agent loop works under the hood

The agent loop is a flat, single-threaded state machine with no branching, no sub-agents, and no planning hierarchy. Here is the exact flow:

1. Template rendering. The system prompt and instance prompt are Jinja2 templates. They are rendered with variables from the model, environment, and agent config. The system prompt tells the model it has a terminal, a workspace, and the ability to write and execute Python/Playwright scripts. The instance prompt contains the task description and the current workspace state.

2. Model query. The rendered messages are sent to the LLM. The model returns a JSON response with a thought field (free-text reasoning) and an actions array. Each action has either a bash_command or python_code field. The model can also set done: true with a final_response string.

3. Action execution. Each action is executed sequentially in the environment. The environment runs the command in a subprocess with a 60-second timeout. stdout, stderr, and the exit code are captured. If the command is a Playwright script, screenshots are saved to the workspace.

4. Observation formatting. The model’s format_observation_messages method converts the raw execution output into a structured observation message. This includes the terminal output, any error tracebacks, and references to screenshot files. The ARIA snapshot is included in full for the most recent observation; older observations have their ARIA snapshots pruned to save context.

5. History compaction. Every summary_every_n_steps steps, the agent asks the model to summarize the entire conversation so far. The summary replaces all previous messages (except the system prompt) in the message history. The workspace artifacts (scripts, logs, screenshots) are not affected — they persist on disk.

6. Self-reflection gate. When the model emits done: true, the agent checks if require_self_reflection_success is enabled. If so, it verifies that a self_reflect_result.json file exists in the latest final_runs/run_<id>/ directory and contains predicted_label == 1. If not, the agent injects an error message explaining exactly what the model needs to do to pass the gate.

Advanced Pattern 1: Tool-augmented small model workflows

Even small models benefit from the terminal-native paradigm when given pre-built tools. Here is the pattern for augmenting Qwen-3.5-9B with reusable scripts:

# tools/search_products.py — a reusable tool the agent can import
import argparse
import json
from playwright.sync_api import sync_playwright

def search_products(query: str, max_results: int = 5) -> list[dict]:
    """Search an e-commerce site and return structured product data."""
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page = browser.new_page()
        page.goto("https://example.com/search?q=" + query)
        page.wait_for_selector(".product-card", timeout=10000)
        products = []
        for card in page.query_selector_all(".product-card")[:max_results]:
            products.append({
                "name": card.query_selector(".title").inner_text(),
                "price": card.query_selector(".price").inner_text(),
                "url": card.query_selector("a").get_attribute("href"),
            })
        browser.close()
    return products

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--query", required=True)
    parser.add_argument("--max-results", type=int, default=5)
    args = parser.parse_args()
    print(json.dumps(search_products(args.query, args.max_results)))

The agent’s system prompt includes a manifest of available tools. The model writes scripts that import and call these tools rather than writing raw Playwright code for every task. This is how Qwen-3.5-9B reaches 66.2% on the hard split — the tools handle the complex navigation; the model handles the orchestration.

Advanced Pattern 2: Self-reflection judge implementation

The self-reflection judge is a separate LLM call that evaluates whether the agent’s final script actually completed the task. Here is the exact pattern the agent follows:

# The agent writes this self-reflection config and runs the judge
# self_reflect_config.json
{
  "task": "Find the cheapest used 8-cylinder BMW made between 2005-2015, 
           priced $25k-$50k, mileage under 50k miles.",
  "success_criteria": [
    "At least one result was found",
    "The result includes a VIN number",
    "The price is between $25,000 and $50,000",
    "The mileage is under 50,000 miles",
    "The vehicle has 8 cylinders",
    "The model year is between 2005 and 2015"
  ],
  "output_files": [
    "final_runs/run_3/results.csv",
    "final_runs/run_3/screenshots/result_page.png"
  ]
}

# The agent runs the judge
python -m webwright.tools.self_reflection \
    --config self_reflect_config.json \
    --workspace-dir "/workspace/task_123" \
    --output final_runs/run_3/self_reflect_result.json

The judge reads the output files, checks each success criterion against the evidence, and returns predicted_label: 1 (success) or predicted_label: 0 (failure) with a detailed explanation. If the judge returns 0, the agent must diagnose the failure from the explanation, fix the script, re-run in a new final_runs/run_4/ directory, and re-run the judge.

This pattern is directly inspired by SWE-agent’s approach to software engineering tasks, adapted for the web domain. The key difference is that the judge evaluates visual and structural evidence (screenshots, extracted data) rather than test pass/fail signals.

Advanced Pattern 3: Multi-session parallel exploration

The terminal-native paradigm allows the agent to launch multiple browser sessions simultaneously. This is useful for comparison tasks:

# The agent writes this script to compare prices across 3 sites in parallel
import asyncio
from playwright.async_api import async_playwright

async def check_site(name: str, url: str, selector: str) -> dict:
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        page = await browser.new_page()
        await page.goto(url)
        await page.wait_for_selector(selector, timeout=15000)
        price = await page.query_selector(selector)
        text = await price.inner_text() if price else "N/A"
        await browser.close()
        return {"site": name, "price": text}

async def main():
    sites = [
        ("Site A", "https://site-a.com/product", ".price-tag"),
        ("Site B", "https://site-b.com/item", ".sale-price"),
        ("Site C", "https://site-c.com/offer", ".current-price"),
    ]
    results = await asyncio.gather(*[check_site(*s) for s in sites])
    for r in results:
        print(f"{r['site']}: {r['price']}")

asyncio.run(main())

The agent writes this script, executes it, and gets all three prices in a single step. A conventional agent would need 15-30 individual actions to achieve the same result.

Advanced Pattern 4: Task2UI mode for HTML output rendering

Webwright’s Task2UI mode (added May 2026) renders task results into an interactive HTML web app. This is useful when the task output is a dataset, a comparison, or a report that benefits from visual presentation:

# The agent writes a script that outputs structured data + Task2UI config
# final_script.py
import json
from playwright.sync_api import sync_playwright

def extract_flight_prices(origin: str, dest: str, date: str) -> list[dict]:
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page = browser.new_page()
        page.goto(f"https://google.com/flights?q=flights+from+{origin}+to+{dest}+on+{date}")
        page.wait_for_selector(".gws-flights-results__result", timeout=15000)
        flights = []
        for result in page.query_selector_all(".gws-flights-results__result")[:10]:
            flights.append({
                "airline": result.query_selector(".airline-name").inner_text(),
                "price": result.query_selector(".price").inner_text(),
                "duration": result.query_selector(".duration").inner_text(),
                "stops": result.query_selector(".stops").inner_text(),
            })
        browser.close()
    return flights

if __name__ == "__main__":
    import argparse
    parser = argparse.ArgumentParser()
    parser.add_argument("--origin", required=True)
    parser.add_argument("--dest", required=True)
    parser.add_argument("--date", required=True)
    args = parser.parse_args()
    results = extract_flight_prices(args.origin, args.dest, args.date)
    # Task2UI: output JSON that the showcase dashboard renders as HTML
    print(json.dumps({
        "task": f"Flights from {args.origin} to {args.dest} on {args.date}",
        "results": results,
        "columns": ["airline", "price", "duration", "stops"],
        "sort_by": "price",
        "sort_ascending": True,
    }))

When run with -c task_showcase.yaml, the harness detects the structured JSON output and generates an HTML dashboard with sortable tables, screenshots, and the full execution trace. This turns a one-shot agent run into a shareable, interactive report.

Production considerations

Monitoring: Each run produces a trajectory.json with the full message history and a runtime_errors.jsonl with error logs. Monitor these files for patterns: high format error rates indicate the model is struggling with the action format; repeated self-reflection failures indicate the task is too complex or the success criteria are ambiguous.

Error handling: The environment wraps each command in a 60-second timeout. If a Playwright script hangs (e.g., a page never finishes loading), the subprocess is killed and the error is returned to the agent. The agent then edits the script to add a shorter timeout or a fallback selector.

Rate limiting: The model endpoint wrappers do not include built-in rate limiting. If you are running many tasks in parallel, add a semaphore or queue to avoid hitting API rate limits. The run_one function in cli.py is synchronous — parallel execution requires wrapping it in a thread pool or async runner.

Cost management: The first 50 steps deliver ~82% accuracy. Set step_limit: 50 as a default and only increase for tasks that need it. The self-reflection gate adds one extra LLM call per completion — this is negligible compared to the cost of false-positive completions that require manual rework.

The Results

Metric	Conventional Agent (GPT-5.4, xy-coord)	Webwright (GPT-5.4)	Improvement
Online-Mind2Web accuracy	~33.5%	86.7%	+53.2 pts
Odysseys success rate	33.5%	60.1%	+26.6 pts
Steps per task (Odysseys)	200+	76.1	2.6x fewer
Cost per task (Online-Mind2Web)	~$8-12 (estimated)	$2.37	3-5x cheaper
False-positive completion rate	~18-25% (estimated)	~3-5% (with self-reflection)	4-6x reduction
Reusable output	None	Parameterized Python script	Infinite reuse

What this means for you: If you are building a web agent today, the single highest-leverage change you can make is to switch from a persistent-browser, per-action prediction paradigm to a terminal-native, code-generation paradigm. The model you use matters less than the harness you give it. A GPT-5.4 model in a conventional harness performs worse than a Qwen-3.5-9B model in the Webwright harness on hard tasks (33.5% vs 66.2%).

The terminal-native paradigm is not just about accuracy — it is about durability. Every task produces a reusable script. Every run produces an auditable workspace. The agent’s work does not evaporate when the browser session ends.

What to Watch Out For

1. The self-reflection gate will frustrate you at first. When require_self_reflection_success: true, the agent cannot emit done: true until it has run its final script in a fresh directory and passed the self-reflection judge. This adds 2-3 extra steps to every task. Resist the urge to disable it — it prevents the single most common failure mode (false-positive completion).

2. Playwright scripts are fragile across site redesigns. A script that works today may break tomorrow if the site changes its CSS classes or DOM structure. The agent can re-explore and fix the script, but that costs time and tokens. For production deployments, run scripts on a schedule and alert on failures.

3. History compaction can lose nuance. Every 20 steps, the agent asks the model to summarize the conversation. The summary is lossy — details about specific selectors, error messages, and edge cases may be dropped. If a task requires precise knowledge of earlier failures, increase summary_every_n_steps or disable compaction entirely.

4. Headed mode requires a display server on Linux. If you run Webwright on a headless Linux server, --debug mode will fail because there is no display. Use xvfb-run or set headless: true in the config.

5. Token costs add up on long tasks. A 30-step task with GPT-5.4 costs ~$2.37 on average. A 100-step task can cost $8-12. The first 50 steps deliver 82% accuracy; the next 50 add only 3-4 points. Set step_limit: 50 as a sensible default.

Lesson 1: The terminal-native paradigm is not a niche optimization — it is a fundamentally better decomposition of the web agent problem. The 15-35 point accuracy gains are not from a better model or a better prompt; they are from giving the model the right action space.

Lesson 2: Self-reflection is not optional. Every agent framework should include a verification step before accepting completion. The cost is one extra LLM call per task. The benefit is a 4-6x reduction in false-positive completions.

Lesson 3: Reusable scripts are the killer feature. The ability to parameterize, version-control, and share agent outputs transforms web agents from one-shot experiments into production tools. Build your agent workflows around durable artifacts, not ephemeral browser sessions.

6. The doctor command is your first debugging step. Run python -m webwright.run.cli doctor before anything else. It validates that Playwright is installed, Chromium is available, your API keys are set, and the config files parse correctly. A failing doctor command means every task will fail.

7. Workspace artifacts are not automatically cleaned up. Each run creates a timestamped output directory. After 50 runs, you will have 50 directories. Add a cleanup cron job or set output_dir to a rotating path. The trajectory.json files can be large (2-5 MB per run) — archive or delete old runs.

8. The model’s code quality varies by provider. GPT-5.4 writes cleaner, more robust Playwright scripts than Claude Opus 4.7 on average, but Claude is better at debugging failed scripts. If you are stuck on a task, try switching models mid-task — the workspace artifacts are model-agnostic.

9. Environment variables for API keys are read at import time. If you set OPENAI_API_KEY after importing webwright, the model endpoint will not see it. Set API keys before importing or in a .env file that is loaded at the top of your entrypoint.

10. The keep_last_n_observations setting is your context window budget. Set it to 3-5 for most tasks. Each observation includes the full ARIA snapshot, which can be 10-50k tokens on complex pages. Pruning older observations keeps the context window focused on recent state.

Lesson 4: The workspace is the source of truth, not the conversation history. When debugging a failed run, start with the workspace artifacts (scripts, logs, screenshots), not the LLM message history. The artifacts are concrete and inspectable; the message history is a lossy record of what the model was thinking.

Lesson 5: Task descriptions should include the output format. The model will produce a more useful script if you specify “output as CSV with columns X, Y, Z” or “save screenshots to the screenshots/ directory” in the task prompt. The output format is the contract between the agent and whoever consumes its work.

Lesson 6: The self-reflection judge is only as good as its success criteria. Vague criteria (“the task was completed successfully”) produce unreliable judgments. Specific criteria (“the results.csv file has at least 3 rows, each row has a non-empty VIN column, the price column values are between 25000 and 50000”) produce reliable gates.

Advice for getting started: Clone the repo, run the doctor command, and try one task in debug mode. Watch the agent write and execute Playwright scripts in real time. Then disable debug mode, run the same task, and inspect the workspace artifacts. The difference between the two experiences — watching the browser vs. reading the code — is the entire point of Webwright. Start with a simple task (single page, single output) and work up to multi-site workflows. The reusable script pattern is the payoff — once you have a working script, you never need to run that task manually again.

---

Next in the Open-Source AI Tools Mastery series: OpenMonoAgent.ai