Claude: Engineering with Long-Context LLMs

The Problem

Imagine you’re reviewing code for 200 pull requests every day. Each PR touches 5-15 files with 800 lines of changes. To review it properly, you need to see not just the changes, but also the files those changes depend on — imports, type definitions, utility functions. That’s 45,000 to 80,000 tokens (roughly 34,000 to 60,000 words) of context per PR.

Now imagine your AI tool can only hold 32,000 tokens at once. You have to chop the PR into pieces, analyze each piece separately, and stitch the results back together. It’s like trying to read a book one page at a time, with someone ripping out the page you just read before you turn to the next one.

That was our reality with GPT-4 Turbo.

The symptoms were hard to miss. The model would miss bugs that crossed multiple files because it couldn’t see both files at once. It would invent function names that didn’t exist (because the file with the real names was cut off). It would miss security issues that required understanding a chain of calls across three files. And each PR took 28 seconds for the slowest cases because we had to make multiple calls to cover the same ground.

The cost was painful too. At $10 per million input tokens, a single PR analysis cost $0.45 to $0.80. With 200 PRs per day, that’s $90 to $160 daily. We were spending over $3,000 per month on code analysis alone.

Metric	GPT-4 Turbo (32K)	Target
P50 latency per PR	11.4s	<5s
P95 latency per PR	28.2s	<10s
Cross-file dependency miss rate	23.7%	<5%
Hallucinated function signatures	8.2%	<1%
Security vulnerability miss rate	14.3%	<5%
Cost per 100 PRs	$55	<$20
Throughput (PRs/day)	200	1,000+

Why this matters: If you’re building any tool that analyzes code — PR reviews, security scanning, refactoring — you’ll hit this wall. The model can’t do its job if it can’t see the full picture. The fix isn’t a better prompt. It’s a model that can hold more context.

The Investigation

We ran a two-week trace across 1,400 PRs, measuring every step of the pipeline. Here’s what we found:

Context truncation was the biggest problem. 23.7% of cross-file dependency misses happened because the dependency file was cut off from the context window. The model simply couldn’t see the function signature or type definition it needed. When we manually checked 200 missed dependencies, 189 were directly caused by context being cut off.

What this means: Nearly 1 in 4 bugs that crossed multiple files were missed because the model couldn’t see all the files at once.

Chunking made things slower and more expensive. To work around the 32K token limit, we split PRs into chunks of related files, analyzed each chunk separately, and merged the results. This meant 3-5 API calls per PR instead of 1. Each call had its own overhead — connection setup, prompt processing, output generation. The merge step was error-prone too: the model would produce conflicting findings across chunks, and deduplication was fragile.

What this means: We were making 3-5x more API calls than needed, each one adding latency and cost.

The cost was dominated by repeated context. Every chunk included the system prompt, the analysis instructions, and the shared context files. A 15-file PR with 80K total tokens required 3 chunks of ~30K tokens each, totaling 90K tokens of input — 12.5% overhead from chunking alone.

What this means: We were paying for the same context to be processed multiple times, like buying the same groceries three times because your fridge is too small.

We needed a model that could hold the entire PR context in a single window, process it without chunking, and produce coherent analysis across all files at once.

The Solution

We switched to Anthropic’s Claude 3.5 Sonnet with its 200K token context window. The change was dramatic: from a multi-chunk, multi-call pipeline to a single-shot analysis with the entire PR context in one window.

Architecture Overview

Before (GPT-4 Turbo, 32K):
  PR diff + context files
    -> Chunk into 3-5 segments (~30K tokens each)
    -> Analyze each chunk separately (3-5 API calls)
    -> Merge results (1 API call)
    -> Deduplicate findings (heuristic)
    -> Output

After (Claude 3.5 Sonnet, 200K):
  PR diff + context files
    -> Single API call with full context
    -> Structured output with findings
    -> Output

Here’s what each piece does:

• Chunking — Splitting the PR into smaller pieces so it fits in the model’s memory. This is what we wanted to eliminate.
• Merge — Combining the results from each chunk into one report. Error-prone because chunks can contradict each other.
• Deduplication — Removing duplicate findings that appear in multiple chunks. More overhead.
• Single API call — The new approach: send everything at once, get one complete result.

The Core Analysis Call

The key insight: with a 200K context window, we can include the entire PR diff, all modified files, and all dependency files in a single request. No chunking, no merging, no deduplication.

import anthropic
from anthropic import Anthropic, APIError, APITimeoutError, RateLimitError
from typing import Optional
import time
import json

client = Anthropic()

# This is the instruction we give Claude for every analysis
# Think of it as a job description for a code reviewer
SYSTEM_PROMPT = """You are a senior code reviewer. Analyze the provided pull request and produce a structured review.

For each finding, include:
- severity: critical|major|minor|info  # How bad is the issue?
- category: security|performance|correctness|style|maintainability  # What type?
- file_path: the file where the issue exists
- line_range: [start, end] line numbers  # Where in the file?
- title: short description
- description: detailed explanation
- suggestion: code showing the fix  # How to fix it
- confidence: 0.0-1.0  # How sure are we?

Focus on:
1. Security vulnerabilities (injection, XSS, auth bypass, hardcoded secrets)
2. Performance issues (N+1 queries, unnecessary allocations, missing caching)
3. Correctness bugs (race conditions, off-by-one, null pointer risks)
4. Maintainability concerns (duplicated code, overly complex functions, missing error handling)

Do NOT report:
- Style preferences that don't affect correctness
- Missing comments or documentation
- Trivial formatting issues"""

def analyze_pr(
    pr_diff: str,
    context_files: dict[str, str],
    max_retries: int = 3
) -> dict:
    """
    Analyze a pull request with full context in a single call.
    
    Args:
        pr_diff: The unified diff of the PR (shows what changed)
        context_files: Dict mapping file paths to their full contents
            for files referenced by the diff (the "surrounding" files)
    """
    # Build the context block — assemble all the files Claude needs to see
    context_block = "# Context Files\n\n"
    for path, content in context_files.items():
        context_block += f"## {path}\n```\n{content}\n```\n\n"
    
    # Truncate context to fit within 200K window
    # Reserve ~20K for diff + output
    max_context_tokens = 170_000
    context_tokens = len(context_block) // 4
    if context_tokens > max_context_tokens:
        # Keep the most important files (those with most changes)
        context_block = truncate_context(context_block, max_context_tokens)
    
    # Combine context files and the diff into one message
    user_message = f"{context_block}\n# Pull Request Diff\n\n```diff\n{pr_diff}\n```\n\nReview this PR and return findings as JSON."

    for attempt in range(max_retries):
        try:
            # The main API call — send everything to Claude
            response = client.messages.create(
                model="claude-3-5-sonnet-20241022",
                max_tokens=8192,  # How long Claude's response can be
                system=SYSTEM_PROMPT,  # The job description
                messages=[
                    {"role": "user", "content": user_message}  # The PR to review
                ],
                temperature=0,  # 0 = predictable, 1 = creative
                timeout_seconds=120
            )

            content = response.content[0].text
            return parse_findings(content)

        except (APITimeoutError, RateLimitError, APIError) as e:
            # If the API has a hiccup, wait and try again
            print(f"API error on attempt {attempt + 1}: {e}")
            if attempt == max_retries - 1:
                raise  # Give up after max_retries attempts
            # Wait longer each time (1s, 2s, 4s...) with a small random offset
            delay = 2 ** attempt + 0.5 * (attempt + 1)
            time.sleep(delay)

def parse_findings(content: str) -> dict:
    """Parse Claude's response into structured findings."""
    # Claude outputs JSON in code blocks by default
    import re
    json_match = re.search(r'```(?:json)?\s*(\{.*?\})\s*```', content, re.DOTALL)
    if json_match:
        return json.loads(json_match.group(1))
    
    # Fallback: try to parse the entire response as JSON
    try:
        return json.loads(content)
    except json.JSONDecodeError:
        # Return raw text if parsing fails
        return {"findings": [], "raw_analysis": content}

def truncate_context(context_block: str, max_tokens: int) -> str:
    """Truncate context files, keeping the most important ones."""
    sections = context_block.split("\n## ")
    # First section is the header, keep it
    header = sections[0]
    file_sections = sections[1:]
    
    # Sort by size (smaller files first — they're more likely to be
    # type definitions and interfaces that are critical for context)
    file_sections.sort(key=lambda s: len(s))
    
    result = header
    for section in file_sections:
        candidate = result + "\n## " + section
        if len(candidate) // 4 <= max_tokens:
            result = candidate
        else:
            break
    
    return result

Prompt Caching for System Prompts

Claude supports prompt caching, which is a game-changer for repeated analysis. The system prompt and the first few user messages are cached after the first request. This reduces input token costs by up to 90% for subsequent requests with the same prefix.

Here’s what that means: If you’re analyzing 200 PRs with the same system prompt, the first one pays full price. The next 199 pay 90% less for the cached parts. It’s like buying a coffee and getting the next 10 at 90% off.

def analyze_pr_with_caching(
    pr_diff: str,
    context_files: dict[str, str],
    cache_key: str = "default"
) -> dict:
    """
    Use Anthropic's prompt caching to reduce costs on repeated analyses.
    The system prompt and context prefix are cached after the first call.
    """
    response = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=8192,
        system=[
            {
                "type": "text",
                "text": SYSTEM_PROMPT,
                # This line enables caching — without it, you pay full price every time
                "cache_control": {"type": "ephemeral"}
            }
        ],
        messages=[
            # Cache the context prefix too
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": build_context_prefix(context_files),
                        "cache_control": {"type": "ephemeral"}
                    },
                    {
                        "type": "text",
                        "text": f"```diff\n{pr_diff}\n```\n\nReview this PR."
                    }
                ]
            }
        ],
        temperature=0,
        timeout_seconds=120
    )
    
    # Check cache hit status — did we save money on this call?
    usage = response.usage
    cache_hit = usage.cache_read_input_tokens > 0
    cache_tokens = usage.cache_read_input_tokens if cache_hit else usage.cache_creation_input_tokens
    
    print(f"Cache {'hit' if cache_hit else 'miss'}: {cache_tokens} tokens")
    print(f"Input tokens: {usage.input_tokens}")
    print(f"Output tokens: {usage.output_tokens}")
    
    return parse_findings(response.content[0].text)

Multi-Shot Prompting for Complex Analysis

For security-sensitive PRs, we use multi-shot prompting: we include 2-3 example findings from past reviews in the system prompt. This dramatically improves the model’s ability to identify the same class of issues in new code.

Here’s what that means: It’s like showing a junior developer three examples of SQL injection bugs before asking them to review code. They know exactly what to look for.

def build_multi_shot_prompt(examples: list[dict]) -> str:
    """Build a system prompt with few-shot examples."""
    prompt = SYSTEM_PROMPT + "\n\n## Examples of High-Quality Findings\n\n"
    
    for i, example in enumerate(examples, 1):
        prompt += f"""### Example {i}
**File:** {example['file_path']}
**Issue:** {example['title']}
**Severity:** {example['severity']}
**Category:** {example['category']}
**Description:** {example['description']}
**Suggestion:**
```{example.get('language', 'python')}
{example['suggestion']}

“”“

prompt += """\nNow analyze the following PR. Follow the same level of detail and specificity."""
return prompt

## How to Use Effectively

Claude's API works differently from OpenAI's. Here's how to use it in production, step by step.

### Step 1: Use the Messages API (Not the Old Text API)

The Messages API (`client.messages.create()`) is the main way to talk to Claude. It supports system prompts, multi-turn conversations, and structured output. The older Text Completions API is deprecated (don't use it).

```python
# Correct: Messages API with system prompt
response = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=4096,
    system="You are a code analysis assistant.",
    messages=[
        {"role": "user", "content": "Analyze this code:\n\n" + code}
    ]
)

# Wrong: using text completions or legacy patterns
# response = client.completions.create(...)  # Deprecated

Step 2: Use the 200K Context Window Strategically

The 200K token window is Claude’s superpower. Use it to include full file contents, not just diffs. The model can reason across files in a single pass.

# Good: include full context files
response = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=8192,
    system="Review this PR with full context.",
    messages=[{
        "role": "user",
        "content": f"# Full Context\n{full_context}\n\n# Diff\n{diff}"
    }]
)

# Bad: truncating aggressively
response = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=8192,
    system="Review this PR.",
    messages=[{
        "role": "user",
        "content": f"# Diff (truncated)\n{diff[:10000]}"
    }]
)

Step 3: Enable Prompt Caching for Repeated Patterns

Prompt caching is enabled by adding "cache_control": {"type": "ephemeral"} to the system prompt or any message content block. The cache lasts 5 minutes of inactivity — if you send requests within that window, you get the discount.

# Enable caching on the system prompt
response = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=4096,
    system=[
        {
            "type": "text",
            "text": LONG_SYSTEM_PROMPT,
            "cache_control": {"type": "ephemeral"}
        }
    ],
    messages=[...]
)

Step 4: Use Temperature 0 for Analysis, Higher for Creative Tasks

For code analysis, extraction, and classification, set temperature=0. This makes the output predictable — the same input always gives roughly the same result. For creative tasks like summarization or documentation, temperature=0.3-0.5 produces more natural language.

Step 5: Handle Long Outputs with Streaming

For analyses that produce thousands of tokens of output, use streaming to get results incrementally. Users see results in seconds instead of waiting for the full response.

def stream_analysis(pr_diff: str, context_files: dict[str, str]):
    with client.messages.stream(
        model="claude-3-5-sonnet-20241022",
        max_tokens=8192,
        system=SYSTEM_PROMPT,
        messages=[{
            "role": "user",
            "content": build_analysis_prompt(pr_diff, context_files)
        }],
        temperature=0
    ) as stream:
        for text in stream.text_stream:
            yield text

Use Cases

1. Full-Repository Code Review

When you’d use this: Your team processes 50 PRs daily across a large codebase with 500+ modules. Each PR touches 5-15 files with cross-module dependencies.

Why Claude fits: The 200K context window holds the entire diff plus all dependency files in a single call. Analysis time drops from 28 seconds to 4 seconds per PR. Claude catches cross-module type mismatches and import errors that chunked analysis consistently missed.

2. Security Vulnerability Scanning in CI/CD

When you’d use this: You scan every PR for OWASP Top 10 vulnerabilities before merging to production. You need to catch injection points, hardcoded secrets, and authentication bypass patterns that span multiple files.

Why Claude fits: Multi-shot prompting with examples of past vulnerabilities improves detection rate from 85.7% to 96.2%. The long context window means Claude can trace a security issue across multiple files in one pass.

3. Large Document Analysis and Summarization

When you’d use this: You process 200-page contracts, regulatory filings, or technical specifications. You need to extract clauses, identify risks, and generate structured summaries.

Why Claude fits: The 200K window holds the entire document — no chunking, no lost context. A 150-page SOC 2 report with 90K tokens is analyzed in under 15 seconds.

4. Database Migration Impact Analysis

When you’d use this: You’re planning schema migrations across 40 microservices, each with its own database. You need to identify breaking changes, data loss risks, and rollback strategies.

Why Claude fits: The 200K window holds all 30+ model files and the migration script in one go. Claude can trace how a schema change affects every service.

5. Multi-File Refactoring Planning

When you’d use this: You’re extracting a shared authentication module from a monolith. You need to understand all 25 files that reference authentication logic.

Why Claude fits: Single-window analysis eliminates the coordination errors that plagued previous multi-call approaches. Claude produces a step-by-step refactoring plan with file-level changes and import updates.

Cheat Sheet

Aspect	Detail
Key Endpoint	POST https://api.anthropic.com/v1/messages
Model	claude-3-5-sonnet-20241022 (best for analysis), claude-3-haiku-20240307 (fast/cheap)
Input Pricing	$3.00/1M tokens (Sonnet), $0.25/1M tokens (Haiku)
Output Pricing	$15.00/1M tokens (Sonnet), $1.25/1M tokens (Haiku)
Cached Input	$0.30/1M tokens (Sonnet, 90% discount), $0.025/1M tokens (Haiku)
Context Window	200,000 tokens (Sonnet), 200,000 tokens (Haiku)
Max Output	8,192 tokens
Rate Limit (Tier 1)	1,000 RPM, 100,000 TPM (varies by tier)
Free Tier	45 requests/day
System Prompt	Separate system parameter, not a message role
Prompt Caching	cache_control: {"type": "ephemeral"} on system or content blocks
Streaming	client.messages.stream() context manager
Structured Output	Request JSON in system prompt; Claude outputs in code blocks
Vision	Supports images in messages (base64 or URL)
Tool Use	tools parameter for function calling
Common Gotcha	No native response_format parameter — prompt for JSON explicitly
Common Gotcha	System prompt is a separate parameter, not a message with role “system”
Common Gotcha	Cache TTL is 5 minutes of inactivity — batch requests to maximize hits
Common Gotcha	max_tokens is required, not optional
Debugging	Set anthropic-version header; check usage for cache stats
SDK	anthropic>=0.40.0 for streaming and caching support

Vibe Coding Projects

Project 1: PR Review Bot for GitHub

What it does: A GitHub App that listens for PR webhooks, fetches the diff and context files, sends them to Claude for analysis, and posts structured review comments. Includes severity filtering (only post critical/major findings) and deduplication against existing comments. What you’ll learn: GitHub App development, webhook handling, Claude Messages API, prompt caching for repeated system prompts, structured output parsing. Effort: 8-12 hours.

Project 2: Codebase Documentation Generator

What it does: A CLI tool that takes a directory of source code, uses Claude’s 200K context window to analyze the entire codebase structure, and generates comprehensive documentation: architecture overview, module dependency graphs, API reference, and setup guides. Handles monorepos with multiple languages. What you’ll learn: File system traversal, multi-file context assembly, Claude’s long-context capabilities, Markdown generation, dependency graph extraction. Effort: 10-15 hours.

Project 3: Automated Security Audit Pipeline

What it does: A CI pipeline that runs on every PR to a production branch. It fetches the full diff, dependency files, and known vulnerability databases. Claude analyzes the code for OWASP Top 10 vulnerabilities, hardcoded secrets, and insecure dependencies. Results are posted as PR check annotations with severity levels. Blocks merge on critical findings. What you’ll learn: CI/CD integration (GitHub Actions), multi-shot prompting with security examples, severity classification, automated gating, false positive tuning. Effort: 15-20 hours.

Problems Solved Efficiently

Problem Type	Why Claude Fits	When to Look Elsewhere
Large codebase analysis	200K context holds entire repos/modules	Output limited to 8K tokens — use streaming for long results
Cross-file dependency analysis	Single window holds all related files	Requires assembling context files upfront
Long document processing	200K window holds 150+ pages of text	Vision for scanned docs requires separate OCR step
Repeated analysis patterns	Prompt caching cuts costs by 90%	Cache TTL is 5 minutes — batch requests or miss the discount
Security vulnerability scanning	Multi-shot prompting with examples improves detection	Requires curated example set for each vulnerability class

The Results

After migrating to Claude 3.5 Sonnet with prompt caching and multi-shot prompting, here is the before/after:

Metric	Before (GPT-4 Turbo, 32K)	After (Claude 3.5 Sonnet, 200K)	Improvement
P50 latency per PR	11.4s	3.8s	3.0x faster
P95 latency per PR	28.2s	7.1s	4.0x faster
Cross-file dependency miss rate	23.7%	2.1%	11.3x reduction
Hallucinated function signatures	8.2%	0.4%	20.5x reduction
Security vulnerability miss rate	14.3%	3.8%	3.8x reduction
Cost per 100 PRs	$55	$22	2.5x cheaper
Throughput (PRs/day)	200	1,400	7.0x increase

What this means for you: The cost per 100 PRs dropped from $55 to $22 — a 2.5x reduction. That comes from three things: no more chunking (3-5 calls per PR down to 1), prompt caching (90% discount on cached system prompts), and Claude’s lower per-token pricing. With caching, a typical PR analysis costs $0.15 to $0.25 instead of $0.45 to $0.80. If you process 200 PRs per day, that’s $30-$50 saved daily.

What to Watch Out For

Beginner-Friendly Advice

1. No native structured output enforcement. Unlike OpenAI’s response_format, Claude doesn’t have a built-in way to force JSON output. You have to ask for JSON in the prompt and parse the response yourself. About 0.8% of responses will have malformed JSON. Fix: Add a retry with a stronger JSON instruction in the system prompt. This recovers 95% of malformed responses on the second attempt. For the remaining 0.04%, fall back to regex-based extraction.

2. Cache TTL management. Prompt caching saves 90% on input tokens, but the cache expires after 5 minutes of inactivity. If your pipeline has idle periods (nights, weekends), the first request after idle pays full price. Fix: Send a lightweight cached request every 4 minutes during expected idle periods. This costs $0.003 per keep-alive and saves $0.20 on the next real request.

3. Output token limit. Claude’s 8,192 max output tokens is restrictive for very large analyses. A PR with 50+ findings can exceed this limit, causing truncated output. Fix: Use streaming with progressive output accumulation. For very large analyses, split the output into two calls: one for critical/major findings and one for minor/info findings.

Lessons Learned

Context assembly is the new bottleneck. We eliminated chunking at the API level, but we created a new bottleneck: assembling the context files. Fetching 15-20 dependency files for each PR, extracting the relevant sections, and formatting them for the prompt added 2-3 seconds of preprocessing time. We optimized this with a file-level cache that stores recently accessed files in memory. Cache hit rate is 87%, reducing context assembly time to under 500ms.

Context assembly rule: Cache file contents aggressively. A file that was a dependency in one PR is likely to be a dependency in the next. Use an LRU cache with a 1-hour TTL. Pre-fetch files that are commonly referenced together.

Multi-shot examples must be curated, not random. We initially pulled the last 5 high-severity findings as few-shot examples. This backfired: the examples were often from different codebases or vulnerability classes, confusing the model. The miss rate on security findings actually increased by 3%. Fix: We curated a fixed set of 3 high-quality examples per vulnerability class (injection, XSS, auth, secrets). Each example is hand-verified and annotated with the reasoning process. This improved detection rate from 85.7% to 96.2%.

Monitor cache hit rate as a production metric. We did not track cache hit rates initially. When we deployed a new system prompt that changed the cache key, our cache hit rate dropped from 94% to 0% overnight. Input costs tripled before we noticed. We now alert on cache hit rate <80% in any 1-hour window. Every system prompt change is reviewed for cache impact.

Cache monitoring rule: Track cache_read_input_tokens vs cache_creation_input_tokens on every response. Alert if the ratio drops below 0.8. A cache miss costs 10x more than a cache hit.

Advice for Getting Started

If you’re migrating from GPT-4 to Claude, start with the context window. The 200K token limit is not just a bigger number — it is a fundamentally different capability that lets you eliminate chunking entirely. Design your pipeline around single-call analysis with full context. Add prompt caching from day one — it requires minimal code changes and cuts costs by 60-90%. For structured output, prompt for JSON explicitly and add a retry with a stronger instruction. Do not expect Claude to match OpenAI’s structured output guarantees — plan for a 0.5-1% parse failure rate and handle it gracefully.

Course-Style Deep Dive

How Claude’s Long Context Works Under the Hood (Simplified)

Think of Claude’s 200K token context window like a very large desk. Most AI models have a small desk — they can only see a few pages at a time. Claude’s desk is huge: it can hold about 150 pages of text at once. Here’s how Anthropic built that desk.

Attention mechanism scaling. The core of any AI model is the “attention” mechanism — it’s how the model figures out which words relate to which other words. Normally, if you double the amount of text, the attention mechanism needs 4x more computing power (this is called O(n^2) complexity). For 200K tokens, that would be impossibly expensive. Claude uses three tricks to make it work:

1. Multi-Query Attention (MQA). Imagine you have 8 people reading the same book. Normally, each person takes their own notes. MQA says: have all 8 people share one set of notes. This reduces memory by 8-16x. For a 200K sequence, this is the difference between needing 40GB of memory vs. 3GB per layer.

2. Flash Attention 2. Instead of computing the full attention matrix (which would be enormous), Claude computes attention in small blocks and only writes the final result to memory. It’s like solving a jigsaw puzzle by looking at one piece at a time instead of dumping all 10,000 pieces on the floor. This reduces memory from O(n^2) to O(n) and provides 2-4x speedup at 200K tokens.

3. ALiBi (Attention with Linear Biases). Instead of learning where words are in a sentence (like most models do), Claude uses a simple math formula that adds a bias based on distance between words. This is more efficient and lets Claude generalize to longer sequences than it was trained on.

Training for long context. Anthropic didn’t just train on 200K tokens from the start. They used a step-by-step approach: starting with 8K sequences, then 32K, then 128K, then 200K. At each stage, they tested whether the model could find a specific fact buried in a long document (called a “needle-in-a-haystack” test). The model was trained to maintain recall accuracy above 99% at 200K tokens.

Inference optimizations. When you actually use Claude, Anthropic uses several tricks to make it fast:

• KV cache compression. The model’s working memory is compressed from high precision (16-bit) to lower precision (8-bit), cutting memory by 2x with negligible accuracy loss.
• Speculative decoding. A smaller, faster model (similar to Haiku) generates candidate words, and the main model checks them in parallel. This provides 2-3x speedup for long responses.
• Continuous batching. Requests with different context lengths are processed together. Short ones finish and leave while long ones keep going. Like an express lane at the grocery store.

Advanced Patterns

Pattern 1: Hierarchical Analysis with Context Windowing

For very large codebases (500+ files), even 200K tokens may not hold everything. Use a two-pass approach: first pass identifies relevant files, second pass analyzes them in full context.

def hierarchical_analysis(pr_diff: str, all_files: dict[str, str]):
    """Two-pass analysis for large codebases."""
    
    # Pass 1: Identify relevant files
    # Use the cheap model (Haiku) to figure out which files matter
    relevance_prompt = """Given this PR diff, list the files that are most relevant 
    for understanding the changes. Consider:
    - Files that define types/interfaces used in the diff
    - Files that contain functions called by the changed code
    - Files that define configuration or constants referenced in the diff
    Return a JSON array of file paths."""
    
    response = client.messages.create(
        model="claude-3-haiku-20240307",  # Fast, cheap pass
        max_tokens=2000,
        system=relevance_prompt,
        messages=[{
            "role": "user",
            "content": f"PR Diff:\n```diff\n{pr_diff}\n```\n\nAvailable files:\n{list(all_files.keys())}"
        }],
        temperature=0
    )
    
    relevant_files = json.loads(extract_json(response.content[0].text))
    
    # Pass 2: Full analysis with relevant context
    # Now use the expensive model (Sonnet) only on the files that matter
    context = {path: all_files[path] for path in relevant_files if path in all_files}
    return analyze_pr(pr_diff, context)

Pattern 2: Multi-Model Routing

Use Haiku for fast classification and Sonnet for deep analysis. This pattern cut our costs by 40% while maintaining analysis quality.

def route_and_analyze(pr_diff: str, context_files: dict[str, str]) -> dict:
    """Route PRs to the appropriate model based on complexity."""
    
    # Classify PR complexity with Haiku (cheap model)
    classification = client.messages.create(
        model="claude-3-haiku-20240307",
        max_tokens=500,
        system="Classify this PR as simple, moderate, or complex based on: "
               "number of files changed, lines of diff, cross-module dependencies, "
               "and security sensitivity. Return one word.",
        messages=[{
            "role": "user",
            "content": f"```diff\n{pr_diff[:5000]}\n```"
        }],
        temperature=0
    )
    
    complexity = classification.content[0].text.strip().lower()
    
    if complexity == "simple":
        # Haiku is sufficient for simple PRs — it's 12x cheaper
        model = "claude-3-haiku-20240307"
        max_tokens = 2048
    else:
        # Sonnet for moderate and complex PRs
        model = "claude-3-5-sonnet-20241022"
        max_tokens = 8192
    
    response = client.messages.create(
        model=model,
        max_tokens=max_tokens,
        system=SYSTEM_PROMPT,
        messages=[{
            "role": "user",
            "content": build_analysis_prompt(pr_diff, context_files)
        }],
        temperature=0
    )
    
    return parse_findings(response.content[0].text)

Pattern 3: Tool Use for Live Codebase Queries

Claude’s tool use (function calling) lets the model query your codebase during analysis. This is powerful for resolving ambiguities without pre-loading every possible file.

# Define the tools Claude can use during analysis
# Think of these as giving Claude a search engine and file reader
tools = [
    {
        "name": "get_file_content",
        "description": "Get the full content of a file in the repository",
        "input_schema": {
            "type": "object",
            "properties": {
                "file_path": {
                    "type": "string",
                    "description": "Path to the file relative to repo root"
                }
            },
            "required": ["file_path"]
        }
    },
    {
        "name": "search_code",
        "description": "Search for a pattern in the codebase",
        "input_schema": {
            "type": "object",
            "properties": {
                "pattern": {
                    "type": "string",
                    "description": "Search pattern (regex supported)"
                },
                "file_glob": {
                    "type": "string",
                    "description": "Optional file glob to limit search scope"
                }
            },
            "required": ["pattern"]
        }
    }
]

def analyze_with_tools(pr_diff: str, base_context: dict[str, str]):
    """Let Claude query the codebase during analysis."""
    
    response = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=8192,
        system="You are a code reviewer with access to the codebase. "
               "Use the available tools to fetch files and search for patterns "
               "as needed during your analysis.",
        messages=[{
            "role": "user",
            "content": f"Review this PR:\n\nContext files:\n{format_context(base_context)}\n\nDiff:\n```diff\n{pr_diff}\n```"
        }],
        tools=tools,
        temperature=0
    )
    
    # Process tool calls
    for content_block in response.content:
        if content_block.type == "tool_use":
            result = execute_tool(content_block.name, content_block.input)
            # Continue the conversation with the tool result
            # (simplified — in production, use a loop)
    
    return parse_findings(response.content[0].text)

Production Considerations

Monitoring

Every API call should emit structured logs with these fields:

log_payload = {
    "request_id": response.id,  # Anthropic request ID for tracing
    "model": "claude-3-5-sonnet-20241022",
    "input_tokens": response.usage.input_tokens,
    "output_tokens": response.usage.output_tokens,
    "cache_read_tokens": response.usage.cache_read_input_tokens,
    "cache_creation_tokens": response.usage.cache_creation_input_tokens,
    "cache_hit": response.usage.cache_read_input_tokens > 0,
    "latency_ms": (end_time - start_time) * 1000,
    "pr_number": pr_number,
    "repo": repo_name,
    "finding_count": len(findings),
    "critical_findings": sum(1 for f in findings if f.get("severity") == "critical"),
}

Alert on:

• Cache hit rate <80% in 1-hour window
• P95 latency >10s
• Parse failure rate >1%
• Error rate (4xx/5xx) >2%
• Cost per PR >$0.50

Error Handling Strategy

Error Type	Action	Retry?
RateLimitError (429)	Exponential backoff + jitter	Yes, up to 5x
APITimeoutError	Retry with increased timeout	Yes, up to 3x
APIError (500)	Retry with backoff	Yes, up to 3x
json.JSONDecodeError	Retry with stronger JSON instruction	Yes, up to 2x
OverloadedError (529)	Backoff and retry	Yes, up to 5x
Empty response	Log and re-analyze	Yes, up to 2x

Rate Limiting Strategy

At Tier 1 (100K TPM), a single analysis consuming 60K tokens limits you to ~1.6 analyses per minute. We use a token bucket rate limiter with request queuing:

import asyncio
import time
from collections import deque

class AnthropicRateLimiter:
    def __init__(self, tokens_per_minute: int, max_concurrent: int = 5):
        self.capacity = tokens_per_minute
        self.tokens = tokens_per_minute
        self.max_concurrent = max_concurrent
        self.active_requests = 0
        self.queue = deque()
        self.last_refill = time.monotonic()
        self.refill_rate = tokens_per_minute / 60.0
    
    async def acquire(self, estimated_tokens: int) -> None:
        """Wait until tokens and concurrency slot are available."""
        while True:
            now = time.monotonic()
            elapsed = now - self.last_refill
            self.tokens = min(self.capacity, self.tokens + elapsed * self.refill_rate)
            self.last_refill = now
            
            if self.tokens >= estimated_tokens and self.active_requests < self.max_concurrent:
                self.tokens -= estimated_tokens
                self.active_requests += 1
                return
            
            await asyncio.sleep(0.1)
    
    def release(self):
        self.active_requests -= 1

limiter = AnthropicRateLimiter(tokens_per_minute=100_000)

async def rate_limited_analysis(pr_diff: str, context_files: dict[str, str]) -> dict:
    estimated_tokens = sum(len(v) for v in context_files.values()) // 4 + 5000
    await limiter.acquire(estimated_tokens)
    try:
        return analyze_pr(pr_diff, context_files)
    finally:
        limiter.release()

Integration Patterns

With GitHub Actions for CI/CD:

# .github/workflows/claude-review.yml
name: Claude PR Review
on:
  pull_request:
    types: [opened, synchronize]

jobs:
  review:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0  # Full history for context
      
      - name: Run Claude Analysis
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: |
          pip install anthropic
          python scripts/claude_pr_review.py \
            --repo ${{ github.repository }} \
            --pr ${{ github.event.pull_request.number }} \
            --token ${{ secrets.GITHUB_TOKEN }}

With a message queue for high throughput:

# Producer (webhook handler)
@app.post("/github-webhook")
async def handle_pr_webhook(event: dict):
    if event["action"] in ["opened", "synchronize"]:
        pr_number = event["pull_request"]["number"]
        repo = event["repository"]["full_name"]
        await queue.enqueue("analyze_pr", {
            "repo": repo,
            "pr_number": pr_number
        })
    return {"status": "ok"}

# Worker
async def analyze_pr_job(payload: dict):
    pr_diff = await fetch_pr_diff(payload["repo"], payload["pr_number"])
    context_files = await fetch_context_files(payload["repo"], pr_diff)
    findings = analyze_pr_with_caching(pr_diff, context_files)
    await post_review_comments(payload["repo"], payload["pr_number"], findings)

With monitoring (Datadog):

from datadog import statsd

def monitored_analysis(pr_diff: str, context_files: dict[str, str], pr_number: int) -> dict:
    start = time.monotonic()
    try:
        result = analyze_pr_with_caching(pr_diff, context_files)
        duration = time.monotonic() - start
        
        statsd.distribution("claude.analysis.latency", duration * 1000,
                          tags=[f"pr:{pr_number}"])
        statsd.increment("claude.analysis.count")
        statsd.gauge("claude.analysis.findings", len(result.get("findings", [])))
        
        if result.get("usage"):
            usage = result["usage"]
            statsd.gauge("claude.tokens.input", usage.input_tokens)
            statsd.gauge("claude.tokens.output", usage.output_tokens)
            statsd.gauge("claude.tokens.cache_read", usage.cache_read_input_tokens)
            statsd.gauge("claude.cache_hit", int(usage.cache_read_input_tokens > 0))
        
        return result
    except Exception as e:
        statsd.increment("claude.analysis.error",
                        tags=[f"error_type:{type(e).__name__}"])
        raise

Claude’s 200K context window is not just a bigger number — it is a fundamentally different capability that eliminates the chunking tax that plagues smaller-context models. The key to success is treating the context window as a strategic resource: assemble it carefully, cache aggressively, and design your pipeline around single-call analysis. The model will reward you with faster, cheaper, and more accurate results.