Design a Coding Agent (Claude Code / GitHub Copilot / Cursor)

TL;DR. A coding agent is three products sharing one backend: sub-200 ms autocomplete on a small model, conversational chat grounded in the repo, and an autonomous ReAct loop that edits files and runs shell commands for minutes per task. GitHub Copilot handles 400M+ completion requests per day at ~8,000 QPS peak¹, Cursor crossed $2B ARR by early 2026², and agent benchmarks on real GitHub issues have climbed rapidly: Devin scored 13.86% on the original SWE-bench in March 2024³, and Claude Opus 4.7 now leads SWE-bench Verified at 87.6% (April 2026)⁴. The pivotal trade-off is model tiering: a cheap small model for autocomplete, a frontier model for chat, and frontier-plus-tools for the autonomous loop, yielding roughly an order-of-magnitude cost reduction over a single-model deployment based on published model pricing differences⁵.

Learning Objectives

• Design a three-mode coding agent (autocomplete, chat, autonomous loop) with distinct latency and cost budgets for each mode
• Architect a hybrid repo-context retrieval layer combining embeddings, tree-sitter symbol graphs, and git-awareness
• Justify per-session Firecracker microVM isolation for untrusted agent tool calls and size the warm pool
• Estimate capacity for 1M concurrent sessions across three modes with different token economics
• Stream generated diffs to the editor with atomic apply and one-key rollback
• Build an eval harness on SWE-bench Verified that gates every model rollout before production

Intuition

A coding agent looks like autocomplete with extra steps. Type a few characters, get a suggestion. Easy at 10 users. At one million concurrent sessions it collapses, and the reason is that you are actually running three different products behind one interface.

Autocomplete must return ghost text before the developer types past it. That means sub-200 ms end-to-end, which means a small model, aggressive prompt caching, and HTTP/2 stream-reset for the 45-50% of requests the developer cancels by typing through¹. Chat runs at conversational latency on a frontier model with retrieved repo context prepended. The autonomous loop is the hardest: a multi-step agent that reads files, edits them, runs shell commands, observes results, and iterates for minutes on a single prompt, all inside a sandbox that prevents it from destroying the host.

The naive approach (one frontier model for everything) fails on cost and latency simultaneously. A single GPT-4-class completion costs roughly 100x what a small-model autocomplete costs⁵. And the autonomous loop introduces a new failure mode: the agent can write code that exploits its own sandbox, install malicious packages, or enter an infinite retry loop burning tokens until someone notices.

The insight that unlocks the design: treat each mode as a separate product with its own model tier, latency SLO, and safety envelope, but share the repo-context retrieval index, the diff-application path, and the eval harness across all three.

Requirements

Clarifying Questions

• Q: Which modes do we ship at launch? Assume: All three (autocomplete, chat, autonomous loop). Each has a separate latency budget.
• Q: Cloud service or local-first CLI? Assume: Cloud SaaS (like Copilot/Cursor) with enterprise option for on-prem indexing (like Claude Code).
• Q: Do we own the models or route to third-party providers? Assume: We operate inference for autocomplete (small tuned model) and route chat/autonomous to a frontier provider API.
• Q: Can the agent run arbitrary shell commands? Assume: Yes, inside a per-session sandbox with network allowlist and resource ceilings.
• Q: Do we index private repos server-side? Assume: Yes for SaaS; enterprise customers can opt for client-side-only indexing.
• Q: Multi-tenant SaaS or single-tenant? Assume: Multi-tenant with strict isolation. Enterprise tier gets dedicated sandbox pools.

Functional Requirements

• Inline autocomplete: ghost-text completion triggered by typing, dismissable with Escape
• Chat: conversational panel grounded in the open file, selection, and repo index
• Autonomous loop: multi-step agent that reads, edits, runs tests, and iterates until done
• Diff review: every agent edit previewed as a diff the developer accepts or rejects per hunk
• Repo indexing: background job keeps embedding + symbol index in sync with the current branch

Non-Functional Requirements

• Sessions: 1M concurrent across all modes
• Autocomplete latency: p99 < 200 ms from keystroke to ghost text¹
• Chat latency: first-token < 1 s; streaming at 40+ tokens/sec
• Autonomous task duration: up to 5 minutes; continuous progress visible
• Sandbox isolation: agent shell commands cannot escape to host, hit arbitrary internet, or exceed CPU/memory/time budgets
• Cost per autocomplete: < $0.0002 (supports ~$20/month subscription at 100K completions/month)

Capacity Estimation

Key ratios: autocomplete dominates request count but is cheapest per request. Chat dominates daily cost. Prompt caching (90% savings on repeated prefixes⁶) is the single biggest cost lever for chat and autonomous modes.

API and Data Model

API Design

POST /v1/complete
  Body: { "file": "src/main.ts", "cursor": 142, "prefix": "...", "suffix": "...", "lang": "typescript" }
  Returns: 200 { "suggestion": "...", "tokens": 20, "model": "small-v3" }
  Target: p99 200 ms

POST /v1/chat (SSE)
  Body: { "session_id": "uuid", "messages": [...], "context_refs": ["file://src/auth.ts"] }
  Returns: SSE stream of token deltas + tool_call events

POST /v1/task (SSE)
  Body: { "prompt": "Fix the failing test in auth.ts", "repo": "repo-id", "permissions": ["read", "edit", "run_tests"] }
  Returns: SSE stream of thought, tool_call, tool_result, diff, done events

POST /v1/diff/apply
  Body: { "hunks": [...], "workspace_id": "uuid" }
  Returns: 200 { "rollback_token": "uuid" }

POST /v1/diff/rollback
  Body: { "rollback_token": "uuid" }
  Returns: 204

Rate limiting enforces per-tenant token budgets in Redis. A runaway autonomous task hits a spend ceiling and pauses for human approval.

Data Model

-- Sessions (Redis hot + PostgreSQL cold)
CREATE TABLE sessions (
  session_id    UUID PRIMARY KEY,
  user_id       UUID NOT NULL,
  repo_id       UUID,
  mode          TEXT NOT NULL,  -- autocomplete | chat | autonomous
  model         TEXT NOT NULL,
  token_spend   BIGINT DEFAULT 0,
  created_at    TIMESTAMPTZ DEFAULT now()
);

-- Task state (PostgreSQL + object storage for large payloads)
CREATE TABLE task_state (
  task_id       UUID PRIMARY KEY,
  session_id    UUID NOT NULL,
  status        TEXT NOT NULL,  -- running | waiting_approval | succeeded | failed
  scratchpad    TEXT,           -- agent chain-of-thought (summarized)
  sandbox_id    TEXT,
  created_at    TIMESTAMPTZ DEFAULT now()
);

-- Repo index chunks (vector DB + PostgreSQL metadata)
CREATE TABLE repo_chunks (
  chunk_id      UUID PRIMARY KEY,
  repo_id       UUID NOT NULL,
  commit_sha    TEXT NOT NULL,
  file_path     TEXT NOT NULL,
  symbol_name   TEXT,
  embedding     vector(1536),
  last_indexed  TIMESTAMPTZ
);

-- Diff log (training signal)
CREATE TABLE diff_log (
  diff_id       UUID PRIMARY KEY,
  task_id       UUID,
  hunks         JSONB,
  accepted      BOOLEAN,
  created_at    TIMESTAMPTZ DEFAULT now()
);

High-Level Architecture

flowchart LR
    IDE[Editor Client<br/>VS Code / CLI] --> GW[API Gateway<br/>auth, rate limit, route]
    GW --> AC[Autocomplete Service<br/>small model, warm GPU pool]
    GW --> Chat[Chat Service<br/>frontier model + retrieval]
    GW --> Orch[Agent Orchestrator<br/>ReAct loop]
    Chat --> Ret[Repo-Context Retrieval<br/>embeddings + symbols + git]
    Orch --> Ret
    Orch --> SCP[Sandbox Control Plane]
    SCP --> Pool[Warm MicroVM Pool<br/>Firecracker]
    Pool --> VM[Per-session VM<br/>RO repo, tmpfs, allowlist]
    AC --> IDE
    Chat --> IDE
    Orch --> IDE
    Orch --> Diff[Diff Applicator<br/>atomic apply + rollback]

Autocomplete path: The IDE debounces keystrokes, sends prefix + suffix to the gateway, which forwards to a warm small-model GPU pool. The model returns ghost text within 200 ms. HTTP/2 stream-reset handles cancellation without tearing down the TCP connection¹.

Chat path: The gateway loads conversation history, queries the retrieval service for relevant repo chunks, prepends them to the prompt, and streams tokens from a frontier model over SSE.

Autonomous path: The orchestrator runs a ReAct loop⁷: reason (LLM generates a tool call), act (execute in sandbox), observe (capture stdout/exit code), repeat. Each iteration updates the scratchpad. A risk gate requires human approval for destructive actions.

sequenceDiagram
    participant IDE
    participant Proxy as Edge Proxy (HTTP/2)
    participant Model as Small Model Pool
    IDE->>Proxy: POST /v1/complete (prefix, suffix, lang)
    Note over IDE,Proxy: debounce + 50 ms network
    Proxy->>Model: forward on persistent HTTP/2 stream
    Note over Proxy,Model: 80-120 ms inference
    Model-->>Proxy: ghost text tokens
    Proxy-->>IDE: stream suggestion
    Note over IDE: 45-50% cancelled by type-through
    IDE->>Proxy: RST_STREAM (cancel)
    Proxy->>Model: propagate cancellation

Deep Dives

Repo-context retrieval architecture

The retrieval layer is the product differentiator. Embeddings alone miss structural questions like “where is authenticate called?” A hybrid retriever combines three signals.

Signal 1: Semantic embeddings. Code is chunked at roughly function-level granularity and embedded. On each commit, only changed files are re-embedded. The vector DB serves approximate nearest-neighbor queries against the developer’s current query or cursor context.

Signal 2: Tree-sitter symbol graph. Aider parses every source file with tree-sitter to extract definition and reference tags, builds a directed multigraph where each file is a node and each referencer-to-definer edge carries a weight, then runs PageRank with personalization biased toward files in the active chat⁸. Identifiers mentioned by the user get a 10x multiplier; identifiers in chat-active files get 50x⁹. The top-ranked symbols render into a token-budgeted tree (default 1,024 tokens)⁸.

Signal 3: Git awareness. Recently changed files and files in the current PR get a score boost. This mirrors what human developers reach for: the code they are already thinking about.

flowchart LR
    subgraph Indexer
      Commit[Git commit hook] --> Diff[Changed files]
      Diff --> Chunk[Chunker]
      Chunk --> Embed[Embedding model]
      Chunk --> TS[tree-sitter tags]
      Embed --> Vec[(Vector DB)]
      TS --> Sym[(Symbol graph)]
    end
    subgraph Query Path
      Q[Agent query + open file] --> R[Retriever]
      R -->|ANN search| Vec
      R -->|graph walk + PageRank| Sym
      R -->|blame + PR files| Git[(Git metadata)]
      Vec --> Rank[Reranker + token budget]
      Sym --> Rank
      Git --> Rank
      Rank --> Ctx[Context block]
    end
    Ctx --> LLM[LLM prompt]

Sourcegraph Cody uses the same pattern: code search, symbol intelligence, and embeddings layered in a hybrid retriever that can pull context from repositories the developer does not have open¹⁰. The key trade-off is index freshness: a rebase invalidates thousands of chunks. Mitigation: prioritize reindex of files the developer has open, invalidate lazily by commit SHA, and expose index lag to the UI⁸.

Tool-use sandbox isolation

Every agent-initiated shell command runs inside a per-session Firecracker microVM. Firecracker is a Rust VMM built on KVM with thread-specific seccomp filters and a jailer process that applies cgroup/namespace isolation then drops privileges¹¹. AWS Lambda and Fargate run on it. Practical boot latency is ~125-150 ms¹², making a warm pool viable.

Sandbox anatomy:

• Read-only mount of the repo snapshot at task start
• tmpfs scratch for agent writes (destroyed on finish)
• Network allowlist: package registries + model API only
• cgroup-enforced CPU (1 vCPU), memory (2 GB), and wall-clock (60 s per command) ceilings
• Destroy-on-finish: no persistent state survives between tasks

flowchart LR
    Orch[Agent Orchestrator] -->|tool_call| CP[Sandbox Control Plane]
    CP --> Pool[Warm VM Pool<br/>pre-booted Firecracker]
    Pool -->|assign| VM[Per-session MicroVM]
    subgraph VM[MicroVM Internals]
      RO[Read-only repo mount]
      TMP[tmpfs scratch]
      CG[cgroup limits<br/>1 vCPU, 2 GB, 60s]
      SEC[seccomp + jailer]
    end
    VM -->|stdout, exit code| Orch
    VM -->|egress| FW{Allowlist firewall}
    FW -->|registries, model API| Net[Internet]
    FW -->|all else| Drop[Block + alert]
    VM -.->|task done| Destroy[Destroy VM]

Why not containers? A container escape requires only a kernel CVE, not a hypervisor escape. Container-based sandboxes for AI agents remain vulnerable to kernel exploits, as multiple security researchers have documented¹². MicroVM-per-session gives hardware-enforced isolation where a kernel escape costs an attacker a fresh hypervisor CVE rather than a container breakout.

Cognition’s Devin runs each task in a cloud sandbox with browser, editor, and shell³. Cursor adapted its Background Agents VM scheduler to run “hundreds of thousands of concurrent sandboxed coding environments” during Composer RL training¹³.

Streaming diff application

The model streams edits; the client renders them inline as hunks the developer accepts or rejects. The critical insight from Cursor: full-file rewrite outperforms unified diffs for files under 400 lines because models see more full-file code than diffs in pretraining and struggle to count line numbers¹⁴.

Cursor’s fast-apply model: As described in their May 2024 blog post, a specialized 70B model (fine-tuned from Llama 3) rewrites the full file at ~1,000 tokens/sec (3,500 char/s) using speculative edits, a deterministic-draft variant of speculative decoding tuned for code¹⁴. This yields a ~13x speedup over vanilla Llama-3-70b inference and a ~9x speedup over Cursor’s previous GPT-4 speculative edits deployment.

Aider’s search/replace format: Aider interprets unified diffs as search/replace operations: the - and space lines form the search text, the + and space lines form the replacement¹⁵. Cursor adopted a variant of this format with redundant + and - markers that make the parser resilient to minor model failures¹⁴. Both formats avoid line-number hallucinations because the model specifies what to find and what to replace, not where.

Apply semantics: Each apply is atomic. The client accumulates hunks, the developer approves per-hunk or batch, and the filesystem transaction either succeeds completely or rolls back. A rollback token enables single-keystroke undo.

For files over 400 lines, Cursor is training long-context variants to reach 2,500 lines¹⁴. The trade-off: full-file rewrite costs more output tokens but eliminates positional errors that cause silent corruption.

Real-World Example

GitHub Copilot: 1.3M paid subscribers and 400M+ completions per day.

Copilot reached 1.3 million paid subscribers by Q1 2024 with 30% quarter-over-quarter growth, and more than 50,000 organizations had adopted it at the time of the Accenture study¹⁶¹⁷. The service handles 400M+ completion requests daily, peaking at ~8,000 QPS during the overlap of European afternoon and US morning¹.

The architecture centers on a global authenticating proxy written in Go. The IDE exchanges an OAuth credential for a signed 10-30 minute code-completion token. The proxy validates the signature locally, swaps in the real Azure API key, and forwards over a long-lived HTTP/2 connection. octoDNS does weighted geographic routing to the nearest healthy proxy region¹.

HTTP/2 is load-bearing. Stream-reset cancels a single request without tearing down the TCP connection. This matters because 45-50% of completion requests are cancelled by type-through before a response arrives¹. Cloud-provider ALBs that downgrade to HTTP/1 on the backend would have made this impossible, so the team uses GitHub’s internal HAProxy-based GLB¹.

The team explicitly rejected a PoP model: since every completion requires model inference, edge proxies back to centralized models create traffic tromboning with no caching benefit¹. When a model upgrade emitted a bad EOF token, the team added negative token affinity at the proxy rather than waiting weeks for a client rollout¹.

An Accenture enterprise study of Copilot found that developers accepted around 30% of suggestions, committed code from 90% of those accepted suggestions, and retained 88% of Copilot-generated characters; teams saw an 8.69% increase in pull requests and a 15% increase in merge rate¹⁷. Separate GitHub lab studies have measured task-completion speedups of up to 55%¹⁷.

Trade-offs

The meta-decision: model tiering. Without it, autocomplete at frontier-model prices costs ~$0.01 per request, making a $20/month subscription unprofitable at 100K completions/month. With tiering, autocomplete costs ~$0.00005 per request, and the subscription works.

Scaling and Failure Modes

At 10x (6M autocomplete QPS): The warm GPU pool saturates. Mitigation: scale on queue depth (not CPU), request coalescing for identical prefixes within 20 ms, and aggressive prompt-prefix caching.

At 100x (60M QPS): The retrieval index becomes the bottleneck. Mitigation: shard the vector DB by repo, replicate hot repos, and move to a CDN-first pattern where the most common completions are cached at the edge.

At 1000x: The architecture shifts to federated regional inference fleets with a global control plane for routing and billing only.

Failure: Runaway autonomous loop. The agent retries a failing test forever, burning tokens. Detection: per-task step/token/wall-clock ceilings plus duplicate-tool-call detection. Recovery: circuit-breaker halts the task and notifies the developer¹⁹. Anthropic’s April 2026 post-mortem documents exactly this pattern: Claude Code “would continue executing, but increasingly without memory” during a broken-cache bug²⁰.

Failure: Prompt-caching regression. A cache optimization silently corrupts agent memory; quality degrades for days. Anthropic’s clear_thinking bug (March 26 to April 10, 2026) cleared thinking on every turn instead of once, making the agent “seem forgetful and repetitive”²⁰. Detection: per-request diagnostics exposing cache-hit vs fresh; eval suites on every system-prompt change.

Failure: Sandbox escape. Agent installs a malicious package that exploits a kernel CVE. Mitigation: Firecracker hypervisor isolation (not just containers), network allowlist, package-hash verification, destroy-on-finish¹¹¹².

Common Pitfalls

Follow-up Questions

1. How do you keep the repo index consistent during an in-progress rebase?

Invalidate chunks lazily by commit SHA. Prioritize reindex of files the developer has open. Expose index lag to the UI so the developer knows when context may be stale. Rate-limit reindex jobs per repo to avoid thrashing.

2. A customer reports the agent leaked content from their private repo into another session. Walk through the investigation.

Check sandbox isolation (was the VM shared?). Audit the retrieval index for cross-tenant contamination. Verify that embedding shards are keyed by (tenant_id, repo_id). Check prompt-cache key isolation. The fix: strict tenant-scoped cache keys and per-tenant vector DB partitions.

3. How do you A/B test a new autocomplete model when accept-rate takes hours to stabilize?

Use a larger sample size (10% canary minimum). Measure both immediate accept-rate and 24-hour retention of accepted suggestions (did the developer revert?). Gate on both metrics. Copilot’s 45-50% cancellation rate means you need to exclude cancelled requests from the denominator¹.

4. A single autonomous task enters an infinite loop. How does the orchestrator detect and break it?

Hard ceiling on steps (50), tokens (200K), and wall-clock (10 minutes). Duplicate-tool-call detection on the action log (same tool + same args 3x in a row triggers halt). Per-tenant budget circuit breaker. Anthropic recommends “stopping conditions to maintain control” as a first-class agent design principle¹⁹.

5. How would you redesign for a regulated customer who cannot send code to a cloud?

Client-side retrieval index (Claude Code CLI pattern)¹⁸. On-prem small model for autocomplete. Frontier model calls route through a customer-controlled proxy with DLP scanning. Sandbox runs on customer infrastructure. The trade-off: no cross-repo search, slower cold start, customer bears GPU cost.

6. A new model claims 20% better SWE-bench Verified. What checks before 100% rollout?

Run internal regression set (not just SWE-bench). Replay the diff_log: does the new model produce diffs that match historical accepts? 1% canary measuring accept-rate and task-success-rate. Check for mode-specific regressions (a model tuned for autonomous may regress autocomplete)²⁰. Soak for 48 hours minimum.

Exercise

Exercise 1: Sandbox fleet sizing

Your coding agent has 10,000 concurrent autonomous tasks. Each task holds a Firecracker microVM for an average of 3 minutes. Your warm pool boots VMs in 125 ms. Tasks arrive at a Poisson rate. How large should the warm pool be to ensure 99% of tasks get a VM within 500 ms?

Key Takeaways

• Three modes, three systems. Autocomplete, chat, and autonomous loop share infrastructure but have fundamentally different latency budgets, model tiers, and safety envelopes.
• Retrieval quality is the product. Embeddings alone miss structural queries; pair them with a tree-sitter symbol graph and git-recency signal⁸¹⁰.
• Sandbox isolation is non-negotiable. Any agent running shell commands on untrusted input needs per-session microVM isolation with network allowlist and destroy-on-finish¹¹¹².
• The diff is the interface. Full-file rewrite at 1,000 tokens/sec beats unified diffs because models cannot count line numbers¹⁴.
• Evals gate the rollout. SWE-bench Verified, diff-log replay, and online canary metrics catch regressions that user-facing signals reveal only days later²³²⁰.

Further Reading

• GitHub Copilot: Building a Low-Latency Global Code Completion Service. The canonical reference for HTTP/2 proxy architecture, octoDNS routing, and the 200 ms latency budget.
• Cursor: Editing Files at 1000 Tokens per Second. Fast-apply model, speculative edits, and why full-file rewrite beats diffs for files under 400 lines.
• Anthropic: Building Effective Agents. Agent design patterns (prompt chaining, routing, orchestrator-workers) with the recommendation to start simple.
• Anthropic: An Update on Recent Claude Code Quality Reports. April 2026 post-mortem documenting three compounding regressions; canonical example of agent eval failures.
• Aider: Building a Better Repository Map with Tree Sitter. Tree-sitter + PageRank for code context retrieval; the open-source reference implementation.
• SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. Yang et al., NeurIPS 2024; defines the ACI concept that every coding agent now uses.
• Anthropic: Prompt Caching with Claude. 90% cost and 85% latency reductions; the single biggest cost lever for agents that re-read repo context.
• Cursor: Composer - Building a Fast Frontier Model with RL. MoE + RL training on agent tasks at scale with hundreds of thousands of concurrent sandboxed environments.

Flashcards

References