Design a coding agent that takes a ticket, works in the company's repositories, and opens a pull request with passing tests.
- 5 min — scope: languages, repo sizes, test coverage, who reviews
- 10 min — agent loop, tools and repo context strategy
- 10 min — sandbox: isolation, snapshots, egress, credentials
- 10 min — evaluation and metrics: SWE-bench-style internal set, merge and revert rates
- 5 min — estimate, rollout and ownership
A coding agent is an agent loop whose tools are a file system, a shell and a test runner inside a sandbox. Three things decide quality:
- Repo context — context engineering: ripgrep search, symbol/LSP lookup, optional embeddings over files, repo instruction files (AGENTS.md / CLAUDE.md).
- Feedback — fast, scoped test runs, linters and type-checkers on every iteration.
- Stop conditions — turn budget, cost budget, 'tests green and self-review passed'.
On SWE-bench Verified, leading agents went from under 50% in mid-2024 to roughly 70–80% by late 2025, mostly from better models and scaffolds rather than retrieval tricks.
Ticket → orchestrator boots a Firecracker or gVisor microVM from a per-repo snapshot at the ticket's base SHA with dependencies pre-installed (without the snapshot, cold start is 2–5 min; with it, ~20 s) → the agent explores with grep, read_file, list_symbols, edits with a diff tool, runs only affected tests (test-impact analysis) and iterates → on green, a self-review pass and a policy check (no secrets, no CI-config edits, diff-size cap) → PR with a summary and test log.
Egress is allowlisted to the package mirror. The only credential is a short-lived, repo-scoped token that can push a branch and never merge.
- Blast radius is the design. The July 2025 Replit incident — an agent deleted a production database during a code freeze — is the lesson: the agent never holds production credentials, and destructive operations need human approval enforced outside the model.
- Measure merged, not opened. Merge rate, reviewer minutes per PR, 14-day revert rate, cost per merged PR. An agent opening 1,000 PRs at a 15% merge rate is a tax on reviewers.
- Failure modes: flaky tests teaching the agent to retry until green (quarantine flakes); the agent editing tests to pass (block test edits unless the ticket asks); context blow-up on monorepos (cache the stable prefix, compact old turns).
- Org scope: developer-productivity owns sandboxes and policy; repo owners own instruction files and test hygiene. Buy the model, build the sandbox and policy layer, and roll out repo by repo starting where coverage is high.
Assumptions: 5,000 tasks per workday; 40 turns/task; ~60k-token context per turn, ~54k of it a cached prefix and ~6k new; 1k output tokens/turn; illustrative prices $3/M input, $15/M output, cache write 1.25× ($3.75/M), cache read 0.1× ($0.30/M); 15 min wall clock per task.
- Tokens per task: cache reads 40 × 54k = 2.16M; cache writes 40 × 6k = 240k; output 40k.
- $ per task: 2.16M × $0.30 = $0.65 + 240k × $3.75 = $0.90 + 40k × $15 = $0.60 ≈ $2.15. Uncached, the same task is 2.4M × $3 + $0.60 ≈ $7.80 — caching is a 3.6× lever.
- Model spend: 5,000 × 22 workdays × $2.15 ≈ $237k/month.
- Sandbox concurrency: peak 2× average over 8 h → 1,250 tasks/h × 0.25 h ≈ 310 sandboxes; at 4 vCPU / 8 GB each ≈ 1,250 vCPU ≈ 20 × 64-vCPU nodes at peak, ~$20k/month with autoscaling.
- Provider limits: ~310 loops × one call per ~20 s ≈ 15 calls/s × 60k tokens ≈ 54M input tokens/min — negotiate tokens-per-minute, not requests.
- Unit metric: at a 45% merge rate, ≈ $4.80 of model spend per merged PR; reviewer time is the larger cost.
What the answer sounds like at each level
Wraps a model with read/write/run tools and loops until tests pass.
Adds sandboxed execution, scoped credentials, test-impact selection, turn and cost budgets, and tracks merge rate.
Designs the sandbox fleet and policy layer as a platform, sets the metric as cost per merged PR, and sequences rollout by repo readiness with clear ownership.
Follow-ups the interviewer will push
1Tasks grow 10×.
The sandbox pool saturates before the model: pre-warm microVM snapshots per repo, share a read-only dependency cache, queue by priority. Route routine tasks (dependency bumps, lint fixes) to a cheaper model and keep the frontier model for multi-file changes.
2A repository handles payments data.
The agent runs in that business unit's account, code never leaves the boundary, the model endpoint is the approved regional one, PRs need two human approvals, and infra-as-code paths are off-limits to the agent.
3The model provider fails mid-task.
Tasks use durable execution: trajectory and workspace snapshot are checkpointed per step, so the task resumes on a fallback model or waits. Nothing is half-pushed because the push happens once, at the end.
Red flags interviewers listen for
- The agent runs on a laptop or shared CI runner with broad credentials.
- Success is 'the model said it's done' rather than tests and review.
- Counts PRs opened rather than merged and reverted.
- No turn or cost budget.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Safety | MicroVM isolation, scoped short-lived credentials, no merge rights, egress allowlist. |
| Feedback loop | Scoped tests, linters and self-review before the PR; flaky-test handling. |
| Metrics | Cost per merged PR, reviewer time, revert rate — not PR count. |
| Cost | Shows prompt caching as the main lever with numbers. |
Sources
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues? — Jimenez et al., 2023paper
- Building effective agents — Anthropic, 2024blog