How would you roll out a new agent to production safely?
- 2 min — four stages and exit criteria
- 3 min — sizing stages with the rule of three
- 3 min — account-access example: shadow findings, canary, autonomy per class
- 2 min — abort metrics, kill switch, pinned models
- 2 min — ownership across product, ops, security, platform
Stage it, with exit criteria set before you start. (1) Offline: pass the eval gate on the golden set (pass^k, not pass^1) and a red-team suite. (2) Shadow: the agent runs on real inputs; actions are logged, not executed, and diffed against what humans did. (3) Canary behind a feature flag — 1% → 5% → 25% → 100% — with approval on every action at first. (4) Earn autonomy per action class: remove approvals for a class only on evidence.
Size stages statistically: with zero wrong actions observed in n trials, the 95% upper bound on the error rate is ≈ 3/n (rule of three). To claim < 0.5% you need ≥ 600 clean actions in that class. The July 2025 Replit incident, where an agent deleted a production database during a code freeze, is the argument for environment separation and gated destructive actions at every stage.
Account-access support agent (verify identity, reset MFA, unlock accounts). Shadow for 2 weeks on 2,300 tickets; canary at 5% with every action approved; then auto-approve unlocks with strong identity signals while MFA resets stay gated. A kill switch reverts all traffic to the human queue within 60 s, and abort thresholds (wrong-action rate > 0.5%, escalations +20% over baseline) trigger it automatically.
Define success and abort metrics before launch — task success, wrong-action rate, escalation rate, cost per resolution, complaints — and wire automated rollback to the abort thresholds. Keep the fallback (graceful degradation to the human process) staffed during ramp. Pin dated model snapshots and re-run the gate on every model, prompt or tool change — any of them is a new release. Ramp on evidence counts, not calendar time.
Org scope. The product team owns the launch; support operations co-owns abort thresholds and keeps the human queue warm; security/fraud signs off on identity-sensitive action classes; the platform team provides shadow-diff tooling, flags and kill switches as shared infrastructure so the second agent launch costs a fraction of the first. Failure mode: shadow mode that can't simulate the consequences of actions it didn't take, so downstream effects (the user's next ticket) are invisible until canary.
Assumptions: 1,100 access tickets/day, ~1.4 agent actions per ticket. Shadow 14 days ≈ 2,300 tickets in the routed slice. Canary at 5% ≈ 77 actions/day → ~15 days to reach 1,140 actions; zero errors gives an upper bound of 3/1,140 ≈ 0.26%. Reviewer load during canary: 77 × 40 s ≈ 51 min/day.
What the answer sounds like at each level
Test offline, run in shadow mode, then a small canary with human approval, and ramp up if it goes well.
Sets abort thresholds before launch, sizes stages with the rule of three, earns autonomy per action class, pins model snapshots and keeps a kill switch.
Splits launch ownership across product, support ops, fraud/security and platform, and turns shadow-diff, flags and kill switches into shared infrastructure for every subsequent agent.
Follow-ups the interviewer will push
1How many canary actions before removing approval on a class?
Enough for the error bound you need: zero errors in n gives ≈ 3/n at 95%. For < 0.5%, ≥ 600 clean actions in that class; for irreversible or fraud-prone classes, keep the gate regardless.
2The provider releases a new model version mid-ramp. Do you take it?
Not mid-ramp. Pin the dated snapshot; evaluate the new one offline and in shadow in parallel; switch only through the same gate. A model change is a new release.
3Now a regulated tenant (bank) wants it. What changes?
Their own shadow period on their traffic, their compliance sign-off on gated classes, region-pinned processing, audit export of every approved action, and a contractual kill switch they can trigger.
Red flags interviewers listen for
- Ramps on calendar time rather than evidence counts.
- No abort thresholds or automated rollback defined before launch.
- Treats a model or prompt change as not requiring re-evaluation.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Staging | Offline → shadow → canary with approvals → autonomy per action class. |
| Statistics | Stage sizes justified with error bounds. |
| Safety | Pre-set abort thresholds, automated rollback, warm fallback, environment separation. |
| Org scope | Named owners per concern; rollout tooling as shared infrastructure. |
Shadow first, then earn autonomy
Illustrative exampleWe built an agent to resolve account-access tickets — about 31% of our support volume, ~1,100 tickets/day, median human resolution time 4 h 10 min. Offline it passed 92% of a 180-task golden set (pass^3 84%). The head of support wanted it on for all access tickets within two weeks to hit a quarterly backlog target.
I was tech lead for a team of five (three engineers, one data scientist, one support-ops analyst). My job: launch without a single wrongly unlocked account and produce evidence for how much autonomy each action class should get. The head of support was the dissenting voice; the blast radius of a mistake was account takeover for real customers.
I proposed a 2-week shadow and committed to a date for the first canary so the timeline stayed concrete. Engineers built the shadow diff; the data scientist owned the comparison; I reviewed every disagreement category myself. The shadow ran on 2,300 tickets: 91% agreement with human decisions, but 14 proposals (0.6%) unlocked accounts after only weak identity signals (email match without device or recent-login evidence) — a class absent from the golden set. I took those 14 cases to the head of support; seeing real accounts that would have been unlocked changed the conversation in one meeting. We tightened the verification tool, added 40 tasks for that class, then ran a 5% canary with approval on every action and automated abort thresholds.
Canary: 1,140 actions, 0 wrong actions (95% upper bound ≈ 0.26%), 6% of proposals edited by approvers. We auto-approved unlocks with strong signals; MFA resets stayed gated. Six weeks later 38% of access tickets resolved automatically with median time 6 minutes, and the backlog target was met three weeks after the original date. The shadow-diff tooling became the platform default for four later agent launches. What I'd do differently: bring the fraud team in before shadow — they recognised the weak-signal pattern immediately and could have saved a week.
Sources
- Canarying Releases (The Site Reliability Workbook, ch. 16) — Google, 2018book
- τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains — Yao et al., 2024paper