How would you roll out a new model version to production using shadow traffic and canaries?
- 5 min — stages and what each one catches
- 10 min — shadow design: mirroring, side effects, comparisons
- 10 min — canary ramp, gates, rollback
- 10 min — statistics: power, segments, sticky assignment
- 5 min — ownership and sign-off
Stages: offline evals (golden sets, safety, regression suites) → shadow (mirror real requests to the new model, discard its outputs, compare) → canary (a small share of real users, guardrail metrics, automatic rollback) → progressive ramp → full, with the old version kept warm for rollback.
Shadow catches latency, cost, error rate, tool-call validity, refusals and length shifts, plus pairwise judge wins on sampled pairs. Only the canary catches user-behaviour effects: thumbs, regenerations, escalations, retention.
Upgrading a coding assistant from v6 to v7. Shadow 10% of traffic for 48 h, with tool calls executed against read-only or mocked tools so nothing happens twice. Results: tool-call JSON validity 99.6% vs 99.1%, tokens per answer +18% (a cost regression), judge win rate 54% on 2k pairs. Then canary 1% → 5% → 25% → 50% → 100% with a gate at each step and automatic rollback on error rate +0.5 points, thumbs-down +1 point (significant), or latency p95 +20%.
- Shadow must be side-effect free. For agents, stub every write; a shadow model that sends real emails is an incident.
- One change at a time. Changing prompt version and model together makes any regression unattributable.
- Sticky assignment per user, not per request, for consistent UX and valid statistics; check per segment (language, enterprise tenants) because aggregates hide regressions.
- Rollback: keep v6 capacity warm until 72 h after 100%. For hosted models, pin version ids and plan for deprecation dates.
- Org scope: the model platform owns the pipeline; product teams own their gate metrics; a release council signs off tier-1 surfaces. Build on the existing flag and experimentation platform, not a bespoke system.
Assumptions: 2,000 QPS in production; shadow at 10% for 48 h; baseline thumbs-down 8% with ~5% of responses rated; detect +1 point at α = 0.05 with power 0.8; illustrative fleet cost $3.9M/month.
- Shadow cost: 0.1 × $3.9M × 2/30 ≈ $26k per rollout.
- Sample size per arm: n ≈ 2 × (1.96 + 0.84)² × p(1 − p) / d² = 2 × 7.84 × 0.0736 / 0.0001 ≈ 11.5k rated responses — see significance.
- Time: 11.5k / 5% ≈ 231k responses per arm. A 1% canary (20 QPS) needs ~3.2 h; a 5% canary ~40 min. Rarer metrics (escalations at 0.5%) need days or a larger arm.
- Judge cost: 2k pairs/day × ~6k tokens × $3/M ≈ $36/day.
What the answer sounds like at each level
Runs offline evals, then releases to a percentage of users and watches dashboards.
Uses side-effect-free shadow, staged canaries with guardrail metrics and automatic rollback, and sticky assignment.
Sizes each stage by statistical power, gates per segment, budgets shadow cost, and defines who signs off at each step.
Follow-ups the interviewer will push
1Thumbs-down is too sparse; the canary would take weeks.
Use higher-volume proxies (regeneration rate, copy rate, edit distance, task completion) or judge-scored samples of canary traffic, and compute power before starting.
2v7 is better overall but worse for German users.
The per-segment gate fails: hold the ramp for that locale (route de to v6) while fixing. This is why gates are per segment.
3The provider deprecates v6 in 30 days.
Compress: 24 h shadow, one-week ramp, tighter monitoring and an explicit risk sign-off; negotiate an extension in parallel.
Red flags interviewers listen for
- Shadow traffic that executes real side effects.
- Ships prompt and model changes together.
- Per-request randomisation for a conversational product.
- No power calculation, so canary stages are too short to detect anything.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Safety | Side-effect-free shadow; automatic rollback; warm old version. |
| Statistics | Sample size per stage from baseline rate and minimum effect. |
| Coverage | Per-segment gates; cost and length regressions caught in shadow. |
| Process | Clear owners for pipeline, metrics and sign-off. |
Sources
- Canarying Releases (The Site Reliability Workbook) — Google SRE, 2018book
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Zheng et al., 2023paper