Canary & ramped rollout
Release to a small slice of real traffic first (1%, then 5%, 25%…), compare against the old version, and roll back automatically if metrics degrade.
A canary release sends a small share of real users to the new version while everyone else stays on the old one. If error rate, latency or answer quality look worse on the canary, you roll back having affected only a few people. If it looks fine, you widen the slice step by step until everyone is on the new version. It is the default way to ship model, prompt and config changes, usually controlled by a Feature flag.
A chef tasting the soup from one spoon before serving the whole pot, and serving one table before the full dining room.
Gating a canary only on error rate and latency. An LLM prompt regression is a 200 OK with a worse answer, so the canary passes and the regression ships to 100%.
Compare canary to a control of the same size running the old version at the same time, not to yesterday's baseline, so time-of-day and traffic mix cancel out (Google's SRE workbook makes this point). Steps like 1% → 5% → 25% → 50% → 100% with a bake time at each that is long enough to collect the signal you care about: minutes for 5xx, hours for latency tails, a day or more for quality signals like thumbs-down or escalation rate. Gate each step on SLO & SLA-derived thresholds and automate rollback (Argo Rollouts AnalysisTemplates, Flagger, LaunchDarkly-style guarded rollouts). For LLM changes, canary metrics must include quality proxies (refusal rate, Abstention & relevance thresholds rate, citation-validity rate, judge score on a sample) because a bad prompt returns HTTP 200. Route by stable key (user or tenant id hash) so a user doesn't flip between versions mid-conversation. Small canaries can't see rare failures: at 1% of 50k requests/day you get 500 samples, enough for a 5xx spike, not for a 0.2% hallucination rate.
Flag-driven split by hashed user/tenant id; concurrent control slice of equal size; steps 1/5/25/50/100; gates on 5xx, p95 latency, refusal rate, citation validity and thumbs-down; automated rollback (Argo Rollouts/Flagger analysis or flag guardrails); bake ≥1 h at 1–5%, ≥24 h before 100% for prompt or model changes.
A developer-docs assistant ships a new system prompt. Day 1: 1% canary + 1% control, 600 requests each; error rates equal, citation-validity 97.9% vs 98.1%. Day 2: 10%, thumbs-down 3.1% vs 3.4% (n≈6,000 each, not significant). Day 3: 50%; refusal rate jumps from 1.2% to 4.8% on API-reference queries because the new prompt over-applies a safety instruction. The analysis gate trips on refusal rate >2× control and rolls back automatically within 9 minutes.
I'd canary every model, prompt and retrieval change behind a flag, comparing against a concurrent control slice, with quality proxies in the gate because LLM regressions return 200. Steps 1/5/25/50/100 with bake time matched to the slowest signal. Expect: 'how do you know 1% is enough?' I'd show the power calculation: it's enough for crash signals, not for rare quality issues, which is why shadowing and offline evals come first.
Numbers worth knowing
- Samples at 1% of 50k req/day
- 500/daydetects a 5xx spike; cannot detect a 0.2% quality regression
- Samples to see 1% → 2% rate change
- ~2,300 per armtwo-proportion test, 80% power, α=0.05
- Automated rollback time
- 5–10 minanalysis interval 1–5 min, 2 consecutive failures
How to decide
| When | Do this |
|---|---|
| Change can be split per request or user | Canary with a concurrent control. |
| Change is all-or-nothing (index re-embed) | Blue-green deployment with shadow checks instead. |
| Quality signal is rare (<0.5%) | Rely on shadow + offline evals; the canary only guards crashes and big shifts. |
| Regulated tenant or enterprise contract | Exclude them from early steps; canary on internal and free-tier traffic first. |
Cost maths
Canary cost is mostly time: at 50k requests/day, getting 2,300 samples per arm for a quality gate takes ~5 days at 1% or ~11 hours at 10%. The trade: a 10% step puts 5,000 users/day on a possibly bad version. Prefer longer bakes at low percentages for user-visible changes, faster steps for infra-only changes.
Operating it
- More than ~5 concurrent canaries: you need a flag platform with mutual-exclusion layers
- Manual go/no-go decisions on every release: automate analysis gates
- Canary results routinely inconclusive: add shadow evaluation or raise step sizes with tighter gates
- Canary vs control 5xx rate: auto-rollback if >2× control over 10 min
- Canary vs control p95 latency: rollback if >+20%
- Refusal/abstention rate and citation validity vs control: rollback if refusal >2× or validity drops >1 point
- Thumbs-down rate vs control at ≥2,000 samples per arm: block promotion if significantly worse
When it goes wrong
On 19 July 2024 a CrowdStrike content update crashed roughly 8.5 million Windows hosts worldwide.
Mass boot-loop reports from customers within minutes of release.
Per CrowdStrike's public root-cause analysis, the Channel File 291 content update was validated in testing but deployed to all hosts at once rather than through a staged rollout.
CrowdStrike committed to staged deployment of content updates with canary rings and customer control over update timing. The lesson for LLM configs (prompts, guardrail rules) is the same: config is code and needs a canary.
Where it shows up
Sources
- The Site Reliability Workbook, ch. 16: Canarying Releases — Google, 2018book
- External Technical Root Cause Analysis — Channel File 291 — CrowdStrike, 2024postmortem
Check yourself
Three questions. Every option is explained once you answer.