Shadow traffic
Copy real production requests to a new version whose responses users never see, then compare outputs, latency and cost before risking live traffic.
Shadow traffic (also called traffic mirroring or dark launching) sends a copy of real user requests to a new version of your system. Users still get answers from the old version; the new version's answers are recorded and compared offline. It is the safest way to test a new model, prompt or retrieval setup on real queries, because nobody sees the results until you have checked them with a Golden set, an LLM-as-judge or human review.
A trainee pilot flying the same route in a simulator, in real time, using the real flight's inputs. The passengers fly with the captain; the trainee's decisions are reviewed afterwards.
Letting the shadow perform side effects. A mirrored agent with live tools sends the second email, files the second ticket or writes to production, because to the tool it is a real request.
Mirror at the gateway (Envoy/Istio mirror with a percentage, or an application-level async fork) so the shadow call never sits on the user's critical path: fire-and-forget with its own timeout and its own concurrency pool. Side effects are the main danger: the shadow must not send emails, write to the primary DB, call payment tools or consume a user's rate limit, so run agent shadows with tools stubbed or pointed at sandboxes. Compare with paired evaluation on the same input: exact/structured diff for extraction tasks, Pairwise comparison with an LLM judge for free text (randomise order to cancel position bias), plus latency and token deltas. Sample 1–10% of traffic and stratify by intent so rare-but-important query types are represented. Shadowing doubles model spend on the sampled slice and can push you into provider rate limits, so give the shadow its own API key or budget. It answers 'is B at least as good on real inputs?'; it cannot measure user behaviour (clicks, resolution), which still needs a Canary & ramped rollout or A/B test.
Gateway-level async mirror at 5% (Envoy/Istio mirrorPercentage or app fork), separate API key and concurrency pool, side-effecting tools stubbed with recorded responses, outputs logged with the primary's to a comparison table, nightly pairwise judge run with position randomisation, 7–14 day window.
An insurance-claims triage assistant moves from a 70B self-hosted model to a hosted mid-tier model. 5% of requests (~3,000/day) are mirrored for 10 days; the shadow's tool calls go to a stub returning recorded responses. Results: structured-field agreement 96.8% (claim type, severity, coverage flag), judge preference 48/52 (tie within noise on n=1,200 sampled pairs), p95 latency 3.1 s vs 4.4 s, cost per claim $0.011 vs $0.019. Disagreements cluster on water-damage claims, which go to a targeted golden-set fix before the canary.
Before a model or prompt swap on a high-stakes path, I'd shadow 5% for a week with tools stubbed, compare structured outputs exactly and free text pairwise with a debiased judge, and only then canary. The follow-up: 'what can shadowing not tell you?' User outcomes. It tells you B behaves like A on real inputs; it doesn't tell you users prefer it.
Numbers worth knowing
- Sample rate
- 1–10%enough for ≥1,000 paired comparisons per key intent
- Extra model spend
- = sample rate × primary spendplus judge calls on the compared subset
- Pairs to detect a 5-point preference shift
- ~400–600two-sided test near 50/50, 80% power
How to decide
| When | Do this |
|---|---|
| Swapping model, prompt or retriever on a high-stakes path | Shadow first, then Canary & ramped rollout. |
| Output is structured (fields, labels) | Exact agreement rate; review every disagreement class. |
| Output is free text | Pairwise comparison with a judge, order randomised, spot-checked by humans. |
| Change affects UX or user behaviour | Skip to an A/B test; shadow can't measure it. |
Cost maths
Shadow cost = sample × requests × cost per call + judge cost. 60k requests/day × 5% = 3,000 shadows × $0.011 = $33/day; judging 300 pairs/day at ~4k tokens each on a mid-tier judge ($3/$15 per M, illustrative Oct 2026) ≈ $4/day. A 10-day shadow ≈ $370, cheap insurance for a migration touching every claim.
Operating it
- Shadow load pushes the account above 70% of provider rate limits: dedicate a key or lower the sample
- Multiple candidate configs at once: move to an offline replay harness over logged traffic instead of live mirrors
- Judge disagreement with human spot-checks >15%: recalibrate the judge before trusting results
- Shadow call error rate and p95 vs primary: investigate if errors >2× primary
- Structured agreement rate: alert if <95% on any intent with ≥100 samples
- Side-effecting tool calls from shadow: alert on any non-zero count
When it goes wrong
Customers received two identical 'your claim is under review' SMS messages for two days.
Support tickets and the SMS provider's per-recipient duplicate report.
A shadow deployment of a new triage agent was mirrored with its real send_sms tool enabled; the stub configuration was applied to staging only.
Tool layer refuses side-effecting calls when the request carries x-shadow: 1; deploy check fails if a shadow environment has live credentials for outbound channels.
Where it shows up
Sources
- Mirroring — Istio, 2025docs
- The Site Reliability Workbook, ch. 16: Canarying Releases — Google, 2018book
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Zheng et al., 2023paper
Check yourself
Three questions. Every option is explained once you answer.