How do you A/B test an LLM change in production?
- 2 min — unit, primary metric, guardrails
- 3 min — sample size and run length
- 4 min — LLM-specific traps: delayed outcomes, novelty, peeking
- 3 min — shadow first, low-traffic fallback, platform ownership
An A/B test randomly assigns units (users, accounts or sessions) to the current and candidate versions and compares one pre-registered primary metric — task completion, resolution without escalation, conversion — plus guardrails: p95 latency, cost per query, safety-flag rate. Randomisation lets you attribute the difference to the change. Sample size and run length are fixed before launch from the minimum effect worth shipping.
Testing a new model for a B2B support assistant: randomise by account, not user, because colleagues share threads and would see both variants. Shadow traffic runs first for 2 days to catch crashes, cost surprises and format breaks with no user exposure. Primary metric: tickets resolved without human escalation (baseline 60%). Guardrails: p95 latency no worse than +300 ms, cost per resolved ticket no worse than +10%, safety flags no worse than +0.1 points. Run 14 days, then compare with a confidence interval.
LLM-specific traps: outcomes are delayed (a bad answer surfaces as a reopened ticket days later, so measure at a 7-day horizon), novelty effects fade over the first week, and per-message metrics overcount heavy users — analyse at the randomisation unit. Peeking inflates false positives; if you must monitor daily, use a sequential test designed for it. Cost and latency are part of the decision: +1 point of resolution at +40% cost may be a loss — state the exchange rate up front. For low-traffic features that can't reach significance, lean on offline evals plus weekly human review of shadow outputs. Ownership: a central experimentation platform (assignment, exposure logging, analysis) shared with non-LLM experiments; LLM teams add judged-quality metrics as experiment metrics, not a parallel system.
Baseline resolution 60%, minimum detectable effect 2 points absolute, α = 0.05, 80% power → n ≈ 15.7 × 0.24 / 0.02² ≈ 9.4k per arm, ~19k units total. At 3k eligible conversations/day: ~7 days to sample; run 14 to cover two weekly cycles and the 7-day reopen window. With account-level randomisation, inflate n by the design effect (≈1 + (m − 1) × ICC; m = 20 conversations/account, ICC = 0.05 → ~2×).
What the answer sounds like at each level
Split users between old and new versions and compare which performs better.
Pre-register a primary metric and guardrails, size the test, randomise at the right unit, shadow first, and avoid peeking.
I run LLM tests on the company's experimentation platform, set the cost/quality exchange rate before launch, use delayed-outcome windows, and route low-traffic features to offline-plus-human evaluation instead.
Follow-ups the interviewer will push
1After 3 days the candidate is +3 points and significant. Stop early?
No — the test was sized for 14 days and peeking inflates false positives; novelty and day-of-week effects haven't washed out. Use a sequential method if early stopping is a real requirement.
2Quality is +1.5 points but cost is +35%. Ship?
Depends on the pre-stated exchange rate. If a resolved ticket saves $6 of agent time, +1.5 points on 100k tickets/month is ~$9k saved; compare that to the extra model spend and decide on the number, not the vibe.
Red flags interviewers listen for
- Randomises per message so users see both variants.
- Picks the primary metric after looking at results.
- Ignores cost and latency as decision inputs.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Experimental design | Right randomisation unit, pre-registered metric, sized test. |
| LLM realism | Handles delayed outcomes, novelty, and cost/latency as decision inputs. |
| Org fit | Uses the shared experimentation platform rather than a bespoke system. |
Sources
- Trustworthy Online Controlled Experiments — Kohavi, Tang & Xu, 2020book