A/B test
Randomly split live users between the current and new version, then compare a pre-chosen outcome metric with a significance test to decide what ships.
An A/B test shows the current version (A) to one random group of users and the new version (B) to another, then compares an outcome you chose in advance, such as 'issue resolved without a human' or 'suggested code accepted'. Random assignment means the only systematic difference between the groups is the change.
It is the final word on whether an AI change helps real users, after offline evals (Offline vs online evaluation) say it is safe to try.
A clinical trial: patients are randomly given the new drug or the current one, and you compare recovery rates, not how confident the doctors feel.
Stopping the test the first day p < 0.05 appears. With daily peeking over two weeks, the real false-positive rate is several times the nominal 5%.
Pre-register one decision metric, 2–4 guardrail metrics (latency p95, Cost per query, complaint rate) and the minimum detectable effect (MDE). Sample size per arm ≈ 16·p(1−p)/δ² for 80% power at α = 0.05: detecting a 1-point change on a 10% escalation rate needs ~14,400 users per arm. Randomise on user or account, not request, so a user sees one variant and repeated sessions don't double-count; analyse at the same unit.
LLM-specific traps: novelty effects (users try the new assistant more in week 1), variance from long conversations (use per-user metrics), cost asymmetry (B may be 40% pricier; put cost in the decision) and interference (shared caches or Rate limiting pools coupling arms). Check sample-ratio mismatch (chi-square p < 0.001 means broken assignment, so stop and debug). CUPED, which adjusts for each user's pre-period behaviour, commonly cuts variance enough to shorten tests noticeably. Do not peek and stop early on a p-value; use a fixed horizon or a sequential test designed for it. Platforms: GrowthBook, Statsig, Eppo, Optimizely, or an in-house assignment service behind a Feature flag.
User-level randomisation through a feature-flag service (GrowthBook, Statsig, LaunchDarkly + warehouse), 50/50 split, fixed 14-day horizon, one decision metric plus latency, cost and complaint guardrails, SRM check daily, CUPED where pre-period data exists.
Support assistant for an airline: B adds retrieval over fare rules. Decision metric: conversations resolved without human handoff (baseline 62%). MDE 2 points → 16 × 0.236 / 0.0004 ≈ 9,400 users per arm. At 3,000 eligible users a day split 50/50, that is ~7 days; run 14 to cover two weekly cycles. Result: 62.1% → 64.6%, CI on the difference [+1.0, +4.0]; guardrails: p95 latency +380 ms (within the 500 ms budget), cost per conversation +$0.004. Ship.
Say: "I pre-register one decision metric, guardrails and the MDE, size the test from the MDE, randomise per user, check SRM, and don't peek." Expect: "traffic is too low for an A/B, now what?": offline paired eval plus a staged Canary & ramped rollout with guardrails, and accept you can only detect large effects.
Numbers worth knowing
- Users per arm for a 1-pt change on a 10% rate
- ≈ 14,40080% power, α=0.05, two-sided
- Users per arm for a 2-pt change on a 62% rate
- ≈ 9,400same power and α
- Sample-ratio mismatch alarm
- chi-square p < 0.001on assigned counts vs intended split
How to decide
| When | Do this |
|---|---|
| Fewer than ~2,000 eligible users per week | Skip the A/B; use paired offline evals plus a guarded canary and only expect to detect large effects. |
| B is more expensive per query | Pre-register the cost guardrail and the value of a 1-point gain, so the ship decision is arithmetic. |
| SRM fires | Stop and fix assignment or logging; never analyse a test with broken randomisation. |
Cost maths
Cost of testing = extra spend in B × duration. If B costs $0.012 more per conversation on 1,500 conversations/day for 14 days: ≈ $250. Opportunity cost of under-powering is larger: a 7-day test sized for a 4-point MDE cannot see the 2-point win you actually got.
Operating it
- More than 4 concurrent tests on the same surface: add a layering or mutual-exclusion scheme to avoid interaction.
- Decision cycles stall on 2-week horizons: adopt CUPED or sequential testing on the platform.
- Sample-ratio mismatch on every live test, daily: alert at p < 0.001.
- Guardrail metrics per arm: alert when p95 latency or cost per query in B exceeds its pre-registered bound.
When it goes wrong
A code-completion change 'won' an A/B with a 3% acceptance-rate gain and was rolled out; acceptance returned to baseline within three weeks.
The post-launch metric dashboard showed the gain decaying week over week.
The test ran 5 days and was stopped at the first significant result; a novelty effect inflated week-1 acceptance, and requests rather than users were randomised, so heavy users dominated.
User-level randomisation, a fixed 14-day horizon, week-1 excluded as burn-in for UI changes, and a sequential-testing method for any early stop.
Related terms
Sources
- Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing — Kohavi, Tang & Xu, 2020book
- Controlled experiments on the web: survey and practical guide — Kohavi et al., 2009paper
Check yourself
Three questions. Every option is explained once you answer.