Statistical significance
Whether an eval or A/B delta is bigger than the noise; report confidence intervals, use paired tests on shared items, and size n before you run.
When you run an eval twice, or on a different sample, the score changes a bit even if nothing improved. Statistical significance asks: is the difference I see bigger than this random wobble?
The practical tool is the confidence interval: a range like '80% ± 12 points' that says how precisely you measured. Small test sets give wide ranges, so a 3-point improvement on 40 questions tells you almost nothing.
Flipping a coin 10 times and getting 7 heads does not prove the coin is biased; flip it 1,000 times and 700 heads would.
Declaring a winner from overlapping or missing error bars, or slicing the results 15 ways until one slice is 'significant'. Both are how teams ship regressions with a spreadsheet that says they improved.
For a pass rate p on n independent cases, the 95% CI half-width is ≈ 1.96·√(p(1−p)/n): ±12.4 points at n=40 and p=0.8, ±5.5 at n=200, ±2.5 at n=1,000. Width shrinks with √n, so halving it costs 4× the cases. Near 0% or 100% use a Wilson interval; the normal approximation breaks down.
Comparing two variants on the same cases, use a paired test (McNemar for binary outcomes, paired bootstrap or paired t-test on per-case scores): the variance comes only from cases where the variants disagree, which typically halves the CI width. Miller (2024) also recommends clustered standard errors when cases share a source (e.g., 5 questions per document) and resampling several generations per case to separate model sampling noise from item difficulty. Multiple comparisons inflate false positives: testing 10 slices at α = 0.05 gives ~40% odds of one spurious 'significant' slice, so correct (Holm, Benjamini–Hochberg) or pre-register the slices that matter. And significance is not importance: with n = 50,000 a 0.2-point gain is significant and irrelevant.
Report every eval score as rate ± 95% CI (Wilson for rates near 0 or 1). Compare variants paired on the same cases with McNemar (statsmodels.stats.contingency_tables.mcnemar) or a 10,000-resample paired bootstrap; cluster by source document; pre-register critical slices and apply Holm correction to the rest.
A RAG answer-quality eval on financial filings. Variant B scores 82.5% vs A's 80.0% on 400 questions. Unpaired, each CI is about ±3.8 points and the intervals overlap. Paired: the variants disagree on 46 questions, B wins 28 and A wins 18; McNemar χ² = (28−18)²/46 ≈ 2.2, p ≈ 0.14: not significant. To detect a 2.5-point gain at this discordance rate with 80% power you need roughly 1,400 questions, or more samples per question.
Say: "I size the eval from the effect I care about, compare paired on the same items, and report the interval, not just the delta." Expect: "how many cases do you need?": walk the √n arithmetic (±12 at 40, ±5.5 at 200, ±2.5 at 1,000 for an 80% pass rate).
Numbers worth knowing
- 95% CI half-width at 80% pass
- ±12.4 (n=40), ±7.8 (n=100), ±5.5 (n=200), ±2.5 (n=1,000)normal approximation, independent cases
- Cost of halving the CI
- 4× the caseswidth ∝ 1/√n
- Chance of ≥1 false 'significant' slice
- ≈ 40%10 independent slices at α = 0.05, no correction
How to decide
| When | Do this |
|---|---|
| Both variants can run on the same cases | Use a paired test; it typically halves the CI width for free. |
| Cases come in clusters (several questions per document or per user) | Use clustered standard errors or bootstrap by cluster, or you will overstate precision. |
| The delta you care about is smaller than the CI half-width | Increase n (or samples per case) before running; don't run and hope. |
Cost maths
Required n for a target half-width h at pass rate p ≈ 1.96²·p(1−p)/h². For ±3 points at 80%: 3.84 × 0.16 / 0.0009 ≈ 680 cases. At ~$0.019 per case-sample (Sonnet 4.6-class generation plus Haiku-class judge, Oct 2026 list) and 3 samples, ≈ $39 per run, $19 on the Batch API. Labelling 680 cases by hand is the real cost, which is why paired designs and judges matter.
Operating it
- Eval deltas you act on are routinely below 3 points: grow the golden set toward 700+ cases or move to paired designs with more samples.
- You report more than 5 slices per run: add multiple-comparison correction to the harness.
- Run-to-run spread of the baseline (re-run weekly, unchanged): alert if it exceeds the reported CI.
- Share of shipped changes whose online effect matched the offline sign: review quarterly, alert below 70%.
When it goes wrong
Over a quarter, a team shipped 6 'improvements' to a legal-research assistant, each +2 to +4 points on a 60-case eval; the quarter-end re-run on a fresh 500-case set showed no net change.
A new hire re-ran the original baseline 5 times and got scores spanning 9 points.
n=60 has a ±10-point CI at an 80% pass rate; every 'improvement' was inside the noise, and the team had picked the best of several runs each time.
Grew the set to 500, switched to paired McNemar with 3 samples per case, and required the CI lower bound above zero before claiming a gain.
Where it shows up
Sources
- Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations — Miller (Anthropic), 2024paper
- Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing — Kohavi, Tang & Xu, 2020book
Check yourself
Three questions. Every option is explained once you answer.