Your new prompt scores 2 points higher on the eval. How do you know it is a real improvement?
- 2 min — sources of noise: sampling and generation
- 4 min — paired test, worked interval on 200 items
- 3 min — sizing for a minimum detectable effect
- 3 min — slices, multiple comparisons, harness defaults
An eval score is an estimate from a finite sample, so it has sampling error — and LLM outputs add generation noise on top. Statistical significance asks whether the difference exceeds what chance produces. For two systems on the same items, use a paired test (paired bootstrap over items, or McNemar's test for pass/fail): it removes item difficulty from the variance and needs far fewer items than comparing two independent means. Setting temperature to 0 does not remove generation noise on hosted APIs — batched inference is not bitwise deterministic — so run each item several times and average per item before pairing.
Financial-filings Q&A, 200 binary items: a 2-point gain is 4 items. About 10% of items flip between the two prompts (the discordant pairs), so the paired 95% interval is roughly ±6 points and spans zero — inconclusive. Options: grow the slice that matters, average 3–5 runs per item to shrink generation noise, or call it neutral and decide on cost or latency. Detecting 2 points reliably on this task needs ~2,000 items, not 200.
Decide the minimum effect worth shipping before running the eval, then size the set for it — halving the detectable difference needs ~4× the items. Report intervals, not point scores, per slice; a significant regression on a 60-item 'restated earnings' slice matters more than a flat mean. Correct for multiple comparisons: try 20 prompt variants and one will 'win' by chance, so confirm the winner on a held-out slice. Cluster standard errors when items share a source document (Miller 2024). Make this the harness default — teams shouldn't each re-derive bootstrap code — and put the interval in the PR comment so reviewers see 'B − A = +2.0 [−4.1, +8.1]', not '+2.0'.
McNemar sizing with discordant rate p_d = 0.10, effect δ = 0.02, α = 0.05, 80% power: n ≈ (1.96√p_d + 0.84√(p_d − δ²))² / δ² ≈ (0.620 + 0.265)² / 0.0004 ≈ 1,960 items. With 200 items the detectable effect is ≈ 2.8 × √(0.10/200) ≈ 6.3 points. Running 3 repeats × 2,000 items × $0.025 ≈ $150 per comparison.
What the answer sounds like at each level
Check whether the difference is statistically significant, maybe by running more examples.
Use a paired bootstrap or McNemar on the same items, average repeated runs per item, report a 95% interval per slice, and pre-set the minimum effect.
I make paired intervals, clustered errors and held-out confirmation the harness default, size eval sets from the minimum detectable effect each team commits to, and put intervals in every PR report.
Follow-ups the interviewer will push
1Why not just set temperature to 0 and run once?
Hosted inference isn't bitwise deterministic at T = 0 (batch-dependent kernels, MoE routing), many reasoning models fix or reject temperature, and you'd be measuring a configuration you don't ship. Run at production settings, 3–5 times, and pair the per-item means.
2The mean is flat but one slice dropped 6 points with p = 0.03. Ship?
Not if that slice is one you gate on — fix it or get an explicit owner sign-off. Also check how many slices you tested: with 20 slices, one p = 0.03 is expected by chance, so confirm on held-out items before reverting work.
Red flags interviewers listen for
- Compares two means from independent runs instead of a paired test on the same items.
- Believes temperature 0 makes hosted LLM evals deterministic.
- Tries many variants and ships the best without held-out confirmation.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Statistical method | Chooses a paired test and explains why it beats independent comparison. |
| Quantification | Works the interval or required n with stated assumptions. |
| Process | Pre-registers the effect size, handles multiple comparisons, ships intervals in PR reports. |
Sources
- Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations — Evan Miller (Anthropic), 2024paper
- Defeating Nondeterminism in LLM Inference — Horace He (Thinking Machines Lab), 2025blog