How many eval examples do you need to detect a 3-point improvement in pass rate with confidence?
- 1 min — state baseline, effect, α, power
- 3 min — unpaired formula, then paired with discordance
- 2 min — what the current 400-item set can actually detect
- 2 min — clustering, judge noise, multiple comparisons, the gate policy
It's a power calculation. For two independent samples at baseline p₁ = 0.70 and target p₂ = 0.73, α = 0.05 two-sided, power 0.8: n per arm = (1.96 + 0.84)² × (p₁q₁ + p₂q₂) ÷ δ² ≈ 3,550 per arm. If both systems run on the same items (paired design, McNemar test), what matters is the discordant rate — items where the two systems disagree — and n drops to ~870 when 10% of items are discordant.
A clinical intake-note summariser with a 400-item golden set. A prompt change scores 73% vs 70%. The product lead wants to ship. With 400 items, the 95% confidence interval on a paired difference is about ±3 points — the result is noise until proven otherwise.
Default to paired comparisons on the same items, report the confidence interval not just the delta, and run each item 3–5 times when sampling temperature > 0 to separate model variance from item variance. Then make the standard organisational: the eval platform team publishes a minimum detectable effect for each suite, and the eval gate refuses to call a win smaller than it. That's the change that stops teams shipping noise.
Assumptions: baseline 70% pass, target 73% (δ = 0.03), α = 0.05 two-sided (z = 1.96), power 0.8 (z = 0.84).
- Unpaired: (2.80)² × (0.70 × 0.30 + 0.73 × 0.27) ÷ 0.03² = 7.84 × 0.407 ÷ 0.0009 ≈ 3,550 per arm.
- Paired (McNemar): discordant rate p_d = 0.10 (6.5% flip to pass, 3.5% flip to fail): n ≈ (1.96 √0.10 + 0.84 √(0.10 − 0.0009))² ÷ 0.0009 = (0.620 + 0.264)² ÷ 0.0009 ≈ 870 items.
- Your 400 items: standard error of the paired difference ≈ √(0.10/400) ≈ 1.6 pts → 95% CI ≈ ±3.1 pts: a 3-point gain is not distinguishable from zero.
- Clustered items (e.g. 5 questions per source document): inflate n by the design effect 1 + (m − 1) × ICC; with m = 5, ICC = 0.2 → ×1.8.
- Cost: 870 items × 2 systems × 3 samples × ~$0.01 per LLM-judged item (illustrative) ≈ $52 per comparison — cheap; labelling the items is the real cost.
What the answer sounds like at each level
Knows 400 items is too few for 3 points and computes an unpaired sample size.
Uses the paired design, explains discordant pairs, and accounts for stochastic outputs and judge noise.
Turns it into a platform standard: published minimum detectable effect per suite, CI-gated, with clustered errors for correlated items.
Follow-ups the interviewer will push
1Now it's a 1-point improvement.
n scales with 1/δ²: ~9× more, about 7,800 paired items at 10% discordance. At that point use online A/B testing on live traffic or accept that 1-point wins aren't measurable offline.
2Now the grader is an LLM judge that agrees with humans 85% of the time.
Judge noise adds variance and can bias the delta. Calibrate the judge on a human-labelled subset, re-score both systems with the same judge version, and widen n by roughly the judge-noise variance — or use the human subset for the final call.
3The team runs 20 prompt variants and picks the best. Problem?
Multiple comparisons: with 20 tries at α = 0.05 you expect one false win. Hold out a fresh test split for the winner, or apply a Bonferroni/Holm correction.
Red flags interviewers listen for
- Reports a single number without an interval and calls it a win.
- Doesn't know that pairing on the same items reduces the required sample size.
- Ignores run-to-run variance at non-zero temperature.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Statistics | Gets ~3,500 unpaired and ~900 paired and explains why they differ. |
| Application | Computes the CI for the existing set and says the result is inconclusive. |
| Rigour | Addresses clustering, temperature variance and multiple comparisons. |
| Org | Proposes minimum detectable effect as a published, enforced standard. |
Sources
- Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations — Miller, 2024paper
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Zheng et al., 2023paper