Eval-driven development
Decide which number a change should move, build the cases that measure it, then change the prompt, model or retriever and read the delta with error bars.
Eval-driven development means you decide how to measure "good" before you start tweaking an AI feature. You collect example inputs with what a good answer looks like (a Golden set), write graders that score each answer, and re-run them every time you change the Prompt & system prompt, the model or the retrieval.
It is test-driven development for systems whose output is a paragraph instead of a return value: instead of trying a change on three questions and deciding it feels better, you check whether it got better on 200, and on which kinds of question it got worse.
A chef who keeps a tasting panel and a scorecard, and re-runs the whole tasting every time the recipe changes, instead of tasting one spoonful and declaring victory.
Starting with an off-the-shelf metric pack (generic 'helpfulness', 'coherence') before reading a single trace. Those scores move with prompt length and style, not with the failures your users hit; the useful cases come from your own Error analysis.
The loop: read 50–100 real traces and cluster the failures (Error analysis), turn each failure mode into cases with a grader, record a baseline, change one variable, re-run, compare per slice. Graders are layered by cost: deterministic checks first (JSON schema, regex, citation present, tool called with the right arguments), then LLM-as-judge against a binary Rubric, with the judge validated against human labels (target Cohen's kappa ≥0.6) before anyone trusts it.
Outputs are stochastic, so a delta is only real if it survives error bars: at n=200 and an 80% pass rate the 95% interval is ±5.5 points, so a 3-point gain on one run is noise. Compare baseline and candidate on the same cases (paired), with 3 samples per case for generative tasks. The suite is alive: production failures from the Data flywheel become cases weekly, and slices that sit at 100% for a month are made harder or retired. Once trusted, it runs in CI as an Eval gate (CI).
A golden-set of 100–300 cases in version control (JSONL), sliced by intent; graders in order of cost: schema/regex/code checks, then a binary LLM judge validated to kappa ≥0.6 on 100 human-labelled cases. Harness: promptfoo, Inspect (UK AISI), Braintrust or a 200-line pytest runner. 3 samples per case for generative outputs; report per-slice pass rate with a 95% CI and a paired delta against the pinned baseline.
Code-search assistant for a 4M-line monorepo, 240 cases in 6 slices (symbol lookup, 'where is X configured', cross-repo call chains…). Baseline 71% pass. A rewritten system prompt plus k=12 instead of k=8 lifts call-chain cases from 58% to 79% (n=40, ±12 pts each, paired delta still significant) but drops 'where is X configured' from 83% to 66% because the longer context buries YAML snippets. Overall: 71% → 73%, which a single headline number would have called a modest win.
Say this: "Before I touch the prompt I write down which slice I expect to move and by how much, and I size the eval so that move is outside the error bars." The follow-up is usually "how big does the set need to be?": answer with the CI arithmetic (±12 points at n=40, ±5.5 at n=200 for an 80% pass rate) and pairing.
| Slice | n | Baseline | Candidate | Read |
|---|---|---|---|---|
| Symbol lookup | 60 | 88% | 90% | noise (±8) |
| Call chains | 40 | 58% | 79% | real, paired |
| Config location | 50 | 83% | 66% | regression |
| Overall | 240 | 71% | 73% | hides both |
Numbers worth knowing
- 95% CI half-width, 80% pass rate
- ±12.4 pts at n=40; ±5.5 at n=200; ±2.5 at n=1,000normal approximation, independent cases
- Traces to read before writing graders
- 50–100enough to see the top 3–5 failure modes repeat
- Judge–human agreement before trusting a judge
- Cohen's kappa ≥0.6, ideally ≥0.7on ≥100 labelled cases per criterion
- Eval run cost, 300 cases × 3 samples
- ≈ $17 at list, ≈ $8.50 via Batch APISonnet 4.6-class generator plus Haiku-class judge, assumptions in costMath
How to decide
| When | Do this |
|---|---|
| The feature has no eval yet and is about to ship | Read 100 traces, write 30–50 cases for the top failure modes this week; do not wait for a 'complete' suite. |
| A change moves the headline score by less than the CI half-width | Treat it as no change; increase n or use paired comparison before claiming a win. |
| A slice passes 100% for four consecutive weeks | Make the cases harder or move them to a cheap nightly smoke run; they no longer discriminate. |
| The grader is an LLM judge | Hold out 100 human-labelled cases and re-check kappa whenever the judge model or rubric changes. |
Cost maths
Run cost = cases × samples × (generation tokens × price + judge tokens × price). Assume Sonnet 4.6-class list prices, Oct 2026: $3 / M input, $15 / M output. 300 cases × 3 samples × (3k in + 500 out) = $8.10 + $6.75 ≈ $14.9 for generation; a Haiku-class judge at $1 / $5 per M with 1.5k in + 150 out adds ≈ $2. Through the Batch API (50% off) ≈ $8.50 per full run. At 20 PRs a day that is ~$170/day, which is why CI runs a 100-case smoke slice per PR and the full suite nightly.
Operating it
- More than one team edits the same prompt or retrieval config: wire the suite into an Eval gate (CI).
- Production failure reports outpace new cases being added: stand up a Data flywheel with weekly triage.
- Full-suite runtime exceeds 15 minutes: split into a per-PR smoke slice and a nightly full run.
- Per-slice pass rate on the nightly full run: alert when any slice drops more than its CI half-width versus the 7-day median.
- Share of production thumbs-down or escalations that map to an existing eval case: alert if <50% (the suite has drifted from reality).
- Judge–human kappa on the monthly re-labelled sample: alert below 0.6.
When it goes wrong
In the public Moffatt v. Air Canada case (BC Civil Resolution Tribunal, February 2024) the airline's website chatbot told a grieving passenger he could claim a bereavement fare refund retroactively; the airline's actual policy said otherwise.
Not by any internal test: the customer kept a screenshot, was refused, and filed a claim. The tribunal held the airline responsible for what its chatbot said and awarded about C$812.
Nothing checked answers against the authoritative policy text. There was no eval case of the form 'policy question → answer must match the cited policy', so a fluent, wrong answer looked identical to a right one.
The general lesson: policy-bearing answers need Faithfulness cases against the source document, a grader that fails any promise not in the source, and an Eval gate (CI) so the check runs on every prompt or model change.
Sources
- Your AI Product Needs Evals — Hamel Husain, 2024blog
- Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations — Miller (Anthropic), 2024paper
- Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences — Shankar et al., 2024paper
Check yourself
Three questions. Every option is explained once you answer.