What does eval-driven development look like day to day?
- 2 min — the loop: failing case → change → full suite → error analysis
- 4 min — worked example including a side-effect regression
- 3 min — tiering, caching, iteration time budgets
- 3 min — measuring the eval program and ownership
Eval-driven development applies test-driven discipline to non-deterministic systems: before changing a prompt, model or retrieval setting, you write or extend the eval that would show the change worked, confirm it fails, then iterate against it. Every bug report becomes an eval case; every change is judged by the harness with intervals, not by eyeballing five outputs. The suite is tiered by cost: a smoke set in seconds, a CI gate in minutes, a full run nightly, human review weekly.
An insurance-claims assistant tells a customer a water-damage claim is covered when the policy excludes gradual leaks. Step one: add the case plus five paraphrases and two near-miss variants to the golden set; confirm all fail. Step two: change the prompt to cite the exact policy clause and abstain when no clause is retrieved. Step three: rerun the full suite — new cases pass, but error analysis shows abstention on answerable questions rose from 3% to 9%. Tune until it is back under 4%, then ship through the eval gate.
The cost is iteration time: if the suite takes an hour and $200, people stop running it. Target under 2 minutes for smoke, under 10 for the CI gate, and cache unchanged item results by hash(item, prompt version, model, params). Measure the eval program itself: what fraction of eval wins held up online, and how many user-reported regressions slipped past the gate per quarter. Where it breaks: evals that test only what was easy to check, teaching to the test, and judge prompts edited outside version control. Org scope: the platform team owns the harness, caching and CI integration once; each feature team owns its cases and rubric; a weekly 30-minute read of real transcripts keeps the suite honest.
Smoke: 80 items × 1 run × $0.025 ≈ $2, ~90 s at 32-way concurrency. CI gate: 300 items × 3 runs ≈ $23, ~6 min. Nightly full: 1,200 items × 3 runs ≈ $90. At 40 PRs/week, eval spend ≈ $1.6k/month — cheaper than one escaped regression.
What the answer sounds like at each level
Write a test case for each bug before fixing it, and run the eval suite after every prompt change.
Tier the suite by cost and latency, turn every bug into cases with variants, gate CI on paired intervals, and read transcripts weekly to catch what evals miss.
I build the harness and caching once for every team, measure the eval program by escaped regressions and offline→online hit rate, and keep feature teams owning their cases and rubrics.
Follow-ups the interviewer will push
1The suite takes 45 minutes and engineers skip it. Fix?
Split into a sub-2-minute smoke set on every edit and a 10-minute CI gate, cache unchanged item results by hash, run the full set nightly, and parallelise at 32–64 concurrent calls within provider rate limits.
2How do you stop teams from overfitting the suite?
Hold out 20% of items nobody sees in day-to-day runs, check it monthly, and promote fresh production failures every week so the target keeps moving.
Red flags interviewers listen for
- Fixes a reported bug without first adding a failing case.
- Runs the whole expensive suite on every edit, or none at all.
- Treats the eval score as the goal with no transcript reading or held-out slice.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Loop discipline | Adds failing cases with variants before changing anything and checks side effects. |
| Engineering economics | Gives time and cost budgets per tier and uses caching. |
| Meta-measurement | Tracks escaped regressions and offline→online hit rate. |
Turning bug reports into a regression suite
Illustrative exampleI was tech lead on a 7-engineer team running an internal developer-docs assistant with ~4,000 weekly users. We shipped prompt changes about three times a week after spot-checking a handful of outputs. In one quarter, 9 of 23 prompt changes caused a regression that users reported, and nobody could say whether quality was trending up or down.
My goal was to make prompt changes measurable without slowing the team. I owned the design and the first version of the harness; two engineers owned labelling and CI wiring with me.
I pulled 140 bug reports from the previous two months and turned them, with one other engineer, into 210 cases with 7 binary rubric criteria. For 4 criteria an LLM judge reached κ ≥ 0.7 against our labels on 150 items, so we automated those; the other 3 became deterministic checks in code. I wired an 80-item smoke set into CI as an eval gate (under 6 minutes) and ran the full suite nightly with per-slice confidence intervals posted to the team channel. Our PM disagreed: she expected the gate to halve shipping velocity. I proposed a two-week trial with an override label and agreed to roll it back if median PR cycle time rose by more than half a day; we tracked it publicly.
Median PR cycle time went from 1.6 to 1.9 days, inside the agreed bound, so the gate stayed. User-reported regressions dropped from 9 of 23 changes to 2 of 26 the following quarter, and the override was used 3 times, each with a written reason. What I'd do differently: I left the judge prompt outside version control for the first month; an untracked edit shifted baselines by 3 points and we spent a week chasing a regression that didn't exist.
Sources
- Your AI Product Needs Evals — Hamel Husain, 2024blog
- Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences — Shankar et al., 2024paper