Eval harness
The code that runs a system against a golden set, scores outputs with metrics and judges, and reports diffs, ideally on every pull request.
An eval harness is the test runner for AI behaviour. It takes your Golden set, runs each item through the real pipeline (retrieval, prompt, model), scores the results with metrics such as Recall@k or Faithfulness, and prints a report comparing this run with the last one. Wired into CI, it stops a change that makes answers worse from shipping.
A car's emissions test rig: the same driving cycle every time, the same sensors, and a pass/fail line.
A harness that re-implements the pipeline in a notebook. It drifts from production within weeks, and you end up evaluating a system nobody runs.
Components: a dataset loader (versioned), a runner that calls the production code path with config overrides, scorers (exact match, retrieval metrics, regex/JSON-schema checks, LLM-as-judge with a pinned judge model and rubric), and a reporter that diffs against a baseline per slice with CIs. Make runs reproducible: temperature 0 where possible, fixed seeds, pinned model snapshot ids, cached retrieval and LLM responses keyed by input hash so unchanged stages are free. Run a fast subset (50–100 items, < 5 min) on every PR and the full set nightly or pre-release; gate merges with an Eval gate (CI). Off-the-shelf as of Oct 2026: promptfoo, Ragas, DeepEval, OpenAI Evals, Inspect (UK AISI), lm-evaluation-harness for model benchmarks, and hosted platforms (Braintrust, LangSmith, Langfuse). The harness is only as good as its judge: calibrate it against human labels before trusting a 2-point delta.
promptfoo or a thin in-house runner around the production service; pinned model snapshot ids for system and judge; response cache keyed by (stage, input hash, config hash); PR tier of 50–100 stratified items in < 5 min; nightly full run; results stored per run for trend charts.
Insurance claims summariser. PR changes the chunking from 512 to 1,024 tokens. Harness runs 80 PR-tier items in 4 min: retrieval recall@10 +3.1, faithfulness −0.02 (within noise), 'cites policy section' check 96% → 89%. The per-slice report shows the drop is entirely in multi-peril claims, where bigger chunks mix sections. The PR is blocked by the gate on the citation check; the author adds section-boundary splitting and reruns: 95%.
Say: the harness calls the same code path as production, caches unchanged stages, runs a 5-minute subset on every PR and the full set nightly, and reports per slice with confidence intervals. Expect: 'the judge is non-deterministic, how do you gate on it?' Pin the judge model and prompt, run twice on disagreement, and gate only on deltas larger than the measured judge noise.
Numbers worth knowing
- PR-tier runtime
- < 5 min50–100 items, cached retrieval, parallelism 16–32
- Judge run-to-run noise
- 1–3 pointssame items, temperature 0, pinned judge; measure yours
- Full-run cost
- $5–30500 items, system + judge calls, mid-tier model prices
How to decide
| When | Do this |
|---|---|
| Team < 5, one product | promptfoo or Ragas in CI; no platform |
| Several teams share models and judges | Central harness service with shared judges, datasets and dashboards |
| Model provider releases a new snapshot | Full run on both snapshots before switching the pin |
Cost maths
Per full run: 500 items × (5k in + 400 out at $3/$15 per M) = $10.50 system + 500 × judge (3k in + 100 out at $0.80/$4 per M) = $1.40 → ~$12. Nightly = ~$360/month; PR tier at 100 items (~$2.40 a run) × 40 PRs/week ≈ $410/month. Caching unchanged stages typically halves both.
Operating it
- PR tier takes > 10 min: developers start skipping it
- More than one team writes its own judge for the same metric
- Release decisions argued over noise: add CIs and repeated runs
- Run-to-run variance on an unchanged control commit; alert if > 3 points
- PR-tier wall-clock time; alert > 10 min
- Share of PRs merged with the gate overridden; review weekly, alert > 10%
When it goes wrong
Nightly faithfulness jumped 6 points overnight with no code change
Trend chart anomaly; the run log showed the judge model id had changed
The judge was configured with a floating alias; the provider moved it to a newer, more lenient snapshot
Pinned dated snapshot ids for judge and system, added a CI assertion that rejects aliases, and re-baselined after calibrating the new judge against 100 human labels
Related terms
Where it shows up
Sources
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Zheng et al., 2023paper
- Ragas: Automated Evaluation of Retrieval Augmented Generation — Es et al., 2023paper
- lm-evaluation-harness — EleutherAI, 2023docs
- Your AI Product Needs Evals — Husain, 2024blog
Check yourself
Three questions. Every option is explained once you answer.