Eval gate (CI)
A CI check that runs the eval suite on every prompt, model or config change and blocks the merge or rollout if quality drops beyond the measured noise.
An eval gate is an automatic quality check in your release pipeline. When someone changes a Prompt & system prompt, swaps the model or edits retrieval settings, CI runs the Eval harness on the Golden set and compares the result with the current production version. If quality got significantly worse, the change is blocked.
It is the AI equivalent of 'tests must pass before merge', with one twist: AI outputs vary from run to run, so the gate has to tell a real regression from noise.
A quality-control station on a production line that pulls a sample from every batch; it does not reject a batch for one slightly odd item, but it stops the line when the sample says the batch is worse.
Gating on a raw score drop of 'more than 1 point' from a single run at temperature 0. Run-to-run noise on 200 cases is several points, so the gate either flaps on noise and gets disabled, or is loosened until it catches nothing.
Design the gate around noise, not determinism. Setting Temperature & sampling to 0 does not make hosted models reproducible: batching and floating-point non-associativity change results from run to run (He / Thinking Machines, 2025), and many Reasoning models & extended thinking fix temperature entirely (Anthropic's Claude 4.7+ models return an error for non-default values; OpenAI's o-series accept only the default). So run baseline and candidate on the same cases, 3 samples each, and compare paired.
The gate rule is a non-inferiority test: block if the lower bound of the 95% CI on (candidate − baseline) is below −δ (e.g. δ = 2 points), or if any critical slice (safety, PII, policy) drops at all on its deterministic checks. Pairing matters: on 300 cases with 10% discordant pairs the 95% CI on the delta is about ±3.6 points, versus ±6.4 for two independent runs. Version everything the result depends on (prompt, model snapshot, retriever config, judge prompt) via Prompt versioning, cache baseline results per version, and run a 100-case smoke slice per PR with the full suite nightly. Passing the gate earns a Canary & ramped rollout, not a full rollout.
GitHub Actions or GitLab CI job running promptfoo, Inspect or a pytest harness; 100-case smoke slice per PR, full 300–500 cases nightly and before release; 3 samples per case for both baseline and candidate; paired bootstrap or McNemar on the delta; block if CI lower bound < −2 points or any safety/PII deterministic check regresses; cache baseline results keyed by (prompt version, model snapshot, retriever config, judge version).
Internal-wiki assistant, prompt v41 → v42. Gate runs 300 cases × 3 samples for both. Paired pass rates: 84.0% vs 82.7%, delta −1.3 points, 95% CI [−3.4, +0.8]. Lower bound −3.4 is below δ = −2, so the gate blocks, even though the point estimate looks like a harmless dip. The author adds 200 cases targeting the changed instructions; with 500 cases the CI is [−2.8, +0.4], still blocked, and per-slice results show the drop is concentrated in table-heavy pages. Fixed in v43, which passes with [−1.1, +1.9].
Say: "My gate is a paired non-inferiority test with repeated samples; I don't rely on temperature 0, because hosted inference isn't deterministic and many reasoning models won't accept it." Expect: "what's your threshold?": a margin like 2 points on the CI lower bound, zero tolerance on safety slices, and why the margin comes from the business, not the statistics.
| Setup | Pass rate | 95% CI on Δ | Verdict |
|---|---|---|---|
| 1 run, T=0, unpaired | 84 → 82.7 | ±6.4 | noise |
| 3 samples, paired, n=300 | 84 → 82.7 | [−3.4, +0.8] | block (< −2) |
| 3 samples, paired, n=500 | v43 | [−1.1, +1.9] | pass |
Numbers worth knowing
- Paired vs unpaired 95% CI on a delta
- ±3.6 vs ±6.4 pts300 cases, 80% pass rate, 10% discordant pairs
- Non-inferiority margin
- 1–3 pointsset by the product owner; zero for safety slices
- Per-PR smoke run budget
- < 10 minutes, < $3100 cases × 3 samples, parallelism 20, cached baseline
How to decide
| When | Do this |
|---|---|
| The model or provider does not support temperature control | Keep defaults, run 3–5 samples per case, and gate on the paired CI. |
| The CI straddles the margin | Add cases on the changed behaviour and re-run; do not re-run the same set until it passes. |
| A safety, PII or policy check regresses on any case | Block regardless of the overall score. |
| The gate passes | Promote to a 5% Canary & ramped rollout with guardrail metrics, not straight to 100%. |
Cost maths
Per PR: 100 cases × 3 samples × 2 variants (baseline cached, so ×1) × (3k in + 500 out) at $3 / $15 per M ≈ 300 × $0.0165 ≈ $5, plus judge ≈ $0.70. With a cached baseline and prompt caching on the shared system prompt (reads at 0.1× input), ≈ $3. 30 PRs a day ≈ $90/day; the nightly full run on the Batch API (50% off) adds ≈ $9.
Operating it
- Gate flaps (blocks then passes with no change) more than once a week: increase samples or n; the margin is inside the noise.
- PR eval queue exceeds 15 minutes: shard cases across workers and cache baselines per version.
- More than one model in production: gate each model × prompt pair separately.
- Gate flap rate (block → pass on re-run with no change): alert above 5% of PRs.
- Gate run time p95: alert above 15 minutes.
- Post-deploy online judge score vs the gate's predicted score: alert on a gap above the CI half-width.
When it goes wrong
A team disabled its eval gate after it blocked 9 of 20 harmless PRs in a month; two weeks later a prompt edit shipped that dropped citation accuracy by 7 points.
Online judge sampling showed the citation-criterion pass rate falling from 91% to 84% the day after deploy.
The gate compared a single temperature-0 run against a fixed number (−1 point). Hosted T=0 outputs varied by ±3 points run to run, so it flapped and lost the team's trust.
Re-enabled with 3 samples per case, paired comparison, a −2 point non-inferiority margin on the CI lower bound, and a zero-tolerance rule on the citation criterion; flaps fell to zero in the next month.
Related terms
Where it shows up
Sources
- Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations — Miller (Anthropic), 2024paper
- Defeating Nondeterminism in LLM Inference — He / Thinking Machines Lab, 2025blog
- Your AI Product Needs Evals — Hamel Husain, 2024blog
Check yourself
Three questions. Every option is explained once you answer.