How do you put eval gates into CI for an LLM application?
- 2 min — what the gate checks: hard thresholds vs non-inferiority
- 4 min — repeats, paired bootstrap, three outcomes
- 3 min — why T = 0 is not the answer
- 3 min — caching, A/A flake monitoring, rebaselining, ownership
An eval gate is a CI step that runs the eval harness on a change and blocks the merge if must-pass criteria fail or quality regresses versus the baseline. Two kinds of check: hard thresholds for must-pass criteria (all 40 safety items pass on every repeat; outputs parse as valid JSON via structured output), and non-inferiority for quality slices — the lower bound of the paired 95% interval on (candidate − baseline) must sit above −2 points.
Every PR touching prompts, retrieval or model config runs a 120-item gate set, 4 repeats per item at production decoding settings. Deterministic checks run first (schema, required citations, banned content) and short-circuit on failure. Judge-scored criteria are averaged per item across repeats, then a paired bootstrap against main's cached baseline gives an interval per slice. Outcome is pass, fail (regression proven or must-pass broken) or 'inconclusive — needs review', posted as a PR comment with the intervals.
Naive 'score must not drop' gates flake, and people learn to override them. Do not 'fix' flakiness by setting temperature to zero: hosted inference is not bitwise deterministic at T = 0 (batch-dependent kernels, MoE routing), many reasoning models fix or reject the temperature parameter, and you would be gating a configuration you don't ship. Instead gate on N repeated runs with a paired test and a confidence interval. Measure gate health with A/A runs: run the gate on no-op PRs weekly; if it fails more than 2% of them, the margin or the sample is wrong. Cache item results by hash(item, prompt version, model snapshot, params) so unchanged items cost nothing. Rebaseline deliberately when the judge model changes — never silently. Ownership: the platform owns harness, caching and the statistics; feature teams own items, thresholds and margins in a reviewed config file.
120 items × 4 repeats = 480 generations + 480 judge calls. At ~$0.015 per generation and ~$0.010 per judge call ≈ $12 per PR. At 40 PRs/week ≈ $480/week before caching; with ~60% of items unchanged per PR, ≈ $190/week. Wall-clock at 32-way concurrency, ~6 s per call: 960 calls × 6 s / 32 ≈ 3 min.
What the answer sounds like at each level
Run the eval suite in CI and fail the build if the score drops below a threshold.
Hard thresholds for must-pass items, paired non-inferiority tests over repeated runs for quality slices, cached results, and an 'inconclusive' outcome instead of flaky fails.
I run gates as shared infrastructure with A/A flake monitoring, reviewed threshold configs per team, deliberate rebaselining on judge changes, and a cost and time budget per PR.
Follow-ups the interviewer will push
1Why not just set temperature to 0 to de-flake the gate?
Because it doesn't make hosted models deterministic, reasoning models often won't accept it, and it tests a configuration different from production. Repeated runs plus a paired interval remove the noise honestly.
2The gate says 'inconclusive' on half of PRs. What do you change?
The set is too small for the margin. Either widen the margin to what the business actually cares about, add items to the slices that are inconclusive, or raise repeats from 4 to 8 — and check the A/A flake rate after.
Red flags interviewers listen for
- Proposes temperature 0 as the fix for non-deterministic evals.
- Gates on a point-score drop with no interval or repeats.
- Threshold logic lives in someone's head or a dashboard, not reviewed code.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Statistical design | Repeated runs, paired test, non-inferiority margin, and an inconclusive state. |
| Determinism realism | Explains why T = 0 doesn't make hosted evals deterministic and what to do instead. |
| Operability | Caching, A/A flake rate, per-PR cost and time budget, reviewed thresholds. |
Sources
- Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations — Evan Miller (Anthropic), 2024paper
- Defeating Nondeterminism in LLM Inference — Horace He (Thinking Machines Lab), 2025blog