Debugging drill: after upgrading the LLM judge model, every system's faithfulness score rose 6 points overnight. What happened and what do you do?
- 1 min — symptoms: uniform shift, unchanged systems moved
- 2 min — ranked hypotheses
- 3 min — calibration-set re-score: offset, κ, parse failures, length slices
- 2 min — fix (re-baseline, version) and prevention (shadow period, ownership)
Symptoms: uniform +6 shift across all systems, including ones that didn't change; rank order mostly preserved; variance changed. Hypotheses ranked: (1) the new judge is more lenient — a calibration offset, not a quality change; (2) the new judge has different biases (prefers longer answers, or position bias in pairwise mode); (3) output-format drift — the score parser falls back to a default on unparseable verdicts; (4) the rubric anchors are interpreted differently; (5) a real system change shipped in the same window.
The one query that splits them: re-score a frozen, human-labelled calibration set (300 items) with both the old and new judge. Compare each to the humans: if agreement (Cohen's κ) is similar but the mean is offset → (1); if agreement drops on long answers → (2); count parse failures and default scores → (3). If the frozen set shows no shift, the change is real → (5). Fix: treat the judge as part of the metric — version it, re-baseline every system with the new judge, never compare scores across judge versions, and publish the calibration result with the upgrade.
Prevention: a judge upgrade is a migration with a shadow period: run both judges in parallel for two weeks, publish the offset and κ against humans, then cut over and re-baseline historical dashboards. The eval platform team owns judge versions and the calibration set; product teams consume a score that carries judge_version so no one compares apples to oranges. Report it honestly to leadership: the 6 points weren't a win, and the dashboards that showed them get annotated.
Calibration set sizing: 300 human-labelled items, binary faithful/unfaithful, base rate 75%. Standard error on the mean shift (paired, same items) ≈ √(p_d / 300); if the judges disagree on 12% of items, SE ≈ 2 pts, so a 6-point offset is ~3 SE — real. κ estimates at n = 300 have a 95% CI of roughly ±0.08, enough to tell 0.75 from 0.55 but not 0.75 from 0.72. Cost: 300 items × 2 judges × ~$0.01 ≈ $6 per calibration run; the human labels (~10 hours at 2 min/item) are the real cost and are reused for every future upgrade.
What the answer sounds like at each level
Suspects the judge and reruns old outputs with the old judge.
Uses a human-labelled calibration set, measures κ and offset, checks parse failures and length bias.
Makes judge upgrades a versioned migration with a shadow period, owns the calibration set at platform level, and corrects the record on dashboards.
Follow-ups the interviewer will push
1Now the new judge agrees with humans better (κ 0.78 vs 0.70). Keep it?
Yes — but re-baseline every system with it, annotate the discontinuity on every dashboard, and adjust any eval gate thresholds that were set in old-judge units.
2Now the judge provider deprecates the old model before you've validated the new one.
Pin the judge with a dated model id, keep the calibration set ready so validation is a one-hour job, and keep an open-weights judge as a fallback whose calibration you control.
Red flags interviewers listen for
- Celebrates the improvement.
- No human-labelled ground truth to calibrate the judge against.
- Compares scores across judge versions on the same chart without annotation.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Diagnosis | Recognises a uniform shift on unchanged systems as a measurement change, not a quality change. |
| Method | Frozen human-labelled set, κ plus mean offset, bias slices. |
| Governance | Judge version as part of the metric, shadow period, dashboard annotation. |
Sources
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Zheng et al., 2023paper
- Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations — Miller, 2024paper