LLM-as-judge without fooling yourself
A model can grade open-ended answers at scale, but only after you have measured its true-positive and true-negative rates against people and corrected the pass rate it reports.
Picture a team that summarises clinical intake notes for a triage nurse. Their LLM judge reports 85% of summaries faithful to the source note, up from 81% last month, and the dashboard is green. Then a nurse flags a summary that dropped “allergic to penicillin”. A clinician hand-labels 200 recent summaries against the same rubric: the judge had passed 18 of the 40 summaries she failed. The judge was not measuring faithfulness. It was measuring faithfulness plus a 45% chance of waving a bad summary through.
LLM-as-judge is the right tool for grading open-ended output at scale — Zheng et al. (2023) found strong judges agree with human preferences about as often as humans agree with each other on their benchmarks. But a judge is a measuring instrument, and an uncalibrated instrument produces confident wrong numbers. This post covers the prompt, the biases, the calibration, and the step most teams skip: correcting the judge’s reported pass rate for its known error rates.
Write the rubric as binary questions
“Rate this summary 1–10” produces numbers without meaning: the judge invents a scale and it drifts between calls. Use a rubric of binary, single-purpose criteria, one criterion per judge call (or one field per criterion), with a pass and a fail example at the boundary. Ask for the evidence before the verdict, so the verdict is conditioned on it and you can debug disagreements by reading it.
Here is a real faithfulness judge prompt for the intake-note case. Note what it does: one criterion, a definition of what counts, explicit handling of the borderline cases that caused disagreements, evidence before verdict, and a fixed JSON shape you can parse.
You are checking whether a summary of a clinical intake note is FAITHFUL
to the note. You are not judging style, length or helpfulness.
FAITHFUL means both:
1. Every clinical fact in the summary is stated in the note.
2. The summary does not omit any of these if present in the note:
allergies, current medications, presenting complaint, red-flag symptoms.
Count as UNFAITHFUL:
- a fact that is plausible but not in the note (e.g. a dose the note omits)
- a negation flipped ("denies chest pain" summarised as "chest pain")
- any omitted allergy, even if the summary is otherwise excellent
Do NOT count as unfaithful:
- rewording, abbreviations (BP, SOB), or merging two sentences
- leaving out non-clinical details (parking, insurance, scheduling)
<note>
{note}
</note>
<summary>
{summary}
</summary>
First list each clinical claim in the summary and quote the supporting
sentence from the note, or write NOT FOUND. Then list any required
category present in the note but missing from the summary.
Finally answer with JSON on the last line and nothing after it:
{"faithful": true|false, "reason": "<one sentence>"}Run it at temperature 0 with a pinned model version, parse only the last line, and treat unparseable output as a judge failure to be counted, not silently dropped.
Pointwise or pairwise
Pointwise grading applies the rubric to one answer and gives an absolute pass rate — the right tool for a regression gate over a golden set. Pairwise grading shows two answers and asks which is better; comparison is easier than absolute scoring for models and people, so it is more sensitive for choosing between two prompts. It cannot tell you whether either is good enough.
Pointwise
- One answer vs rubric
- Absolute pass rate
- Regression gates
- Needs TPR/TNR correction
Pairwise
- Two answers, pick one
- Relative win rate
- Prompt or model A vs B
- Needs order swapping
Biases you will measure, not prompt away
| Bias | Symptom | Counter-measure |
|---|---|---|
| Position | Prefers slot A or B | Judge both orders; a flip counts as a tie |
| Verbosity | Longer looks better | Penalise padding; compare at similar length |
| Self-preference | Favours its own family | Judge with a different model family |
| Leniency | Most answers pass | Measure TNR; correct the pass rate |
Track the flip rate under order swapping as a judge-health metric: above about 20% of pairs, the judge is guessing on close calls and your win rates are mostly noise. For verbosity, check whether a prompt change’s win rate survives when you only compare answers within ±20% of each other’s length.
Calibrate: measure TPR and TNR
Sample 200 real outputs, oversampling borderline ones. Have two people label them with the same rubric, measure their agreement (Cohen’s kappa, 1960), and resolve disagreements to a single human label. Then run the judge and build the confusion table, treating “pass” as the positive class:
- TPR (true-positive rate) = of the answers humans passed, the share the judge passed.
- TNR (true-negative rate) = of the answers humans failed, the share the judge failed. This is the number a gate lives or dies on.
| Judge: pass | Judge: fail | Total | |
|---|---|---|---|
| Human: pass | 152 | 8 | 160 |
| Human: fail | 18 | 22 | 40 |
| Total | 170 | 30 | 200 |
87% raw agreement sounds fine. TNR of 0.55 says the judge misses 45% of bad summaries. Raw agreement hides it because 80% of items are passes; a judge that always said “pass” would score 80% agreement.
Correct the pass rate the judge reports
Now the step that turns a calibrated judge into a usable number. On last week’s 1,000 production summaries, the judge passed 850: an observed rate of 85%. The judge’s observed pass rate mixes true passes it caught with true fails it let through:
observed = TPR × true + (1 − TNR) × (1 − true)
Solve for the true rate and you get the Rogan–Gladen estimator, from epidemiology, where it corrects disease prevalence for an imperfect screening test (Rogan & Gladen, 1978):
true = (observed + TNR − 1) / (TPR + TNR − 1)
Worked through: (0.85 + 0.55 − 1) / (0.95 + 0.55 − 1) = 0.40 / 0.50 = 0.80. The judge says 85%; the best estimate of the real faithfulness rate is 80%. Check it forwards: 0.95 × 0.80 + 0.45 × 0.20 = 0.76 + 0.09 = 0.85. ✓
The correction has its own uncertainty, mostly from TNR being estimated on only 40 human fails. Bootstrap both samples to get an interval:
import random
# Calibration set: 200 answers labelled by a human and by the judge.
# (human, judge, count)
CALIBRATION = [("pass", "pass", 152), ("pass", "fail", 8), ("fail", "pass", 18), ("fail", "fail", 22)]
# Production sample scored by the judge alone.
PROD_JUDGE_PASS, PROD_N = 850, 1000
def expand(cells: list[tuple[str, str, int]]) -> list[tuple[str, str]]:
return [(h, j) for h, j, c in cells for _ in range(c)]
def rates(pairs: list[tuple[str, str]]) -> tuple[float, float]:
"""TPR = judge passes a human pass; TNR = judge fails a human fail."""
tp = sum(h == "pass" and j == "pass" for h, j in pairs)
pos = sum(h == "pass" for h, _ in pairs)
tn = sum(h == "fail" and j == "fail" for h, j in pairs)
return tp / pos, tn / (len(pairs) - pos)
def corrected(observed: float, tpr: float, tnr: float) -> float:
"""Rogan-Gladen: invert observed = tpr*true + (1-tnr)*(1-true)."""
return min(1.0, max(0.0, (observed + tnr - 1) / (tpr + tnr - 1)))
def main() -> None:
pairs = expand(CALIBRATION)
tpr, tnr = rates(pairs)
observed = PROD_JUDGE_PASS / PROD_N
print("judge TPR %.2f TNR %.2f" % (tpr, tnr))
print("judge says %.1f%% pass; corrected %.1f%%" % (100 * observed, 100 * corrected(observed, tpr, tnr)))
# Bootstrap both samples: the correction inherits the noise of a 40-fail calibration set.
rng = random.Random(11)
prod = [1] * PROD_JUDGE_PASS + [0] * (PROD_N - PROD_JUDGE_PASS)
draws = []
for _ in range(4000):
t, n = rates(rng.choices(pairs, k=len(pairs)))
o = sum(rng.choices(prod, k=PROD_N)) / PROD_N
draws.append(corrected(o, t, n))
draws.sort()
print("95%% CI %.1f-%.1f%%" % (100 * draws[100], 100 * draws[3899]))
if __name__ == "__main__":
main()judge TPR 0.95 TNR 0.55
judge says 85.0% pass; corrected 80.0%
95% CI 68.1-88.4%Three things follow. The dashboard should show 80% (68–88%), not 85%. The interval is wide because 40 human fails is few; labelling 100 more fails would tighten it more than any judge-prompt tweak. And the correction only holds while TPR and TNR hold, so re-label 50–100 fresh items every time the input mix, the judge prompt or the judge model changes. When TPR + TNR approaches 1, the denominator approaches zero and the judge carries no information at all; below TPR + TNR ≈ 1.3, fix the rubric before trusting any corrected number.
Keeping the judge honest
- Version the judge like any prompt; a judge change re-baselines every score, so rerun the baseline with it.
- Pin the judge model version. An alias upgrade silently changes TPR and TNR.
- Seed known answers in every run — 10 known-faithful, 10 known-unfaithful. If more than one known-bad item passes, alert.
- Re-calibrate monthly or on any input-mix shift, and recompute the correction.
Key takeaways
- Write the judge as one binary criterion per call, with boundary examples and evidence before the verdict.
- Raw agreement hides leniency when most items pass; report TPR and especially TNR.
- Correct the judge’s observed pass rate with Rogan–Gladen: true = (observed + TNR − 1) / (TPR + TNR − 1). In the worked example, 85% reported is 80% real.
- Bootstrap the correction: the interval is usually dominated by how few human-labelled failures you have.
- Swap order for pairwise, compare at similar lengths, judge with a different model family, and re-calibrate whenever the judge or the inputs change.
Sources
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Zheng et al., 2023paper
- Estimating prevalence from the results of a screening test — Rogan & Gladen, 1978paper
- A Coefficient of Agreement for Nominal Scales — Jacob Cohen, 1960paper