LLM-as-judge
Using an LLM with a rubric to grade outputs (faithfulness, relevance, correctness) so evaluation scales beyond human review.
Checking thousands of AI answers by hand is slow and expensive. LLM-as-judge means using a model to grade answers against clear criteria, like 'Is every claim supported by the provided documents?'
It makes it practical to test every change on hundreds of examples, as long as you have checked that the judge agrees with careful human reviewers.
A trained exam marker with a detailed marking scheme: much faster than the professor grading every paper, and spot-checked by the professor.
Using an uncalibrated judge with a vague 1–10 'quality' score and treating half-point changes as real improvements.
Judges work best on narrow, decomposed questions with an explicit Rubric and a binary or 3-point scale, a short rationale before the verdict, and Structured output / decoding for the output. Common uses: claim-level Faithfulness, answer relevance, Context precision, and Pairwise comparison between two system versions.
Zheng et al. (2023) found strong judges agree with human preferences over 80% of the time on MT-Bench, about the level humans agree with each other, but documented position bias, verbosity bias and self-enhancement; Panickssery et al. (2024) showed judges recognise and favour their own generations. Mitigations: swap order in pairwise and count disagreements as ties, use a judge from a different model family than the system under test, length-control the comparison, and calibrate on a human-labelled set, reporting agreement as Cohen's kappa (Inter-annotator agreement). Pin the judge model version and recalibrate whenever the judge or its prompt changes. Keep deterministic checks (exact match, SQL execution results, citation-id validity) wherever they exist: cheaper and unbiased.
Binary or 3-point rubric per criterion; rationale then verdict in a JSON schema; judge pinned to a dated model version from a different family than the generator; pairwise runs in both orders; 300-item human calibration set with kappa ≥ 0.7; 50 fresh human labels a month to detect judge drift; deterministic checks first.
Customer-support evals: 300 answers labelled by two support leads for faithfulness (human-human kappa 0.81). A claim-extraction judge agrees with the adjudicated labels on 91% of answers, kappa 0.76, so it is trusted for regression testing. It runs nightly on 1,000 sampled production answers; a prompt change that drops judged faithfulness from 0.93 to 0.87 is caught on the 5% canary, before full rollout.
Commit: a judge is a model and gets its own eval. Calibrate on 200–300 human labels, require kappa ≥ 0.7 before trusting it for gating, pin the version, swap positions in pairwise, and use a different model family from the system under test. Follow-up: 'judge scores jumped 6 points overnight, nothing shipped?': check the judge model alias and re-run the fixed anchor set before believing it.
Numbers worth knowing
- Strong judge vs human agreement
- > 80%MT-Bench pairwise preferences, Zheng et al. 2023; similar to human-human agreement
- Trust threshold
- Cohen's kappa ≥ 0.7judge vs adjudicated human labels on 200–300 items
- Position-swap disagreement
- alert above 15%pairwise comparisons; high rates mean the pair is too close to call
How to decide
| When | Do this |
|---|---|
| A deterministic check exists (exact match, execution result, citation-id) | Use it; reserve the judge for what code cannot check. |
| Comparing two system versions | Pairwise with order swap, judge from a third model family. |
| Judge kappa below 0.6 | Narrow the rubric or split the criterion; don't gate releases on it. |
Cost maths
Judge cost = samples × (in + out tokens) × price. Assume $3 / $15 per M (illustrative): 1,000 nightly samples × (3,000 in + 300 out) = $9 + $4.50 = $13.50/night ≈ $410/month. Pairwise with swap doubles it. Human labelling at ~$1 per item: 300-item calibration ≈ $300 plus 50/month ≈ $50: cheap insurance on every release decision.
Operating it
- More than one team relies on the same judge scores for release gates
- Judge spend above the human-labelling budget it replaces
- Judge-human kappa drifting below 0.7 on monthly fresh labels
- Judge-human kappa on 50 fresh monthly labels: alert below 0.7
- Score on fixed anchor set: alert on any shift above 1 point
- Position-swap disagreement rate: alert above 15%
When it goes wrong
Judged answer quality rose 6 points overnight with no product change; a team prepared to celebrate.
Re-running the fixed 300-item anchor set showed the same 6-point jump on old, unchanged outputs.
The judge was called through an unpinned model alias that the provider had moved to a newer, more lenient version.
Judge pinned to a dated model version, anchor set re-scored on every judge change, and judge version stored with every score.
Where it shows up
Sources
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Zheng et al., 2023paper
- LLM Evaluators Recognize and Favor Their Own Generations — Panickssery et al., 2024paper
Check yourself
Three questions. Every option is explained once you answer.