What are the known biases of LLM-as-judge, and how do you control for them?
- 3 min — name the four biases with evidence
- 4 min — controls for each: swap, length test, cross-family, pinning
- 4 min — calibration protocol and thresholds
- 2 min — drift monitoring and ownership
An LLM judge scores outputs with a prompt and a rubric. Documented biases: position bias — in pairwise prompts the verdict can flip when you swap the order (Wang et al. 2023); verbosity bias — longer answers win regardless of quality, which is why AlpacaEval 2.0 added length control (Dubois et al. 2024); self-preference — judges favour outputs from their own model family (Zheng et al. 2023); and judge drift — upgrading the judge model or editing its prompt silently moves every historical score. Zheng et al. found a strong judge agreed with human preferences over 80% of the time on MT-Bench, about the same as humans agree with each other — on general chat, not on your domain.
E-commerce catalog descriptions, pairwise judging of two prompt variants: run every pair in both orders and count only consistent verdicts, scoring flips as ties. Add 'length is not a quality signal; penalise padding' to the rubric, then verify on a 100-item slice where you pad the reference answer with filler — if the padded version wins more than 10% of the time, the bias is still there. Use a judge from a different model family than the system under test, or a 3-judge panel with majority vote for launch decisions.
A judge is a measurement instrument, so calibrate it per criterion before trusting it: 200–300 human-labelled items, compute judge–human κ, and automate only criteria where judge–human agreement is at least human–human agreement. Pin the judge snapshot, prompt version and decoding settings, store them on every score, and treat a judge change as a migration — re-score the baseline, never compare across judge versions. Keep a fixed 200-item 'judge canary' set and re-judge it weekly; a shift of more than 2 points with no input change is judge drift. Where it breaks: domain criteria the judge cannot assess (dosage correctness, tax treatment), and teams tuning prompts until they are optimising for judge quirks. A monthly human audit of ~100 judged items stays as ground truth, owned by the eval platform team, not by each feature team.
Pairwise run on 1,000 items with order swap = 2,000 judge calls × (2k in + 300 out) at $3/$15 per M ≈ 2,000 × $0.0105 ≈ $21 per run. A 3-judge panel triples it to ~$63. Calibration: 250 items × 2 human labellers × 3 min ≈ 25 h once, plus ~5 h/month for the audit.
What the answer sounds like at each level
Judges prefer longer answers and the first answer shown, so swap order and be careful with length.
Control position with order-swapping, verbosity with rubric language plus a padded-answer test, self-preference with a cross-family judge, and calibrate against 200+ human labels per criterion.
I run judges as versioned instruments with a calibration report, a weekly canary set for drift, a re-baseline procedure for judge upgrades, and a standing human audit — shared infrastructure for every team, not per-feature prompts.
Follow-ups the interviewer will push
1The judge vendor deprecates your judge model. What do you do?
Run old and new judges side by side on the calibration set and the last baseline; if per-criterion κ against humans holds, re-score the baseline with the new judge and switch atomically. Never mix scores from two judge versions in one trend line.
2The judge and humans agree only 60% on 'medically safe'. Automate anyway?
No. Keep that criterion human-scored on a sampled basis, use the judge only to triage likely failures for human review, and invest in a narrower criterion definition until agreement improves.
Red flags interviewers listen for
- Uses the same model family as both system under test and sole judge.
- Has never measured judge–human agreement on their own data.
- Upgrades the judge model and keeps comparing scores across the change.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Bias coverage | Names position, verbosity, self-preference and judge drift, each with a concrete control. |
| Calibration rigour | Per-criterion agreement against a human-labelled set with a numeric bar. |
| Operational discipline | Pins and stores judge version, has a canary set and a re-baseline procedure. |
Sources
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Zheng et al., 2023paper
- Large Language Models are not Fair Evaluators — Wang et al., 2023paper
- Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators — Dubois et al., 2024paper