Inter-annotator agreement
How often independent graders agree beyond chance, usually Cohen's kappa; below ~0.6 your labels, and any judge validated against them, are unreliable.
Inter-annotator agreement measures whether two graders, given the same items and Rubric, give the same labels. Raw percent agreement is misleading because graders agree a lot by luck when most answers pass, so we use statistics like Cohen's kappa that subtract the agreement you would expect by chance.
Kappa is 1 for perfect agreement and 0 for agreement no better than chance. The same maths checks whether an LLM-as-judge agrees with humans.
Two doctors reading the same X-rays. If 90% of X-rays are normal, two doctors who say 'normal' to everything agree 82% of the time while catching nothing; kappa is the score that would expose them.
Quoting '88% agreement' as evidence a judge is reliable. On imbalanced data that can be κ ≈ 0.4, with the judge missing half the failures, which are the only items you care about.
Cohen's kappa κ = (pₒ − pₑ) / (1 − pₑ), where pₒ is observed agreement and pₑ is chance agreement from each grader's label rates. Landis & Koch (1977) bands: 0.21–0.40 fair, 0.41–0.60 moderate, 0.61–0.80 substantial, 0.81–1.00 almost perfect. Practical thresholds: ≥0.6 before a rubric is used for a Golden set or to validate a judge, ≥0.7 if the judge gates releases.
Kappa collapses under class imbalance: at a 95% pass rate a grader can agree 94% of the time with κ near 0.4, so also report per-class agreement (agreement on fails specifically) or judge true-positive and true-negative rates. Use Fleiss' kappa or Krippendorff's alpha for more than two graders or missing labels, and weighted kappa for ordinal scales. Estimate on ≥100 items; with 50 items the 95% CI on kappa is about ±0.2. Low agreement is a rubric bug first, a grader bug second.
Cohen's kappa for two graders (sklearn.metrics.cohen_kappa_score), Krippendorff's alpha for 3+ or missing labels, reported with a bootstrap 95% CI on ≥100 items, alongside judge recall on the fail class. Thresholds: 0.6 to use, 0.7 to gate.
Code-review assistant comments labelled 'actionable / not actionable' by two senior engineers on 200 comments. They agree on 170 (pₒ = 0.85). Engineer 1 marks 80% actionable, engineer 2 marks 80%: pₑ = 0.8×0.8 + 0.2×0.2 = 0.68. κ = (0.85 − 0.68)/(1 − 0.68) = 0.53, moderate, despite 85% raw agreement. Reading the 30 disagreements shows 'nit' comments split them; adding 'style nits are not actionable' to the rubric lifts κ to 0.74.
Say: "I don't report percent agreement; I report kappa, plus agreement on the failure class, because with a 90% pass rate percent agreement is mostly luck." Expect: "what kappa is good enough?": 0.6 to use the labels, 0.7 to gate releases on a judge, and low kappa means rewrite the rubric.
| Kappa | Landis & Koch | Use it to |
|---|---|---|
| < 0.40 | slight–fair | rewrite the rubric |
| 0.41–0.60 | moderate | calibrate, re-label |
| 0.61–0.80 | substantial | build golden sets |
| > 0.80 | almost perfect | gate releases |
Numbers worth knowing
- Minimum kappa to trust labels
- ≥0.6 (substantial from 0.61)Landis & Koch bands; ≥0.7 if the judge gates releases
- 95% CI width on kappa
- ≈ ±0.2 at n=50; ±0.1 at n=200binary labels, moderate prevalence; bootstrap to get yours
- Raw agreement vs kappa trap
- 85% agreement → κ = 0.53both graders pass 80% of items (pₑ = 0.68)
How to decide
| When | Do this |
|---|---|
| Pass rate is above 90% | Report agreement on fails (judge recall on the fail class) next to kappa; kappa alone hides missed failures. |
| Three or more graders, or some items labelled by only one | Use Krippendorff's alpha instead of averaging pairwise kappas. |
| Kappa is below 0.6 | Read every disagreement, edit the rubric, re-label a fresh sample; do not tune the judge to match one grader. |
Operating it
- More than 2 labellers: switch to Krippendorff's alpha and track per-labeller drift.
- Judge is used as a release gate: require kappa ≥0.7 and fail-class recall ≥0.85 on a held-out set.
- Judge–human kappa on a monthly fresh sample: alert below 0.6.
- Judge recall on human-labelled failures: alert below 0.85.
- Per-labeller agreement with adjudicated labels: alert on any labeller below 0.6.
When it goes wrong
A contract-review judge flagged almost no risky clauses for a quarter, and the team read that as quality improving.
A lawyer re-reviewed 120 contracts; the judge had missed 11 of 19 risky clauses.
The judge was validated with 92% raw agreement on a set where 90% of clauses were benign; kappa was 0.41 and fail-class recall 0.42, never computed.
Validated on a set enriched to 30% risky clauses, required fail-class recall ≥0.85 and kappa ≥0.7, and rewrote the judge prompt with risky-clause anchors.
Where it shows up
Sources
- The Measurement of Observer Agreement for Categorical Data — Landis & Koch, 1977paper
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Zheng et al., 2023paper
- Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences — Shankar et al., 2024paper
Check yourself
Three questions. Every option is explained once you answer.