Rubric
A written scoring guide: one observable criterion per line, a small scale (usually pass/fail), and an anchored example for each score.
A rubric is a scoring guide. Instead of asking "is this answer good?", it breaks quality into named criteria (for example: states the right deadline, cites the source clause, makes no promise the document does not make) and spells out exactly what earns each score.
Human reviewers (Human annotation) and AI graders (LLM-as-judge) both use rubrics. A good one makes two different graders give the same score to the same answer.
A teacher's marking scheme: 2 points for a clear thesis, 2 for evidence, 1 for spelling. Two teachers using it on the same essay should land on the same mark.
A 1–10 scale with no anchors. Scores bunch at 7–8, a 0.3-point 'improvement' is noise, and when the judge and a human disagree there is no criterion to point at.
Write analytic rubrics (one criterion per dimension), not holistic ones. Use binary pass/fail per criterion: 1–10 scales have poor Inter-annotator agreement, cluster at 7–8, and drift between sessions, while binary criteria are easier to label and to validate. Every criterion must be observable in the output ('names the termination notice period and quotes the clause number'), never a feeling ('is helpful').
Anchor each criterion with one passing and one failing example taken from real traces. For an LLM-as-judge, grade one criterion per call or per JSON field, ask for a 1–2 sentence critique before the verdict, and keep the critique in the logs because that is what you read when the judge and a human disagree. Validate by having two humans label 100 outputs: kappa below 0.6 means the wording is ambiguous, and the fix is the rubric, not the grader. Rubrics evolve; version them like prompts and re-validate after each edit.
3–6 binary criteria per task, each with one real passing and one real failing anchor; stored in git next to the judge prompt with a version id; judge returns JSON {criterion: {critique, pass}}; re-validated on 100 human-labelled cases after every edit.
Legal contract Q&A, three binary criteria: (1) states the correct notice period, (2) cites the clause number, (3) adds no obligation absent from the contract. Answer: '60 days, per clause 14.2; you must also notify by registered post.' Registered post is not in the contract: passes 1 and 2, fails 3. On the old 1–10 'quality' scale, three reviewers gave the same answer 5, 7 and 8.
Say: "I use binary, observable criteria, anchor each with a real pass and fail, and I don't let a judge use a rubric until two humans hit kappa 0.6 on it." The usual push-back is "isn't binary too coarse?": answer that you add criteria rather than scale points, because each criterion is then separately measurable.
| Criterion | Pass (1) | Fail (0) |
|---|---|---|
| Notice period | Exact value from contract | Wrong or missing |
| Citation | Clause number quoted | No clause or wrong one |
| No invented duty | Only contract terms | Adds any obligation |
Numbers worth knowing
- Criteria per rubric
- 3–6beyond ~6, split into separate judges or the judge starts conflating them
- Agreement target before use
- Cohen's kappa ≥0.6 between two humansLandis & Koch 'substantial' starts at 0.61
- Validation sample
- 100 outputs per criteriongives a kappa estimate within roughly ±0.1–0.15
How to decide
| When | Do this |
|---|---|
| Graders disagree on more than 20% of items | Read the disagreements, rewrite the ambiguous criterion, re-label; do not average the scores. |
| You want finer resolution than pass/fail | Add another binary criterion, or switch to Pairwise comparison, rather than adding scale points. |
| A criterion passes on 98%+ of outputs for a month | Drop it from the per-PR judge and keep it in the nightly run; it no longer discriminates. |
Cost maths
Judge cost scales with criteria × calls. One call grading 5 criteria as JSON: ~2k input + 300 output tokens ≈ $0.0035 on a Haiku-class judge ($1 / $5 per M, Oct 2026 list). Five separate calls cost ~4× more (repeated context) but agree with humans better on hard criteria; batch them through the Batch API at 50% off for nightly runs.
Operating it
- More than one team writes rubrics for the same product: maintain a shared criterion library with ids.
- Kappa on any criterion drops below 0.6 after a model change: re-anchor that criterion with fresh examples.
- Per-criterion pass rate trend: alert on a drop larger than the CI half-width.
- Judge–human kappa per criterion on a monthly re-labelled sample: alert below 0.6.
- Share of judge verdicts with an empty or boilerplate critique: alert above 5% (the judge stopped reasoning).
When it goes wrong
A clinical intake summariser scored 8.1/10 on the judge for three months while nurses kept correcting medication lists.
A nurse-led audit of 80 summaries found 14 with an omitted or wrong medication; all 14 had judge scores of 7 or higher.
The rubric was one holistic 1–10 'accuracy and completeness' score; fluent summaries with one missing drug still scored high.
Replaced it with five binary criteria including 'every medication in the source appears with dose', anchored with real failures; the judge pass rate on that criterion fell to 82% and became the metric the team actually improved.
Where it shows up
Sources
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Zheng et al., 2023paper
- Creating a LLM-as-a-Judge That Drives Business Results — Hamel Husain, 2024blog
- Prometheus: Inducing Fine-grained Evaluation Capability in Language Models — Kim et al., 2023paper
Check yourself
Three questions. Every option is explained once you answer.