How do you write a good rubric for grading open-ended LLM output?
- 2 min — what the output is for and who acts on it
- 4 min — draft 4–8 criteria, binary first, with anchors
- 4 min — validate with double labelling and κ, rewrite ambiguous criteria
- 2 min — versioning, conflict priority, revision cadence
A rubric turns 'is this good?' into 4–8 separate, observable criteria, each with a defined scale and anchored examples. Default to binary criteria ('every obligation in the source is mentioned': yes/no) over 1–10 holistic scores: binary items are faster for humans, more stable for an LLM judge, and aggregate into pass rates you can put a confidence interval on. Each criterion needs a definition, one passing and one failing example, and a stated priority when criteria conflict.
Commercial-lease summary feature: (1) every payment obligation in the source appears — binary, checked per obligation; (2) no claim lacks support in the source (grounding) — binary; (3) renewal and termination dates exactly match the source — binary, deterministic string check; (4) under 250 words — binary, code; (5) a lawyer could act on it without reopening the lease — 3-point scale with an anchored example of each. Criteria 3 and 4 never go to a judge: anything code can check, code checks.
Validate the rubric before trusting any number it produces: two annotators label the same 50 items independently and you compute inter-annotator agreement per criterion. Below Cohen's κ ≈ 0.6 the criterion is ambiguous, not the annotators — rewrite its definition and anchors, then re-test. Only criteria that clear the bar get automated with a judge, and the judge must match humans at least as well as humans match each other. State conflict priority explicitly ('completeness beats length'); otherwise annotators resolve it differently and agreement collapses. Version the rubric with the golden set; any criterion change rebaselines historical scores. Where it breaks: rubrics frozen after launch while error analysis keeps surfacing failure modes nobody is scoring.
Validation pass: 50 items × 2 annotators × 4 min ≈ 7 h of expert time per rubric revision. If 2 of 6 criteria fail κ ≥ 0.6, expect one more revision cycle (another ~7 h) before automation.
| Criterion | Scale | Checked by | Gate? |
|---|---|---|---|
| All payment obligations present | binary per obligation | LLM judge (κ 0.78 vs humans) | yes |
| Every claim supported by source | binary | LLM judge (κ 0.71) | yes |
| Dates match source exactly | binary | code | yes |
| Under 250 words | binary | code | no (report) |
| Actionable by a lawyer | 1–3 anchored | human, weekly sample | no (report) |
What the answer sounds like at each level
List the qualities you care about — correctness, tone, format — and score each one, ideally with examples.
Use 4–8 binary criteria with anchored examples, push anything deterministic to code, and check inter-annotator agreement before automating.
The rubric is the product spec in testable form: I gate automation on per-criterion κ, version it with the dataset, make product and domain owners sign off on conflict priorities, and revise it from error analysis on a fixed cadence.
Follow-ups the interviewer will push
1Two annotators get κ = 0.35 on 'helpfulness'. What now?
Split it. 'Helpful' is several things — answers the question asked, gives a next action, no irrelevant content. Write each as a binary item with a pass and fail example, re-label 50 items, and keep only the sub-criteria that clear ~0.6.
2Why not a single 1–10 score from a strong judge?
It hides which dimension moved, judges cluster scores in the 7–8 band, and a 0.3-point mean shift on a 10-point scale has no interpretable confidence interval. Binary pass rates per criterion are debuggable and testable.
Red flags interviewers listen for
- A single holistic 1–10 'quality' score as the only criterion.
- Sends checks that code could do (length, JSON validity, exact dates) to an LLM judge.
- Never measures agreement before automating the rubric.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Criterion design | Observable, independent, mostly binary criteria with pass/fail anchors. |
| Validation | Measures per-criterion agreement with a stated threshold before automating. |
| Division of labour | Routes deterministic checks to code and only judgment calls to humans or a judge. |
Sources
- A Coefficient of Agreement for Nominal Scales — Jacob Cohen, 1960paper
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Zheng et al., 2023paper
- Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences — Shankar et al., 2024paper