Human annotation
People labelling model outputs against a rubric; the ground truth that LLM judges are calibrated against, and the most expensive data in the eval stack.
Human annotation means people read model outputs and label them: pass or fail, which answer is better, which sentence is unsupported. These labels are the ground truth for your Golden set and the yardstick for checking whether an LLM-as-judge can be trusted.
The labellers might be engineers, domain experts (lawyers, nurses) or a vendor's workforce, depending on how much expertise the judgement needs.
The referees' panel whose decisions are used to train and audit the video-review system; the system is faster, but you only trust it because it matches the panel.
Outsourcing labels to generalists for an expert task to save money. A crowd labeller cannot tell a correct drug interaction from a plausible one; you get confident, cheap, wrong ground truth, and the judge you calibrate against it inherits the error.
Three decisions dominate quality. Who labels: domain experts for anything where a wrong label is expensive (clinical, legal, financial), trained generalists for style and format. What they see: the input, the retrieved context and the output, plus a Rubric with anchored examples, and a free-text 'why' field that feeds Error analysis. How you check them: double-label 10–20% of items and track Inter-annotator agreement; adjudicate disagreements with a third person and turn recurring disagreements into rubric edits.
Shankar et al. (2024) found criteria drift: people refine what they mean by 'good' while grading, so the first 50 labels are partly about discovering the rubric. Plan a calibration round, then freeze. Tooling: Argilla, Label Studio, or a spreadsheet for the first 200. The most valuable use of human time is not bulk labelling but labelling the 100–300 items that validate a judge, which then labels the rest.
Engineers label the first 100 traces themselves (fastest way to learn the failure modes). Then a calibration round of 50–60 items with two labellers, 20% overlap thereafter, a free-text 'why' on every fail, Argilla or Label Studio once past ~300 items, and domain experts for clinical, legal and financial content.
Clinical intake summaries. A nurse labels at ~25 items/hour (2.4 min each) against 5 binary criteria. Plan: 60-item calibration round with two nurses (kappa 0.48 → rubric edits → 0.71), then 240 items single-labelled with 20% double-labelled. 300 items ≈ 12 nurse-hours + 3 for overlap ≈ 15 hours. Those labels validate a judge (kappa 0.68 vs nurses), which then grades 3,000 production samples a month.
Say: "Humans label the few hundred items that validate the judge; the judge labels the thousands. I double-label 20% and adjudicate, and experts label anything where a wrong label is expensive." Expect: "how do you keep labels consistent?": calibration round, anchored rubric, overlap, kappa.
Numbers worth knowing
- Labelling throughput
- 20–60 items/hourbinary criteria; ~25/h for expert review of long clinical or legal outputs, 60/h for short answers
- Overlap for agreement tracking
- 10–20% of itemsdouble-labelled by a second person
- Labels needed to validate a judge
- 100–300 per criterionkappa estimate within about ±0.1
How to decide
| When | Do this |
|---|---|
| Wrong labels are costly (clinical, legal, financial) | Use domain experts and budget for them; use generalists only for format and style criteria. |
| Two labellers' kappa is below 0.6 after calibration | Stop labelling; rewrite the rubric, then re-calibrate. |
| You need more than ~1,000 labels a month | Validate an LLM judge on 300 human labels and use humans for audits and disagreements only. |
Cost maths
Cost = items × (1 + overlap) / throughput × hourly rate. Assumptions: domain expert $100/h, 25 items/h; generalist $25/h, 60 items/h. 300 expert items × 1.2 / 25 × $100 = $1,440. The same work with generalists ≈ $150, but only valid for criteria a non-expert can judge. Monthly judge validation (100 items, expert) ≈ $480.
Operating it
- Label backlog exceeds two weeks: move bulk grading to a validated judge.
- More than 3 labellers: add an adjudication role and a weekly calibration session.
- Pairwise kappa on overlapping items, weekly: alert below 0.6.
- Per-labeller pass-rate deviation from the group: flag anyone more than 15 points off.
- Label backlog age: alert when the oldest unlabelled flagged trace exceeds 14 days.
When it goes wrong
A financial-filings Q&A judge reported 94% accuracy; analysts found wrong fiscal-year figures in about one answer in ten.
An analyst spot check of 50 answers found 6 errors the judge had passed.
The judge was validated against labels from a generalist vendor who checked that a number was present and cited, not whether it matched the filing's fiscal-year convention.
Re-labelled the 300-item validation set with two analysts (kappa 0.74), added a 'figure matches source period' criterion, and re-validated the judge, whose true pass rate dropped to 86%.
Where it shows up
Sources
- Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences — Shankar et al., 2024paper
- Your AI Product Needs Evals — Hamel Husain, 2024blog
- The Measurement of Observer Agreement for Categorical Data — Landis & Koch, 1977paper
Check yourself
Three questions. Every option is explained once you answer.