How do you run human annotation for evals, and how do you know the labels are trustworthy?
- 2 min — what the labels are for and who is qualified
- 4 min — pilot: guidelines, double labelling, κ/α, adjudication
- 4 min — scale: QC with seeded items, blinding, throughput
- 3 min — privacy, vendors vs in-house, ownership
Human annotation produces the ground truth every automated metric is calibrated against. Trust comes from four things: written guidelines (a rubric with pass/fail anchors), the right annotators (domain experts where correctness needs expertise), measured inter-annotator agreement — Cohen's κ for two annotators, Krippendorff's α for more or with missing labels, both correcting for chance agreement — and ongoing quality control with known-answer items seeded into every batch.
Clinical intake summaries: two clinicians independently label 100 summaries for safety and correctness, blind to which system produced each. An adjudicator resolves the 14 disagreements, and each one is reviewed for guideline ambiguity — 9 trace to one unclear rule about 'as needed' medications, which gets rewritten. Batch two reaches κ = 0.74 on safety. Only then does labelling scale to four annotators, with 8% of every batch being known-answer items; an annotator who drops below 90% on those is paused and re-briefed.
Expert time is the scarcest resource in the eval stack, so route it: judge calibration, new failure modes, high-stakes slices, and adjudication — never bulk easy items a calibrated judge already handles. Build vs buy: vendors (Scale, Surge, Labelbox as of Oct 2026) work for general-language tasks; regulated domains usually need in-house or contracted specialists under a data-processing agreement. Low agreement is a signal, not noise to average away — it usually means the task is underspecified. Annotators must not see which system produced an output. Production traces must pass through PII redaction before reaching any annotator, with a BAA for US PHI and a GDPR DPA for EU data. The eval platform team owns guidelines, QC and tooling; domain teams own the experts.
Pilot: 100 items × 2 clinicians × 5 min ≈ 17 h; at an assumed $200/h ≈ $3.3k. Steady state: 300 items/month single-labelled + 15% double-labelled for κ tracking ≈ 29 h ≈ $5.8k/month.
What the answer sounds like at each level
Have people label outputs using clear instructions, and check that different labellers agree.
Double-label a pilot, compute κ or α per criterion, adjudicate and fix guidelines, then scale with blinded annotators and seeded known-answer items.
I run annotation as a pipeline with an owner, QC thresholds, a redaction and DPA path per region, and a budget that routes expert hours to calibration and new failure modes rather than volume.
Follow-ups the interviewer will push
1Agreement is high but the judge trained on these labels still fails in production. Why?
The labelled sample likely doesn't cover the failing slice, or the annotators agreed on a shared misunderstanding. Pull production failures, have a fresh expert label them blind, and compare against the original guidelines.
2Now you need labels in German and Arabic for EU and UAE tenants.
Native-speaker experts per language, guidelines translated and back-checked, κ measured per language separately, and data kept in-region with a DPA — never pooled agreement numbers across languages.
Red flags interviewers listen for
- Measures agreement as raw percent with no chance correction.
- Lets annotators see which model produced each output.
- Sends unredacted production traces to an external labelling vendor.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Agreement measurement | Uses chance-corrected agreement per criterion with a threshold and acts on low scores. |
| Quality control | Seeds known-answer items, blinds annotators, tracks per-annotator accuracy. |
| Privacy and sourcing | Redacts before labelling, names DPA/BAA needs, chooses in-house vs vendor by domain risk. |
Sources
- A Coefficient of Agreement for Nominal Scales — Jacob Cohen, 1960paper
- Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences — Shankar et al., 2024paper