You have a new LLM feature and zero eval data. How do you build a golden set from scratch?
- 2 min — clarify the feature, the cost of a wrong answer, and who can label
- 5 min — seed set: slices by intent × difficulty × risk, binary criteria, sizes per slice
- 5 min — growth loop: production sampling, error analysis, promotion, held-out slice
- 3 min — governance: versioning, owners, refresh cadence, PII path, harness choice
A golden set is a fixed, versioned, deliberately stratified collection of inputs, each with a reference answer or binary rubric criteria, that the eval harness runs on every change. It is not a random sample of logs: it over-weights the hard and the expensive-to-get-wrong cases. Each row carries input, expected behaviour, slice tags (intent, difficulty, risk), provenance (hand-written, production, synthetic), date added and the set version. Size it per slice, not in total: below ~30 items a single flip moves a slice by 3+ points, so a slice you report on needs at least 30, and a slice you gate on needs 100+.
Clinical intake-note summariser, day one: I sit with two clinicians for half a day and write 60 cases — one per expected note type, plus empty notes, contradictory medication lists, and out-of-scope requests that must trigger abstention. Week two, with traffic: sample 200 production traces, do open-coded error analysis, build a failure taxonomy (dropped allergies, invented dosages, wrong section), and promote 5–10 examples of every new failure mode. By month two the set is ~400 items: 70% real traffic, 20% curated hard cases, 10% adversarial, with 20% held out that nobody tunes prompts against.
Ownership splits cleanly: the product team owns the content and the labels; the platform team owns the harness, storage and the CI wiring. Buy or adopt the harness (promptfoo, UK AISI Inspect, Braintrust, LangSmith, Langfuse as of Oct 2026); never outsource the data, because the data is the product spec. Expert time is the binding constraint, so spend it on criteria definitions and disagreements, not on easy items. Report per-slice scores with intervals — an aggregate that rises while the 'medication list' slice drops is a hidden regression. The set goes stale as the user mix shifts, so refresh monthly from production through a data flywheel, keep saturated items in a regression slice rather than deleting them, and redact PHI/PII before any annotator or vendor sees a production trace.
Assume 400 items, 6 min of clinician time per item, $150/h loaded rate → 40 h ≈ $6k to build; refresh 50 items/month ≈ 5 h. Run cost per item-run: generation 3k in + 400 out at $3/$15 per M ≈ $0.015, judge 2.5k in + 150 out ≈ $0.010 → $0.025. Full suite at 3 repeats = 1,200 item-runs ≈ $30 and ~6 min wall-clock at 32-way concurrency.
| Slice | Items | Source | Used for |
|---|---|---|---|
| Common note types | 180 | production, redacted | report |
| Medication / allergy edge cases | 100 | hand-written + production | gate |
| Out-of-scope → abstain | 40 | hand-written | gate (must-pass) |
| Adversarial / malformed | 40 | hand-written + synthetic | report |
| Held-out (all slices) | 80 | mixed | quarterly check only |
What the answer sounds like at each level
Write some examples with expected outputs and run them after every change, adding failures as they show up.
Seed 50–100 hand-written cases by intent and risk, grow from production via error analysis, track per-slice scores, and keep a held-out slice.
Treat the set as a versioned product spec with an owner, a slice budget sized for statistical power, a monthly refresh SLA from production, a PII path for annotators, and a harness shared across teams so every feature reports the same way.
Follow-ups the interviewer will push
1The PM wants to launch next week with 20 examples. What do you do?
Ship with the 20 as a must-pass smoke set covering the highest-risk behaviours, label the launch beta, log every trace with prompt and model version, and book two expert-days in week two for error analysis. 20 items cannot detect anything smaller than a ~15-point shift, so don't claim quality numbers from it.
2How do you know the set represents production?
Run an intent classifier over a month of traffic and compare the distribution to the set's slice tags. Every intent above 1% of traffic gets at least 30 items; any intent in production with zero items is a coverage bug filed against the set's owner.
3Now a regulated healthcare tenant joins. What changes?
Their traces only enter the set after PHI redaction, labelling happens under a BAA with access-controlled storage, items carry a tenant tag with a retention date, and that tenant gets its own slice so its quality is reported separately from the aggregate.
Red flags interviewers listen for
- Proposes labelling a random 1,000 production logs with an LLM and calling it the golden set.
- Reports one aggregate score with no slices and no confidence interval.
- Tunes prompts against every item, with no held-out slice and no versioning.
- Treats the set as a one-off project with no owner or refresh cadence.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Stratification | Designs slices by intent and risk and sizes them for the decisions they gate (≥30 to report, 100+ to gate). |
| Labelling economics | Quantifies expert hours and spends them on criteria and disagreements, not on bulk easy items. |
| Lifecycle | Has a production-fed refresh loop, a held-out slice, and versioned sets so scores are comparable. |
| Org scope | Separates content ownership (product) from harness ownership (platform) and adopts rather than builds tooling. |
Sources
- Your AI Product Needs Evals — Hamel Husain, 2024blog
- Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences — Shankar et al., 2024paper