Evaluation & Metrics

Golden set

Budding levelReviewed Oct 2026In 37 interview questionsIn 18 posts

A fixed, versioned set of realistic inputs with known-good answers or relevant documents, used to score every change the same way.

In plain English

A golden set is your answer key. It is a list of real-looking questions, each paired with what a correct result looks like: the documents that should be found, the facts the answer must contain, or a rubric. Every time you change a prompt, a model or the retrieval, you rerun the same set and compare scores, so you know whether you improved things or broke them.

Analogy

A chef's tasting menu that never changes, cooked after every kitchen change, so you can taste whether the new oven made things better or worse.

Common pitfall

Writing the golden set yourself from the documents. Author-written questions reuse the documents' wording, so lexical retrieval looks great and real user phrasing fails.

How it actually works

Build from real traffic first (sampled logs, support tickets, search queries), then add synthetic items only to fill gaps you can name. Each item carries: input, expected relevant chunk ids (for retrieval metrics like Recall@k and nDCG), expected answer facts or a Rubric, and tags (intent, difficulty, language, answerable/unanswerable, facts-needed). Size it from the confidence interval you need: at accuracy p ≈ 0.8, the 95% CI half-width is 1.96·√(p(1−p)/n): ±5.5 points at n = 200, ±3.5 at n = 500, ±2.5 at n = 1,000. Version it in git, freeze a held-out split nobody tunes on, and refresh quarterly from new failure clusters found in Error analysis. Leakage is the silent killer: if items are also in few-shot examples or fine-tuning data, scores inflate (Benchmark contamination).

What I'd ship on day one

200–500 items sampled from production logs, stratified by intent, 10–15% unanswerable; two labellers on 20%+ of items with κ reported; JSONL in git with a schema and a version tag; 70/30 dev/holdout split; quarterly refresh from error-analysis clusters.

Worked example

Code-search assistant for a 3M-line monorepo. v1 golden set: 320 items from six weeks of search logs, stratified across 'where is X defined' (90), 'how do I call Y' (110), 'why does Z fail' (70), cross-repo (30) and unanswerable (20). Each lists 1–4 relevant file paths plus required answer facts. Two engineers labelled independently; disagreements on 11% were adjudicated. A dev split of 220 is used for tuning; 100 are held out and only run before releases.

In the interview

Say: I start from 200–300 items sampled from real logs, stratified by intent, with 10–15% unanswerable, labelled by two people, and I size it by the CI I need to detect the change I care about. Expect: 'how do you keep it from going stale?' Quarterly refresh from production failure clusters, plus a frozen holdout to catch overfitting to the dev split.

Query (from logs)	Relevant files	Must mention	Tags
where is retry backoff set	net/retry.go	maxDelay 30s	where-defined
call billing svc from py	sdk/billing.py	idempotency key	how-to-call
why does make test hang	ci/README.md	docker socket	why-fails
deploy to mars region	—	abstain	unanswerable

Each item scores retrieval (files) and generation (facts) separately

Halving the error bar needs 4× the items

Numbers worth knowing

95% CI half-width: ±5.5 / ±3.5 / ±2.5 pointsn = 200 / 500 / 1,000 at p ≈ 0.8
Labelling time: 3–6 min per itemretrieval ids + answer facts, domain-literate labeller
Refresh cadence: quarterly, 10–20% new itemsproducts with changing corpus or users

How to decide

When	Do this
No production traffic yet	Write 100 items with domain experts, mark them provisional, replace with log samples within a month
Need to detect a 2-point change	n ≈ 1,500+ or a paired test on the same items
Multiple tenants or languages	Stratify and report per slice; a global mean hides regressions

Cost maths

Labelling cost = n × minutes per item × hourly rate. 400 items × 5 min × $60/h = $2,000, plus 20% double-labelling = $2,400. Quarterly refresh of 15% ≈ $360. Compare with one bad release that a 3.5-point CI would have caught.

Operating it

Time to scale up when…

Release decisions hinge on differences inside the CI: grow n
A new intent or tenant > 10% of traffic is not represented
Dev-split score rising while holdout is flat: overfitting, refresh dev

What to monitor

Dev vs holdout score gap; alert > 3 points
Age of newest item; alert if > 120 days
Share of production intents with < 20 golden items; review monthly

When it goes wrong

What users saw

Golden-set accuracy rose from 81% to 89% over three months; user thumbs-down rate did not move

How it was caught

Quarterly holdout run: 80%, unchanged from the start

Root cause

Prompt tweaks and few-shot examples had been tuned on the dev split for 40 iterations; several golden items had been copied into few-shot examples

Fix

Removed leaked items, rotated a fresh dev split from recent logs, and added a CI check that fails if any golden input appears in prompts or training data

Where it shows up

BLOGBreaking into AI engineering: a roadmap from zero BLOGPrompt, retrieve or fine-tune? A decision guide BLOGWhat AI-engineering interviewers test at each level BLOGYour first RAG pipeline in pure Python BLOGHybrid search and reranking: a field guide BLOGTalking a team out of RAG (when it is the wrong tool)BLOGWorkflows before agents: pick the simplest pattern that works BLOGHow do you know if an AI is any good?BLOGYour first eval set in an afternoon BLOGLLM-as-judge without fooling yourself BLOGThe AI engineering ladder: what each level's week looks like BLOGStructured outputs that don't break BLOGAnatomy of a production RAG system BLOGChoose your chunk size by measurement, not folklore BLOGError analysis, step by step: from 100 traces to a failure taxonomy BLOGReading public AI incidents as postmortems BLOGSelf-host or API? A break-even memo with the maths shown BLOGDecision memo: one LLM platform for twelve teams I·7When do you use zero-shot prompting, few-shot examples, or chain-of-thought?I·8When is a reasoning model worth paying for?I·2How do you decide between prompting, RAG and fine-tuning?I·8How do you build a cost model for an LLM feature before launch?I·10How do you upgrade to a new model version without regressions?II·1How would you choose the optimal chunk size?II·8How would you detect that your embedding model is the bottleneck?II·9How would you migrate to a new embedding model in production?II·3When should you use a reranker after vector search?II·1How would you design a multi-tenant RAG system?II·6How would you monitor a RAG system after deployment?II·7What would you do if RAG quality suddenly drops after a production deployment?III·1What actually makes a system an agent, rather than an LLM call with extra steps?III·2When should you NOT build an agent — and use a workflow instead?III·4What makes a good tool for an agent?III·5Describe the agent loop. How does it know when to stop?III·6How do planning and reflection improve an agent, and when are they wasted effort?III·9When does a multi-agent or orchestrator-worker design help, and when does it hurt?III·1How do you evaluate an agent? Outcome vs trajectory evaluation.III·9What does observability look like for agents?III·10How would you roll out a new agent to production safely?IV·1You have a new LLM feature and zero eval data. How do you build a golden set from scratch?IV·2What is the difference between offline and online evaluation, and why do you need both?IV·3How do you write a good rubric for grading open-ended LLM output?IV·8When is synthetic data useful for evals, and what are its risks?IV·9What is benchmark contamination, and how does it affect model selection?IV·10What does eval-driven development look like day to day?IV·3How do you version and roll out prompt changes safely?IV·7A provider announces deprecation of the model you use. What is your upgrade playbook?IV·10How do you monitor quality drift for an LLM feature in production?V·2Design an LLM evaluation platform used by 30 product teams.V·8How would you roll out a new model version to production using shadow traffic and canaries?V·3Estimate $/1M tokens for self-hosting a 70B open-weights model versus calling an API, and the monthly volume at which self-hosting breaks even.V·6How many eval examples do you need to detect a 3-point improvement in pass rate with confidence?V·1Tell me about a time you disagreed with a senior leader about shipping an AI feature.V·7Tell me about trading quality for a deadline.V·9Tell me about building consensus on an eval standard across teams.

Sources

Your AI Product Needs Evals — Husain, 2024blog
BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models — Thakur et al., 2021paper
Introduction to Information Retrieval (ch. 8, Evaluation) — Manning, Raghavan & Schütze, 2008book

Check yourself

Three questions. Every option is explained once you answer.

1ConceptWhy should golden-set questions come from production logs rather than being written from the documents?

2ScenarioYou must tell whether a new prompt is 2 points better than the old one. Your golden set has 400 items. What do you do?

3CalculationAccuracy p ≈ 0.8 on a 500-item golden set. What is the 95% CI half-width?

Was this helpful?

Golden set

Numbers worth knowing

#How to decide

#Cost maths

#Operating it

#When it goes wrong

Related terms