What is benchmark contamination, and how does it affect model selection?
- 2 min — define contamination and why public sets are exposed
- 3 min — evidence: fresh-set and post-cutoff benchmarks
- 4 min — worked model-selection example on private data
- 3 min — protecting and refreshing the private set; ownership
Contamination is when test items, or close paraphrases, are in a model's training data, so it scores well partly by recall rather than capability. Pretraining corpora are scraped from the public web, so any public benchmark is at risk once it has been online for a while. The evidence is consistent: when Zhang et al. (2024) built GSM1k, a fresh set mirroring GSM8K, several model families scored noticeably lower on it; LiveCodeBench counters the problem by only scoring problems published after each model's cutoff.
Choosing a model for a code-search and code-edit assistant: model A leads model B by 6 points on a popular public coding benchmark. On a private 400-task set drawn from our own repositories — tasks written after both models' cutoffs — the gap is 0.5 points with an interval spanning zero. Either A is contaminated or the public benchmark doesn't resemble our workload; in both cases the private set is the evidence, and we choose B on latency and price.
Public leaderboards are a shortlist filter, never a decision. Your private golden set is the contamination defence, and it can leak too: keep it out of public repos, out of fine-tuning uploads, out of few-shot examples, and send it only to endpoints with zero data retention or training opt-out. Add a canary string to eval files so leaks are searchable. Detect contamination with a paraphrase delta: score verbatim items and meaning-preserving rewrites; a model that drops 10+ points on rewrites has memorised. Refresh 10–20% of the set each quarter with items dated after the newest candidate's cutoff. Model selection should be owned centrally (one bake-off harness, one private suite per use case), so 15 teams don't each trust a different leaderboard.
Bake-off: 400 private tasks × 3 candidates × 3 repeats = 3,600 runs; agentic code tasks at ~20k in + 2k out tokens ≈ $0.09 each at $3/$15 per M → ~$325 per bake-off. Quarterly refresh of 60 tasks × 20 min engineer time ≈ 20 h.
What the answer sounds like at each level
Models may have seen benchmark questions in training, so their scores can be inflated.
Use leaderboards only to shortlist, decide on a private set of post-cutoff tasks, and detect memorisation with paraphrase deltas.
I run one central model-selection harness with private per-use-case suites, a leak-prevention policy (ZDR endpoints, canary strings, no fine-tune uploads), and a quarterly refresh tied to model cutoffs.
Follow-ups the interviewer will push
1How do you check whether your own private set has leaked?
Search model outputs and the public web for the canary string, and run the paraphrase-delta test on each new model release; a jump on verbatim items with no jump on rewrites is a leak signal.
2A vendor offers a 'free eval' if you upload your test set. Accept?
Only under a contract that excludes the data from training with deletion on completion — and then treat that set as burned for future vendor comparisons and build a fresh held-out slice.
Red flags interviewers listen for
- Selects a model primarily from a public leaderboard rank.
- Assumes a private eval set cannot leak.
- Has no plan to refresh eval items as model cutoffs move.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Understanding | Explains memorisation vs capability and cites evidence of score drops on fresh data. |
| Detection | Proposes paraphrase deltas, post-cutoff items and canary strings. |
| Governance | Has leak-prevention rules and central ownership of model selection. |
Sources
- A Careful Examination of Large Language Model Performance on Grade School Arithmetic — Zhang et al., 2024paper
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code — Jain et al., 2024paper