Benchmark contamination
When test items (or close paraphrases) leaked into a model's training data, so its score measures memory instead of skill.
Contamination happens when the questions in a test set ended up in the data a model was trained on. The model may then recall the answers instead of working them out, so its score looks better than its real ability.
It affects public benchmarks, which are published on the web that models train on, and it can affect your own eval set if it leaks into a fine-tuning dataset or a vendor's training data.
A student who saw the exam paper the night before. The grade is real; what it measures is not.
Assuming contamination only matters for public benchmarks. The most common case in industry is self-inflicted: eval cases leaking into few-shot prompts, fine-tuning data or synthetic-data seeds.
Public benchmark items get scraped into Pretraining corpora, often with answers from forum discussions. Detection: n-gram overlap between test items and training data (the GPT-3 paper used 13-gram overlap), completion probes (give the first half of an item and see if the model reproduces the rest verbatim), membership-inference scores like Min-K% Prob (Shi et al. 2023), and the strongest test, a fresh parallel set. Zhang et al. (2024) wrote GSM1k, 1,250 new grade-school maths problems matched to GSM8K, and found accuracy drops of up to 8 points for some model families, with the drop correlated with how likely the model was to generate GSM8K items.
For your own evals the risks are internal: golden-set cases reused as few-shot examples, cases copied into a Fine-tuning set, or a Synthetic data generator seeded from the test set. Defences: split by time (test only on items created after the model's Knowledge cutoff, as LiveCodeBench does), keep a private held-out set never sent to a training pipeline, embed a canary string in eval files so leaks can be grepped, and dedupe fine-tuning data against eval data with MinHash at Jaccard ≥0.8.
A private held-out slice (20–30% of the golden set) that never goes into prompts, fine-tuning or synthetic-data seeds; a canary GUID in every eval file; MinHash dedupe (datasketch, Jaccard ≥0.8) of any training or few-shot data against eval data; time-split cases created after the candidate model's cutoff.
A team fine-tunes a support model on 40k historical tickets and reports a jump from 74% to 89% on its 300-case golden set. A MinHash check finds 61 golden cases with near-duplicates in the fine-tuning data (the golden set had been sampled from the same ticket export). On the 239 clean cases the fine-tuned model scores 79%, still a gain, but half the reported one.
Say: "Before I believe a jump, I check the test set against everything the model trained or prompted on, and I keep a private, time-split held-out set that never touches a training pipeline." Expect: "how would you detect it on a closed model?": fresh parallel items and completion probes, since you cannot see the training data.
Numbers worth knowing
- Score drop on a fresh parallel maths set
- up to 8 pointsGSM8K → GSM1k, some model families, Zhang et al. 2024
- Near-duplicate threshold
- MinHash Jaccard ≥0.8 on 5-gram shinglescommon dedupe setting for text; tune on a labelled sample
- Held-out share of a golden set
- 20–30%never used for prompt iteration or training
How to decide
| When | Do this |
|---|---|
| A fine-tune or prompt change produces a surprisingly large jump (>10 points) | Dedupe training and few-shot data against the eval before celebrating. |
| Comparing closed models on a public benchmark older than their cutoffs | Treat the scores as upper bounds and confirm on a fresh or private set. |
| Eval cases are used as few-shot examples | Remove them from the scored set or score them separately. |
Cost maths
Dedupe is cheap: MinHash over 1M training documents vs 5k eval items runs in minutes on one CPU. A fresh parallel set is the expensive control: 300 new expert-written items at ~4 minutes and $100/h ≈ $2,000, which is why you keep a private held-out slice from day one instead.
Operating it
- You start fine-tuning or distilling: add automated dedupe of training data against all eval sets in the pipeline.
- Your eval set has been used for prompt iteration for 6+ months: refresh 30% from recent traffic and retire the most-tuned cases.
- Overlap rate between any new training or few-shot data and eval sets: alert on any exact match, review above 1% near-duplicates.
- Gap between held-out and iterated slices of the golden set: alert when it exceeds 5 points (the iterated slice is overfit).
When it goes wrong
A legal-contracts extraction model scored 93% offline after fine-tuning but 81% on the first month of new customer contracts.
Customer-reported extraction errors, then a re-score on contracts signed after the training cut: 80%.
The golden set and the fine-tuning set were both sampled from the same 2023–2024 contract archive; 70 golden contracts had near-duplicate templates in training.
Time-split the eval (only contracts dated after the training cut), added MinHash dedupe to the training pipeline, and reported the clean-set score as the headline.
Where it shows up
Sources
- A Careful Examination of Large Language Model Performance on Grade School Arithmetic — Zhang et al., 2024paper
- Time Travel in LLMs: Tracing Data Contamination in Large Language Models — Golchin & Surdeanu, 2023paper
- Detecting Pretraining Data from Large Language Models — Shi et al., 2023paper
Check yourself
Three questions. Every option is explained once you answer.