Benchmark
A public, fixed test set and scoring rule (MMLU, GPQA, SWE-bench) for comparing models; a coarse filter for model choice, never a substitute for your own eval.
A benchmark is a public test that many models take so their scores can be compared: multiple-choice exams like MMLU, graduate-level science questions like GPQA, or real GitHub issues to fix like SWE-bench. Model launch posts lead with these numbers.
They are useful for a first shortlist, but they measure someone else's task. Your product needs its own test set (a Golden set) built from your own users' questions.
A standardised exam score on a job application: it filters the pile, but you would never hire without an interview for the actual role.
Choosing a model on a leaderboard delta smaller than the benchmark's own CI, or on a benchmark that tests a different task. A 1.5-point MMLU lead says nothing about whether a model follows your JSON schema.
A benchmark fixes three things: the items, the prompt format and the scorer. Change any one and scores move several points, which is why lab-reported numbers and independent re-runs (HELM, lm-evaluation-harness, Inspect evals) often differ. Three failure modes limit their use. Saturation: once frontier models score 90%+ (MMLU did by 2024), remaining differences are label noise. Benchmark contamination: test items leak into Training data, inflating scores. Construct gap: multiple-choice knowledge says little about multi-turn tool use or grounded answering over your Corpus.
Small benchmarks also have wide error bars: GPQA Diamond has 198 questions, so a model at 70% carries a ±6.4-point 95% CI, and a 3-point gap between two models is not a ranking. For retrieval components, task-matched public leaderboards such as MTEB leaderboard narrow the shortlist; for agents, SWE-bench Verified (500 human-validated tasks) and similar suites test multi-step behaviour. Use benchmarks to pick 2–3 candidates, then decide on your own eval.
Shortlist with task-matched public results (independent re-runs where available), then decide on your own 200–500 case eval run through the same harness for every candidate. For agents, look at SWE-bench Verified and terminal or tool-use suites; for retrieval, MTEB retrieval slices in your language.
Choosing a model for a developer-docs assistant. Public numbers shortlist three models within 4 points of each other on a coding benchmark. On the team's own 250-case eval (API questions over their docs, graded for correct code and cited page), the ranking flips: the model ranked third publicly passes 81% vs 72% for the leader, because it follows the 'cite the doc page' instruction far more reliably. Same price tier, so the decision takes a day, not a debate.
Say: "Benchmarks pick my shortlist; my own eval picks the model. A 2-point gap on a 200-question benchmark is inside the error bars." Expect: "which benchmarks do you trust?": the ones closest to your task, re-run independently, recent enough to be uncontaminated.
Numbers worth knowing
- GPQA Diamond size
- 198 questions → ±6.4 pts at 70%95% normal-approximation CI
- SWE-bench Verified size
- 500 tasks → ±4.0 pts at 70%human-validated subset released by OpenAI, Aug 2024
- MMLU
- 14,042 test questions, 57 subjectssaturated at the frontier (90%+) since 2024
How to decide
| When | Do this |
|---|---|
| Two models differ by less than the benchmark's CI | Treat them as tied on that benchmark; decide on your own eval, price and latency. |
| The frontier scores above 90% on a benchmark | Ignore it for ranking; it is saturated and dominated by label noise. |
| The benchmark predates both models' knowledge cutoffs by years | Assume some Benchmark contamination and prefer newer or private suites. |
Cost maths
Your own eval beats any leaderboard and is cheap: 3 candidate models × 300 cases × 3 samples × (3k in + 500 out) at an average $3 / $15 per M ≈ 3 × $14.9 ≈ $45 at list, ~$22 via Batch API. One engineer-day to set up. A wrong model choice costs that every hour in production.
Operating it
- You re-select models more than twice a year: keep a standing model-comparison eval in the harness.
- Public benchmarks for your task are saturated: build or license a private held-out set.
- Your own eval score per candidate model, re-run when a vendor ships a new snapshot.
- Gap between vendor-reported and your re-run benchmark score: investigate gaps above 5 points.
When it goes wrong
A team switched its e-commerce catalog enrichment to a model that led a public leaderboard by 3 points; attribute-extraction errors doubled.
Merchandisers flagged wrong sizes and materials within a week; the internal eval had never been run on the new model.
The leaderboard measured general knowledge and chat preference; the task was structured extraction from messy supplier text, where the new model ignored the field schema more often.
Rolled back, ran both models on a 300-item extraction eval (old 91%, new 84%), and made the internal eval a required step for any model change.
Sources
- Measuring Massive Multitask Language Understanding — Hendrycks et al., 2020paper
- Holistic Evaluation of Language Models — Liang et al., 2022paper
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues? — Jimenez et al., 2023paper
Check yourself
Three questions. Every option is explained once you answer.