Design an LLM evaluation platform used by 30 product teams.
- 5 min — users, run volume, latency needs (PR vs nightly), budget
- 10 min — data model: datasets, configs, runs, item scores, judges
- 10 min — execution: fan-out, gateway budgets, batch API, caching
- 10 min — judges, calibration, statistics, gate semantics
- 5 min — rollout to 30 teams, ownership, override policy
An eval platform is three products sharing one data model:
- Dataset registry — versioned golden sets with provenance, slices and contamination checks, addressed by content hash.
- Execution service — runs a candidate config (model, prompt version, retrieval index, tools) over a dataset at scale, through the gateway, under a per-team budget.
- Scoring and reporting — deterministic checks, LLM judges calibrated against human labels, paired significance tests and run-to-run diffs.
The contract with teams is a config file plus a CI hook, so an eval gate blocks a merge exactly like a failing unit test.
A team commits evals.yaml: dataset support-triage@v14, scorers (exact match on routing label, rubric judge on reply quality, latency and cost), gate 'no metric drops >2 points with p < 0.05 vs main'. On a PR, CI calls the platform; workers fan out 2,000 items, judges score, and the PR receives a diff table plus the 20 largest regressions with full traces. Nightly full runs use the provider batch API at ~50% off. Because datasets and configs are hashed, a run from six months ago is reproducible.
- The judge is a model you ship. Version it; measure agreement with humans per dataset (target Cohen's κ ≥ 0.6, see inter-annotator agreement); re-calibrate when the judge model changes, or every team's trend line jumps on the same day for reasons unrelated to their product.
- Statistics as a service. Paired bootstrap on item scores. With 2,000 items a 2-point move on a 70% metric is near the edge of significance, so the platform — not each team — defines 'regression'.
- Failure modes: teams over-fitting a golden set they can see (keep a hidden holdout); judge position bias in pairwise scoring (swap order and average); provider timeouts counted as model failures (separate infra errors from scores).
- Org scope: a 4–6 person platform team. Build registry, runner and gating — they encode your policy. Adopt an existing tracing UI rather than building one. Rollout: three design-partner teams, then CI integration as the paved road, then gates required for tier-1 launches.
Assumptions: 30 teams × 5 runs/day (PR + nightly) = 150 runs/day; 2,000 items/run → 300k items/day; each item = 1 candidate call (2k in / 500 out) + 1 judge call (3k in, 2k of it a cached rubric prefix / 300 out); illustrative prices $3/M input, $15/M output, cache reads 0.1× ($0.30/M), batch 50% off.
- Calls: 600k/day.
- $/day at list: candidate 600M in × $3 = $1.8k + 150M out × $15 = $2.25k; judge 300M uncached × $3 = $0.9k + 600M cached × $0.30 = $0.18k + 90M out × $15 = $1.35k → ≈ $6.5k/day ≈ $195k/month. Moving the nightly 60% to batch → ≈ $135k/month.
- Throughput: work lands in ~8 working hours → 21 calls/s average, PR bursts 5× ≈ 100 calls/s; at ~10 s per call that is ~1,000 in flight. Provider rate limits, not workers, bind — 10 async worker pods are enough.
- Storage: 300k results × ~20 KB (prompt, output, scores, trace) ≈ 6 GB/day ≈ 2.2 TB/year in object storage plus a columnar table for queries.
What the answer sounds like at each level
Describes a script that runs prompts over a test set and scores them with an LLM, with results in a dashboard.
Versions datasets, separates runner from scorers, calibrates the judge against humans, adds significance tests and CI gates.
Treats judges and datasets as governed artefacts, designs per-team budgets and holdouts, sequences adoption across 30 teams, and defines who may override a gate.
Follow-ups the interviewer will push
1One team needs evals on regulated health data.
Same runner, different policy: the dataset and traces live in that team's tenant and region, judges run on an approved endpoint, raters are trained and contractually covered, and reports show aggregates only. Storage location and endpoint are config, not a fork.
2The judge provider is down during a release freeze.
Gates fall back to deterministic scorers plus a secondary judge pre-calibrated on the same dataset (its κ is recorded). If neither is available, tier-1 gates fail closed and others warn; any override is explicit and audited.
3Eval spend triples in a quarter.
Show per-team spend; cache candidate outputs by (config hash, item) so unchanged configs never re-run; sample 500 items on PRs and run the full set nightly; replace the frontier judge with a fine-tuned small judge once its agreement matches.
Red flags interviewers listen for
- Treats the LLM judge as ground truth with no human calibration.
- No dataset versioning, so a result cannot be reproduced.
- Reports one aggregate number with no significance test or per-slice view.
- Builds for one team's use case and calls it a platform.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Data model | Hashes datasets and configs; runs are reproducible and comparable item by item. |
| Judge rigour | Human calibration with an agreement threshold, bias controls, re-calibration on judge change. |
| Cost | Quantifies token spend, uses batch and caching, sets per-team budgets. |
| Adoption | Paved-road CI integration, staged rollout, clear gate ownership. |
Sources
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Zheng et al., 2023paper
- Holistic Evaluation of Language Models — Liang et al., 2022paper