How do you evaluate whether a fine-tune actually worked?
- 1 min — baseline and three axes
- 2 min — holdout design and statistics
- 2 min — judges, humans, pairwise
- 2 min — gate, shadow, canary
Compare against the strongest cheap baseline — the base model with your best prompt — on held-out data, on three axes: target-task quality, regression on general capability and safety, and operational metrics (latency, cost). Falling training loss shows fit to training data, not generalisation. Eval size matters: at 80% accuracy on n=500, the 95% CI is ±3.5 points, so a 2-point "win" is noise unless you use a paired test on the same items.
A fine-tuned support-reply model for a developer-tools company scores 4.3 vs 4.0 (5-point rubric) over the prompted baseline. Before shipping: (1) re-score on 400 tickets from a later month than training data — gap shrinks to 0.15; (2) safety/refusal suite — over-refusal up from 1% to 4% on security questions; (3) blind pairwise review of 300 pairs by 3 support leads — 58% prefer the fine-tune; (4) fix the refusal regression with 200 counter-examples, re-run, then shadow for a week before a canary.
- Hold out by time or by customer, not random split — near-duplicate tickets leak across random splits and inflate scores.
- LLM-as-judge for scale, validated against human labels on 200+ items with reported agreement; re-validate when the judge model changes.
- Report confidence intervals and use paired tests (McNemar for pass/fail, bootstrap for rubric means).
- Write the eval gate before training: which metric must improve by how much, which regressions block.
- Org scope: an evals team owns the shared regression and safety suites every fine-tune must pass; the product team owns the task set. No team grades its own fine-tune alone.
- Where it breaks: evaluating on data from the same synthetic pipeline that produced training data — the model learns the generator's quirks and the eval rewards them.
What the answer sounds like at each level
Compares to the base model on a held-out set with a task metric.
Uses time-based holdout, regression and safety suites, pairwise human review, and CIs.
Pre-registers eval gates, separates graders from builders organisationally, and validates judges against humans as an ongoing service.
Follow-ups the interviewer will push
1Eval set has 200 items; fine-tune wins by 3 points. Ship?
At ~80% accuracy, ±5.5-point CI on n=200 — not enough alone. Run a paired test on discordant items; if p>0.05, extend the set or rely on pairwise human preference.
2Now production metrics disagree with offline evals.
Check distribution shift (new ticket types), check judge drift, and compare online outcome metrics (resolution, escalations) — online wins; refresh the offline set from recent traffic.
Red flags interviewers listen for
- Cites training loss as evidence the fine-tune works.
- Random train/test split on near-duplicate data.
- No regression or safety evaluation.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Design | Strong baseline, time/customer holdout, regression suites. |
| Statistics | CIs with n and paired tests. |
| Process | Pre-registered gate, independent grading, staged rollout. |
Sources
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Zheng et al., 2023paper