Model distillation
Training a smaller, cheaper student model to imitate a larger teacher model's outputs on your task.
Distillation gets much of a big model's quality at a fraction of the cost. You run the large, expensive model (the teacher) on lots of examples of your task, save its answers, and then train a small model (the student) to produce the same answers.
The student is faster and cheaper, but it is only good at the specific task it was trained on.
An apprentice who shadows a master on one kind of job until they can do that job almost as well, at a lower day rate.
Distilling a general-purpose assistant and expecting the student to match the teacher everywhere; it only matches on the distribution it was trained on.
Classic knowledge distillation (Hinton et al., 2015) trains the student on the teacher's softened probability distribution, which carries more signal than hard labels; DistilBERT kept ~97% of BERT's GLUE score with 40% fewer parameters and 60% faster inference. With API-only teachers you do sequence-level distillation: generate teacher outputs (sometimes with reasoning traces), filter them with deterministic checks and an LLM-as-judge, then supervised Fine-tuning (usually LoRA) of the student.
It pays on narrow, high-volume steps with a clear eval: Query rewriting, Intent classification & routing, extraction, small rerankers distilled from LLM rerankers, and embedding models. Risks: inherited teacher errors, distribution shift when live traffic drifts from the generation set, and provider terms that restrict using outputs to train competing models; check them before generating. Compare student and teacher per slice on a held-out Golden set and on Shadow traffic before switching.
Teacher: the best available frontier model with the production prompt. Data: 20k–100k real inputs sampled from logs (deduplicated, PII-redacted), teacher outputs filtered by a judge and deterministic checks. Student: LoRA on an 8B open-weights model, served on vLLM. Rollout: 2 weeks of shadow, then a cascade that sends low-confidence inputs back to the teacher.
E-commerce catalog search: query rewriting with a frontier model cost ~$3.30 per 1,000 calls at 1.1 s p95. The team generated 60,000 teacher rewrites, kept 44,000 after a judge filter, and LoRA-tuned an 8B model. On the golden set, downstream recall@20 went from 0.88 to 0.86; on the rare-brand slice it fell from 0.81 to 0.71, so those queries (6% of traffic, detected by a brand-dictionary miss) stay on the teacher. p95 is now 140 ms and blended cost ~$0.31 per 1,000.
Commit: distil the hot, narrow step, not the assistant. Use the big model to build the dataset, the small one to serve it, and the golden set (per slice) to prove the trade, then roll out via shadow and a cascade that routes low-confidence inputs to the teacher. Follow-up: 'traffic drifts next season?': monthly regeneration from fresh logs and a shadow agreement metric that alerts below 90%.
| Metric | Teacher (frontier API) | Student (8B, vLLM) |
|---|---|---|
| Recall@20, golden set | 0.88 | 0.86 |
| Recall@20, rare brands | 0.81 | 0.71 |
| p95 latency | 1,100 ms | 140 ms |
| Cost per 1k calls | $3.30 | $0.12 |
Numbers worth knowing
- DistilBERT trade
- 97% of quality, 40% smaller, 60% fasterGLUE, Sanh et al. 2019
- Teacher-generated dataset size
- 20k–100k examplesnarrow rewriting or classification task
- Typical cost reduction
- 10–30×frontier API teacher vs self-hosted 8B student at steady high volume
How to decide
| When | Do this |
|---|---|
| One narrow step runs above ~1M calls/day with a stable definition | Distil it onto a small model and keep the teacher as fallback. |
| Student trails the teacher on a slice by more than 5 points | Route that slice to the teacher (cascade) rather than ship one model. |
| The task changes monthly | Don't distil yet; the regeneration cadence will eat the savings. |
Cost maths
Teacher per 1k calls = 1k × (800 in × $3 / M + 60 out × $15 / M) = $2.40 + $0.90 = $3.30 (illustrative prices). Student at 2M calls/day (≈23 QPS average, ~70 peak): two H100s for redundancy at ~$2.50/hour each (illustrative) = $120/day, plus ~$120/day of on-call and platform overhead = $240/day ≈ $0.12 per 1k. Teacher at the same volume: $6,600/day, 27× more. Dataset generation: 60k × $0.0033 ≈ $200, once a month.
Operating it
- Teacher spend on one step above ~$5k/month
- Latency of the step is on the critical path and above 300 ms p95
- Student-teacher shadow agreement below 90%
- Student-teacher agreement on 1% shadow sample: alert below 90%
- Per-slice golden-set metrics: alert when any slice drops 5 points
- Cascade escalation rate: alert on 2× baseline (drift signal)
When it goes wrong
Search conversion on seasonal queries dropped 8% in late August after the student model took over query rewriting.
Per-segment click-through dashboard; offline replay showed recall@20 on 'back to school' queries down 12 points.
The distillation set was sampled from spring logs; the student had never seen back-to-school vocabulary.
Monthly regeneration from the latest 30 days of logs, a 1% shadow comparison against the teacher, and a cascade sending unknown-vocabulary queries to the teacher.
Where it shows up
Sources
- Distilling the Knowledge in a Neural Network — Hinton et al., 2015paper
- DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter — Sanh et al., 2019paper
Check yourself
Three questions. Every option is explained once you answer.