How do you decide between prompting, RAG and fine-tuning?
- 1 min — the escalation ladder
- 2 min — knowledge vs behaviour diagnosis
- 2 min — cost/time per option with a worked example
- 2 min — carrying cost, compliance, rollout
Escalate in order of cost and reversibility. Prompting first — hours to iterate, zero infra: instructions, examples, structured output. RAG when the gap is knowledge: private, large, changing or permissioned facts — 2–6 weeks to production-grade. Fine-tuning when the gap is behaviour prompting can't fix reliably: strict format, house style, a narrow classifier, or distilling a large model into a small one for 5–20× lower cost per call — 4–8 weeks plus a carrying cost.
A medical-coding assistant: code sets and guidelines change yearly → retrieval. Output must follow a strict structure with house phrasing → prompting plus structured output gets to 93% on a 500-chart golden set. Volume is 3M charts/month and the frontier model costs $0.02/chart = $60k/month → fine-tune an 8B model on 20k frontier outputs that coders approved; it reaches 92% at ~$0.002/chart self-hosted. Retrieval stays in front of both.
- Ask for the eval first: no fine-tune without a golden set showing prompting plateaued across ≥3 prompt iterations.
- Fine-tunes carry cost: retrain on base upgrades, data pipelines, and provider deprecation of the base (hosted fine-tunes die with their base model).
- RAG gives attribution and deletion (GDPR erasure is a document delete); weights give neither.
- Org scope: the platform team offers all three as paved roads (prompt registry, retrieval service, fine-tune pipeline); product teams bring the eval. Rollout of a distilled model: shadow for 2 weeks, then canary 5% with the frontier model as fallback.
- Where it breaks: fine-tuning to inject facts. Slow to update, unattributable, and the model hallucinates confidently around half-learned facts.
3M charts/month. Frontier: 4,000 in / 500 out at $3 / $15 per M = $0.012 + $0.0075 ≈ $0.02 → $58.5k/month. Distilled 8B self-hosted on 2×H100 at $2.50/hr each ($3.6k/month) handling ~4,000 charts/hour at 70% utilisation covers ~2M charts/month per pair; 2 pairs = $7.2k/month plus ~0.5 FTE. Break-even inside the first quarter if accuracy holds.
What the answer sounds like at each level
Knows RAG is for knowledge and fine-tuning for behaviour; tries prompting first.
Requires a plateaued eval before fine-tuning, plans the carrying cost, and uses distillation for cost.
Provides all three as platform paved roads with eval gates, and makes the decision with cost-per-call and deletion/attribution requirements.
Follow-ups the interviewer will push
1Now a regulated tenant needs right-to-erasure.
Keep personal data out of weights entirely. RAG with per-document deletion satisfies erasure; a fine-tune trained on that data would need retraining.
2Distilled model is 1 point worse but 10× cheaper. Ship?
Depends on the cost of that point: compute expected error cost per 1k cases vs $ saved. For medical coding with human review downstream, usually yes, with the frontier model on low-confidence cases.
Red flags interviewers listen for
- Fine-tunes before trying structured prompting.
- Uses fine-tuning to keep facts current.
- Ignores base-model deprecation of hosted fine-tunes.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Diagnosis | Classifies the gap as knowledge vs behaviour vs cost. |
| Evidence | Demands a golden set and plateau evidence before escalating. |
| Economics | Works cost per call and break-even for distillation. |
Sources
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Lewis et al., 2020paper
- LoRA: Low-Rank Adaptation of Large Language Models — Hu et al., 2021paper