Why use RAG instead of fine-tuning an LLM?
- 1 min — separate knowledge from behaviour
- 2 min — decision rule with one example each way
- 2 min — attribution, revocation, cost crossover
- 1 min — where it breaks: domain vocabulary → embedding fine-tune
They solve different problems. Fine-tuning changes behaviour — format, tone, task-specific reasoning, domain vocabulary. RAG changes knowledge — what facts the model can see at answer time. Weights are a lossy, expensive, slow-to-update store of facts. An index is a cheap, fast, auditable one.
Knowledge changes weekly → RAG. Output shape is wrong → fine-tune. Often you need both.
An internal policy assistant. HR updates leave policy every quarter. Fine-tuning means a retraining cycle per change plus no way to show the source. RAG means re-index one document and the answer cites it. Conversely, if that same assistant must always reply in a four-field JSON schema with a legal disclaimer, that's a fine-tune (or structured decoding), not retrieval.
- RAG gives you attribution, which is usually a compliance requirement, not a nicety.
- RAG gives you revocation — delete a doc, the model forgets. You cannot un-train a fact.
- Fine-tuning buys you token cost and latency, since you stop stuffing context. At very high QPS that matters.
- Where it breaks: RAG can't fix a model that doesn't understand your domain's language well enough to form a good query. That's the case for a fine-tuned embedding model, not a fine-tuned LLM.
- Org scope: retrieval is usually a platform-team service that product teams consume; fine-tuning is a per-product decision with its own eval and retraining owner. Default to buying the index (managed pgvector, OpenSearch, Qdrant Cloud) and building only the eval harness. A fine-tune needs a named owner for every base-model deprecation, or it becomes an orphan.
What the answer sounds like at each level
RAG adds knowledge at query time; fine-tuning changes behaviour. Use RAG when facts change or need citations.
Splits the decision by failure: wrong facts → RAG, wrong shape or tone → fine-tune or structured decoding. Names revocation and attribution as the deciding requirements, and notes they combine.
Frames it as an operating-model choice: who owns the index vs who owns a model that must be retrained on every base-model deprecation. Puts a cost crossover on the table (context tokens × QPS vs training plus hosting) and sets a revisit trigger.
Follow-ups the interviewer will push
1Now traffic is 10× and context stuffing dominates the bill. Does the answer change?
Partly. Keep RAG for knowledge; cut tokens first with prompt caching on the static prefix (reads bill at ~0.1× input on Anthropic), tighter top-k and a reranker. Only then consider a LoRA fine-tune to drop long instructions and few-shot examples — it removes behavioural tokens, not knowledge tokens.
2Now a regulated tenant (health, finance) requires right-to-erasure.
That settles it for RAG: delete the document, tombstone its chunks, and it is gone on the next query. A fact trained into weights cannot be provably removed, so fine-tuning on customer data is off the table without a retrain-on-request process nobody wants.
3Now the hosted model provider is down.
RAG is model-agnostic: the gateway fails over to a second provider with the same retrieved context. A fine-tuned model is pinned to one vendor or your own GPUs, so your fallback is a non-fine-tuned model with worse behaviour — plan and eval that degraded mode.
Red flags interviewers listen for
- Says fine-tuning is how you 'teach the model your documents' with no mention of staleness or revocation.
- Treats RAG and fine-tuning as mutually exclusive.
- No mention of how either choice is evaluated before and after.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Problem framing | Separates knowledge vs behaviour before naming a technique. |
| Trade-offs | Names revocation, attribution, token cost and update cadence with a concrete crossover. |
| Org scope | Says who owns the index vs the model and what happens on a base-model deprecation. |
Sources
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Lewis et al., 2020paper
- LoRA: Low-Rank Adaptation of Large Language Models — Hu et al., 2021paper