Prompt, retrieve or fine-tune? A decision guide

A platform team spends three weeks preparing data to fine-tune a model on 2,000 pages of its engineering wiki, so the internal assistant will “know” the handbook. On launch day someone asks who is on call for payments. The model answers with last quarter's rota, the snapshot it was trained on, and cannot say which page the answer came from. A retrieval index over the same 2,000 pages would have taken two days to build, re-indexed itself nightly, and linked every answer to its source.

The team picked a technique before naming the problem. The fix is to diagnose first: what, precisely, is wrong with today's output?

Symptom on the golden set	Problem type	First fix
Ignores a rule, wrong format	Instruction	Prompt + 3–5 examples
Doesn't know your data or this week's change	Knowledge	Retrieval
Understands, but style drifts on 1 in 10	Behaviour	Supervised fine-tune
Right, but too slow or costly at volume	Cost	Distil into a small model
Two acceptable answers; reviewers prefer one	Preference	Preference tuning (DPO)

Name the failure; the technique mostly picks itself. Each row is a different problem, not a different taste.

Prompting and retrieval, briefly

Default to prompting: clear rules, a fixed output schema and 3–5 worked examples. Escalate only when the golden set shows a failure that survives two honest rewrites. Prompts deploy in minutes and roll back with prompt versioning.

Add retrieval when the knowledge exceeds roughly 100,000 tokens, changes weekly or faster, must be filtered per user by access control, or must be cited. Below that size and churn, put the whole corpus in the prompt with prompt caching; at a 0.1× cached-read rate, a stable 50,000-token reference costs about as much per call as 5,000 uncached tokens, with no index to maintain.

A worked case: claims triage

An insurer routes incoming claim notes into 12 queues and extracts six fields: policy number, incident date, claim type, estimated amount, injury flag and third-party flag. Volume is 40,000 claims a day, about 1.2 million a month. The golden set is 500 historical claims, each labelled independently by two adjusters, with disagreements resolved by a third. The accuracy figures below are assumptions for the arithmetic, not benchmark results; the shape of the decision is the point.

Step 1, prompt a frontier model. A 14-rule prompt with six examples is about 2,500 input tokens, and the JSON answer is about 150 output tokens. Suppose it scores 93% queue accuracy on the golden set. At illustrative prices of $3 / $15 per million input/output tokens, that is about $11,700 a month.

Step 2, consider retrieval, and skip it. The knowledge is 14 rules, not 40,000 documents. Retrieval would add latency and moving parts and nothing the prompt lacks. Diagnosis says the remaining problem is cost.

Step 3, distil. Distillation uses the big model as a teacher. Run it over 20,000 historical claims, have adjusters correct the 5–10% it gets wrong, and fine-tune a small model on the result. The learned behaviour replaces the long prompt, so each training example is about 700 tokens. Train with LoRA on an 8B open model, or through a hosted fine-tuning API.

example.pypython35 lines

# Every price here is an illustrative assumption (Oct 2026), not a quote.
FRONTIER_IN, FRONTIER_OUT = 3.00, 15.00   # $ per 1M tokens, large hosted model
SMALL_IN, SMALL_OUT = 0.30, 1.20          # $ per 1M tokens, small fine-tuned model
HOSTED_TRAIN = 3.00                       # $ per 1M training tokens, hosted fine-tune
GPU_HOUR = 3.00                           # $ per H100-hour, self-run LoRA
LORA_TOKENS_PER_S = 4_000                 # training throughput, 8B model + LoRA, one H100

CLAIMS_PER_MONTH = 40_000 * 30


def per_call(tokens_in: int, tokens_out: int, price_in: float, price_out: float) -> float:
    return (tokens_in * price_in + tokens_out * price_out) / 1_000_000


def main() -> None:
    prompt_only = per_call(2_500, 150, FRONTIER_IN, FRONTIER_OUT)
    print(f"prompt-only, frontier : ${prompt_only:.5f}/claim  ${prompt_only * CLAIMS_PER_MONTH:>9,.0f}/month")

    teacher_labels = 20_000 * prompt_only               # distillation: big model labels history
    train_tokens = 20_000 * 700 * 3                     # examples x tokens x epochs
    hosted_job = train_tokens / 1_000_000 * HOSTED_TRAIN
    lora_hours = train_tokens / LORA_TOKENS_PER_S / 3600
    print(f"teacher labelling     : ${teacher_labels:,.0f} for 20,000 claims")
    print(f"training tokens       : {train_tokens / 1e6:.0f}M -> hosted job ${hosted_job:,.0f}")
    print(f"self-run LoRA         : {lora_hours:.1f} H100-hours -> ${lora_hours * GPU_HOUR:,.0f}")

    distilled = per_call(700, 60, SMALL_IN, SMALL_OUT)
    print(f"distilled small model : ${distilled:.5f}/claim  ${distilled * CLAIMS_PER_MONTH:>9,.0f}/month")
    saving = (prompt_only - distilled) * CLAIMS_PER_MONTH
    one_time = teacher_labels + hosted_job
    print(f"monthly saving ${saving:,.0f}; one-time spend ${one_time:,.0f} pays back in {one_time / saving * 30:.1f} days")


if __name__ == "__main__":
    main()

texttext6 lines

prompt-only, frontier : $0.00975/claim  $   11,700/month
teacher labelling     : $195 for 20,000 claims
training tokens       : 42M -> hosted job $126
self-run LoRA         : 2.9 H100-hours -> $9
distilled small model : $0.00028/claim  $      338/month
monthly saving $11,362; one-time spend $321 pays back in 0.8 days

The compute is trivial: $321, and a hosted job that finishes in one to four hours including queue time, not weeks. The real cost is the one or two engineer-weeks spent curating labels and building the eval, which still pays back within the first month. The decision rests on one condition: the distilled model must match the frontier model on the same 500-claim golden set. Suppose it scores 94%. Without that number, the saving is a guess.

Step 4, preference tuning, only if needed. Adjusters now say both models route correctly, but they prefer short, specific rationales. That is a preference, not an error. Collect 1,000–2,000 pairs from their reviews (chosen rationale, rejected rationale) and run Direct Preference Optimization, which learns from pairs directly without training a separate reward model. Several hosted APIs offer it alongside supervised fine-tuning. Do it after supervised fine-tuning, never instead of it: DPO shapes choices between acceptable answers, it does not repair wrong ones.

Step	One-time	Per month	Golden-set accuracy	Wall clock
Prompt, frontier model	$0	$11,700	93%	1–2 days
+ retrieval	skipped	—	no gain expected	—
Distil + LoRA/SFT	$321 + 1–2 eng-weeks	$338	94%	hours of training
+ DPO on 1,500 pairs	≈$100	$338	same, preferred style	hours

The claims-triage case end to end. Accuracy figures are assumed for illustration; costs follow from the stated prices.

What a fine-tune ties you to

A LoRA adapter is a set of low-rank weight deltas for one exact base checkpoint. It cannot be moved to a different base model, not even the next point release of the same family. A base-model upgrade therefore always means retraining. Hosted fine-tunes carry the same dependency: when the provider retires the base, your tuned model goes with it (see model deprecation).

Plan for it. The durable assets are the training set, the golden set and a training script that runs end to end. Treat the weights as a build artifact. With those three in version control, moving to a new base is an afternoon of compute and an eval run; without them, it is the original project again.

The rules, compressed

Default to prompting. Escalate only when the golden set shows a failure that survives two honest rewrites.
Retrieve when knowledge is over ~100k tokens, changes weekly, is per-user, or must be cited. Under that, use a cached prompt.
Fine-tune for behaviour or cost: a narrow, stable task running more than ~100,000 times a month, where a small model can match the large one on the golden set.
Distil before you hand-label. A frontier teacher plus human correction of its errors is usually faster than labelling from scratch.
Preference-tune last, with at least 1,000 real pairs, to choose between acceptable answers.

Key takeaways

The fork is diagnostic, not technical: name the failure on the golden set and the technique largely follows.
In 2026 fine-tuning pays mostly as a cost move, shifting a narrow, proven task from a frontier model to a small one, rather than as a quality move.
Weights are disposable and expire with their base model; the labelled data, the eval and the training script are what you actually own.
Knowledge belongs where it can be filtered, cited and deleted. Weights can do none of those.

Sources

LoRA: Low-Rank Adaptation of Large Language Models — Hu et al., 2021paper
Direct Preference Optimization: Your Language Model is Secretly a Reward Model — Rafailov et al., 2023paper
Distilling the Knowledge in a Neural Network — Hinton, Vinyals & Dean, 2015paper
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Lewis et al., 2020paper
QLoRA: Efficient Finetuning of Quantized LLMs — Dettmers et al., 2023paper

#Prompting and retrieval, briefly

#A worked case: claims triage

#What a fine-tune ties you to

#The rules, compressed