Fine-tuning
Continuing to train a pretrained model on your own examples to change its behaviour, format or style. Not a reliable way to add facts.
Fine-tuning takes a model that already understands language and trains it a little more on your own examples, such as 3,000 pairs of 'product description' and 'correctly filled attribute form'. The model's weights change, so it behaves like your examples by default.
It is good for teaching a format, tone or narrow task. It is usually the wrong tool for keeping facts up to date; for that, use Retrieval-Augmented Generation (RAG).
Sending an experienced chef on a short course in your restaurant's house style: they cook your way afterwards, but they still need today's menu card.
Fine-tuning on internal docs to inject knowledge, then being surprised the model still hallucinates and goes stale after the next policy change.
Variants: full fine-tuning updates every weight and needs ~16 bytes per parameter with Adam (≈130 GB for 8B before activations). LoRA & parameter-efficient fine-tuning trains low-rank adapters, about 0.5% of parameters at rank 16 on all linear layers of an 8B model (~42M), and fits on one 24–48 GB GPU. QLoRA keeps the frozen base in 4-bit and fine-tuned a 65B model on a single 48 GB GPU. Preference optimization (DPO) (DPO) shapes which of two outputs the model prefers. Hosted APIs offer supervised and sometimes preference tuning on selected models.
What it fixes: format adherence, label sets, domain vocabulary, shorter prompts (behaviour learned instead of described), and Model distillation of a large model into a small one. Costs: forgetting of general skills, safety regressions, overfitting under ~1,000 examples, and a re-tune every time the base model is upgraded. For retrieval quality, Fine-tuning an embedding model is often the higher-leverage target than tuning the generator.
Start with LoRA rank 16 on all linear layers of an 8B open-weights model (or the provider's hosted fine-tuning), 2–3 epochs, learning rate ~1e-4, 2,000–10,000 examples, 10% held out. Evaluate on the task golden set plus a 200-item general-capability regression set. Serve adapters with vLLM multi-LoRA so tenants share one base model.
E-commerce catalog: extracting 40 attributes from supplier descriptions needed a 2,400-token prompt with 12 examples on a frontier model, at 91% field accuracy. The team LoRA-tuned Llama-3.1-8B (rank 16) on 3,000 labelled products: 45 minutes on one H100. Prompt fell to 300 tokens, field accuracy rose to 96% on a 500-item held-out set, and cost per 1,000 products dropped from ~$8 to ~$0.40 on a self-hosted vLLM node. Brand and price facts still come from the product database, not the weights.
Commit to an order: prompt first, RAG for knowledge, fine-tune for behaviour, latency or cost at volume, and only with a held-out eval that proves the gain. Quote the economics: training is cheap (tens of dollars for LoRA on 8B), serving and re-tuning on each base-model upgrade are the real costs. The follow-up is usually 'what if the base model is deprecated?': keep the training set and eval in version control so a re-tune is a pipeline run, not a project.
| Method (8B model) | Trainable params | GPU memory | Use when |
|---|---|---|---|
| Full fine-tune | 8B (100%) | ≈130 GB+ | Large data, new skills |
| LoRA r=16 | ≈42M (0.5%) | ≈20–24 GB | Default for behaviour |
| QLoRA 4-bit r=16 | ≈42M (0.5%) | ≈10–12 GB | One 24 GB GPU |
| DPO on top | same as base method | + reference pass | Preference shaping |
Numbers worth knowing
- LoRA trainable share
- ≈0.5%rank 16 on all linear layers of an 8B model (~42M params)
- Minimum useful dataset
- 1,000–3,000 examplesnarrow format or extraction task; below ~500 it mostly overfits
- QLoRA reach
- 65B on one 48 GB GPU4-bit NF4 base, per the QLoRA paper
- LoRA compute cost
- $10–200 per run8B model, 3k–50k examples, rented H100 at ~$2.50/hour (illustrative, Oct 2026)
How to decide
| When | Do this |
|---|---|
| Problem is missing or changing facts | Use Retrieval-Augmented Generation (RAG); do not fine-tune. |
| Prompt exceeds ~2k tokens of format instructions and examples on a route above ~1M calls/month | Fine-tune the behaviour in and cut the prompt. |
| A large model's quality is needed at small-model price on one narrow task | Distil: generate labels with the large model, LoRA-tune a small one. |
| Fewer than ~500 good examples | Stay with few-shot prompting until you have more data. |
Cost maths
Total cost = training + eval + serving + re-tunes. Assume an H100 at $2.50/hour (illustrative). LoRA on 3,000 examples × 1,000 tokens × 3 epochs ≈ 9M tokens ≈ under 1 GPU-hour → ~$3 compute; with evals and retries call it $50. Serving one dedicated H100 24/7 ≈ $1,800/month, which only pays back if it replaces more than $1,800/month of API spend. Each base-model upgrade means another run plus a full eval.
Operating it
- API spend on one narrow route above ~$5k/month with a stable task definition
- Format errors persist above 2% despite structured outputs
- Prompt length for instructions plus examples exceeds 4k tokens
- Task golden-set score vs base model + prompt: alert if the gap shrinks below 2 points
- General-capability regression set: alert on −3 points
- Format error rate on production traffic: alert above 0.5%
When it goes wrong
An IT-helpdesk model fine-tuned on two years of resolved tickets kept telling employees to connect to the old VPN hostname, retired six months earlier.
Weekly QA sample of 100 responses flagged 14 with the dead hostname.
The training data contained facts (hostnames, version numbers, owners) that changed; the model memorised them as behaviour.
Stripped volatile entities from training targets and replaced them with placeholders filled by retrieval at runtime, plus a regression slice of 'facts that changed since training' in the eval.
Related terms
Where it shows up
Sources
- LoRA: Low-Rank Adaptation of Large Language Models — Hu et al., 2021paper
- QLoRA: Efficient Finetuning of Quantized LLMs — Dettmers et al., 2023paper
- Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs — Ovadia et al., 2023paper
Check yourself
Three questions. Every option is explained once you answer.