LoRA & parameter-efficient fine-tuning
Low-Rank Adaptation: fine-tune by training small add-on matrices while the original weights stay frozen; cheap, fast, swappable per task or tenant.
Fully Fine-tuning a large model means changing all of its billions of Parameters & weights, which needs a lot of expensive hardware. LoRA is a shortcut: freeze the original model and train a tiny add-on, well under 1% of its size, that adjusts its behaviour.
Because the add-on is small, you can train it on one GPU and keep many of them, one per customer or task, all sharing the same base model.
Clip-on lenses for a camera. Rather than rebuilding the camera for each kind of shot, you snap on a small lens that changes how it sees, and swap lenses as needed.
Expecting an adapter to keep working after the base model is upgraded. LoRA weights are deltas to one specific checkpoint; a new base means retraining and re-evaluating, even when the shapes still load without error.
For a frozen matrix W (d × k), LoRA learns ΔW = B·A with A (r × k) and B (d × r), rank r = 8–64, scaled by α/r; B starts at zero so training begins at the base model's behaviour. Only A and B get gradients and optimiser state, so memory drops sharply: the LoRA paper reports 10,000× fewer trainable parameters and 3× less GPU memory than full fine-tuning of GPT-3 175B. Apply it to all linear layers (attention and MLP); attention-only LoRA underperforms on most tasks.
At serving time you either merge B·A into W (zero overhead, one model per task) or keep adapters separate and hot-swap them: vLLM's multi-LoRA (--enable-lora, --max-loras, --max-lora-rank) and S-LoRA batch requests for different adapters over one base. QLoRA freezes the base in 4-bit NF4, fitting a 65B fine-tune on one 48 GB GPU. Limits: LoRA learns format, style and narrow tasks well and is weaker than full fine-tuning for large knowledge or capability shifts; an adapter is a delta to one exact checkpoint.
Rank 16, alpha 32, dropout 0.05, all linear layers, lr 1e-4–2e-4, 2–3 epochs, BF16 base (QLoRA 4-bit only when memory forces it). Store the base model revision hash in adapter metadata and refuse to load on mismatch. Serve with vLLM multi-LoRA when you have > 3 adapters on one base.
Llama-3.1-8B, rank 16 on all seven linear projections: per layer q 131k + k 82k + v 82k + o 131k + gate/up/down 3 × 295k ≈ 1.31M parameters; × 32 layers ≈ 42M trainable (0.5%). The BF16 adapter is ~84 MB vs 16 GB for the base. A SaaS insurer-onboarding product keeps one adapter per enterprise tenant (50 tenants = 4.2 GB) and serves them all from one base deployment with tenant facts supplied by Retrieval-Augmented Generation (RAG).
For 'customise a model for 50 tenants cheaply', answer: shared base + per-tenant LoRA via multi-adapter serving for behaviour, per-tenant RAG for facts, adapters pinned to a base revision, and a per-tenant eval gate. Expect: 'what happens when the provider upgrades the base?' — every adapter is retrained and re-evaluated; budget that as recurring cost.
Full fine-tune
- Update all 8B weights
- ~16 B/param state
- Multi-GPU memory
- 16 GB copy per task
LoRA (r = 16)
- Base weights frozen
- Train ~42M (0.5%)
- One GPU
- ~84 MB adapter
Numbers worth knowing
- Trainable params
- ~42M (0.5%)Llama-3.1-8B, r = 16, all linear layers
- Adapter size
- ~84 MBsame config, BF16
- QLoRA reach
- 65B fine-tune on one 48 GB GPU4-bit NF4 frozen base
How to decide
| When | Do this |
|---|---|
| Many tenants or tasks need different behaviour on one base | Per-tenant LoRA with multi-adapter serving. |
| A single task, latency-critical | Merge the adapter into the weights; no runtime overhead. |
| LoRA at r = 64 still trails the target | Full fine-tune, or the base lacks the capability — try a larger base first. |
Cost maths
Serving cost for N tenants: full fine-tunes need N deployments (or slow swaps); LoRA needs one. 50 tenants × an 8B model on one L40S-class GPU each at an assumed $1.50/hour = $54,750/month; one base with 50 adapters on 2 GPUs (for throughput) ≈ $2,190/month. Retraining 50 adapters after a base upgrade ≈ 50 × 1 GPU-hour ≈ $150 plus eval time.
Operating it
- Adapters per base > 20 → multi-LoRA serving with adapter caching (max_loras tuned to the hot set)
- Adapter swap/load latency adds > 100 ms p95 → raise max_cpu_loras or pin hot adapters
- Tenants needing knowledge rather than behaviour → move that need to RAG
- Per-tenant canary eval drop > 3 points after any deploy
- Adapter load failures or base-hash mismatches > 0
- Adapter cache miss rate > 10% (swap thrash)
When it goes wrong
After an upgrade to a newer revision of the same base model, one tenant's field-extraction accuracy fell from 91% to 78% with no errors in logs.
The nightly per-tenant canary eval flagged the drop the morning after the deploy.
The adapter was trained against the old revision; shapes matched, so vLLM loaded it silently, but the deltas no longer corresponded to the new weights.
Pinned the base revision hash in every adapter's metadata, made the loader refuse mismatches, and added an automatic retrain-and-eval job triggered by any base upgrade.
Related terms
Sources
- LoRA: Low-Rank Adaptation of Large Language Models — Hu et al., 2021paper
- QLoRA: Efficient Finetuning of Quantized LLMs — Dettmers et al., 2023paper
- S-LoRA: Serving Thousands of Concurrent LoRA Adapters — Sheng et al., 2023paper
- vLLM: LoRA adapters — vLLM project, 2026docs
Check yourself
Three questions. Every option is explained once you answer.