What is LoRA, and why is it the default way to fine-tune?
- 2 min — mechanism and parameter math
- 2 min — QLoRA and memory
- 2 min — multi-adapter serving
- 1 min — forgetting and base upgrades
LoRA freezes the base weights W and learns a low-rank update ΔW = B·A (rank r, 8–64; default 16) on selected linear layers. For Llama-3.1-8B with r=16 on all attention and MLP projections, that is ~42M trainable parameters — 0.5% of the model — and an adapter of ~84 MB in BF16. Training memory drops because optimizer state exists only for the adapter. QLoRA keeps the frozen base in 4-bit NF4, so a 65–70B model fine-tunes on a single 48–80 GB GPU.
A legal-tech platform serves one open-weights 8B base to 50 law-firm tenants, each with an adapter for clause style and citation format. vLLM multi-LoRA serving loads adapters per request on the shared base (e.g. --max-loras 16, hot adapters on GPU, the rest in CPU memory), so 50 variants run on 2 GPUs instead of 50 deployments. Adapter swap overhead is a few percent of throughput.
- Knobs that matter: rank (16 is a good default), target modules (all linear layers beat attention-only), learning rate (1e-4 to 2e-4), epochs (1–3). Data quality dominates all of them: 2k clean examples beat 20k noisy ones.
- Full fine-tuning can win on large distribution shifts (new language, new domain vocabulary); for behaviour/format changes LoRA lands within ~1 point at 1–5% of the compute.
- Adapters are bound to an exact base version. Base upgrade = retrain and re-eval every adapter; budget it per quarter.
- Org scope: the ML platform owns the training recipe, adapter registry (adapter ↔ base hash ↔ eval report) and multi-LoRA serving; tenant or product teams own data. Buy managed fine-tuning if <5 adapters; build on vLLM when tenant count grows.
- Where it breaks: catastrophic forgetting still happens with narrow data. Always run a general regression set (instruction following, safety refusals) beside the target task.
Llama-3.1-8B, r=16, all 7 linear projections per layer: q 131k + k 82k + v 82k + o 131k + gate/up/down 3 × 295k ≈ 1.31M per layer × 32 layers ≈ 42M params ≈ 0.5% of 8B; 84 MB in BF16. QLoRA on 70B: NF4 base ≈ 35–40 GB, fits one 80 GB H100 with room for activations.
What the answer sounds like at each level
Explains low-rank adapters on frozen weights and why they're cheap.
Picks rank, target modules and LR; uses QLoRA for large bases; runs regression evals for forgetting.
Runs an adapter registry and multi-LoRA serving for many tenants, budgets base-upgrade retraining, and decides managed vs self-built by adapter count.
Follow-ups the interviewer will push
1How many parameters does r=16 add to one 4096×4096 projection?
r × (in + out) = 16 × 8,192 = 131k, vs 16.8M in the full matrix — 0.8%.
2Now 500 tenants. Does multi-LoRA still work?
Yes with tiering: keep the top ~20 adapters on GPU, page others from CPU/NVMe, and group requests by adapter in the batcher; watch adapter-load latency p95 and cold-adapter hit rate.
Red flags interviewers listen for
- Thinks LoRA adds knowledge as reliably as retrieval.
- Evaluates only on the target task, never general regressions.
- Doesn't tie adapters to a base model version.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Mechanism | ΔW = BA, frozen base, rank and target modules, with numbers. |
| Operations | Multi-LoRA serving and adapter registry tied to base version. |
| Risk | Regression eval for forgetting and retrain budget. |
Sources
- LoRA: Low-Rank Adaptation of Large Language Models — Hu et al., 2021paper
- QLoRA: Efficient Finetuning of Quantized LLMs — Dettmers et al., 2023paper