How would you serve 2,000 fine-tuned customer adapters on top of one base model?
- 5 min — scope: base model, adapters, traffic skew, SLO
- 10 min — LoRA mechanics and batched kernels
- 10 min — memory tiering and affinity routing
- 10 min — registry, evals, upgrades, failure modes
- 5 min — estimate and ownership
LoRA adapters are small low-rank deltas (W + BA). Multi-LoRA serving keeps one base model in GPU memory and applies a different adapter per request inside the same batch using batched kernels — Punica's SGMV, S-LoRA's unified paging. Adapters are tiered: GPU slots (hot) ← host RAM (warm) ← object storage (cold). vLLM exposes --enable-lora, --max-loras (adapters per batch), --max-lora-rank and --max-cpu-loras; SGLang and LoRAX offer the same pattern.
Per-tenant support-tone adapters on Llama-3.1-8B. A request names tenant-123-v4; the gateway maps it to an adapter id; the router uses adapter affinity (consistent hashing on adapter id with bounded load) so one adapter's traffic lands on a few replicas and loads rarely; on a miss the replica loads from host RAM (~5 ms) or object storage (~1 s). The adapter registry versions each adapter, requires a passing eval before activation, and enforces rank ≤ 16.
- Trade-offs: higher rank and more target modules buy quality at the cost of memory and kernel time; mixing adapters in a batch costs ~10–30% throughput versus base-only; merging an adapter into weights is fastest but ends multi-tenancy.
- Failure modes: cold-start storms when the long tail is hit right after a deploy (pre-warm the top N by traffic); a base-model upgrade invalidating all 2,000 adapters (budget it as a migration); wrong-adapter routing exposing one tenant's tuning to another (registry isolation + tests).
- When not to: if prompting plus RAG gets 80% of the quality, skip adapters.
- Org scope: the serving platform owns kernels, routing and registry; the customer-ML team owns the training pipeline. vLLM/SGLang supply the kernels; build routing, registry and the eval gate.
Assumptions: Llama-3.1-8B (32 layers, hidden 4,096, KV dim 1,024, MLP 14,336); rank-16 LoRA on q/k/v/o/gate/up/down; 2,000 adapters with Zipf traffic (top 100 = 80% of requests); 500 QPS × 300 output tokens; ~5,000 output tok/s per H100 at SLO including LoRA overhead; illustrative $2.50/H100-hour.
- Adapter size: per layer r × (in + out) = 16 × (8,192 + 5,120 + 5,120 + 8,192 + 3 × 18,432) ≈ 1.31M params × 32 layers ≈ 42M ≈ 84 MB in BF16.
- All 2,000: ≈ 168 GB — fits host RAM, not HBM. Keep ~100 hot per GPU ≈ 8.4 GB.
- Load time: host → GPU over PCIe Gen5 (~25 GB/s effective) ≈ 3–5 ms; cold from object storage ≈ 0.5–1.5 s.
- GPUs: 500 × 300 = 150k tok/s ÷ 5,000 ≈ 30 H100s; run 40 ≈ $73k/month.
- Alternative: one dedicated GPU per tenant = 2,000 GPUs ≈ $3.65M/month — multi-LoRA is ~50× cheaper.
What the answer sounds like at each level
Knows LoRA adapters are small and can be swapped on one base model.
Uses batched multi-LoRA kernels, memory tiering and adapter-affinity routing, with numbers on adapter size.
Compares cost with per-tenant deployments, plans base-model migrations, gates adapters with evals and sets ownership across serving and customer-ML teams.
Follow-ups the interviewer will push
1You upgrade the base model.
Adapters do not transfer. Retrain all 2,000 from stored datasets (~0.5 GPU-hour each ≈ 1,000 GPU-hours ≈ $2.5k), evaluate each against the old one, and serve both bases side by side during migration.
2Some tenants need rank 64.
Give them a separate pool with --max-lora-rank 64 so they don't inflate memory and kernel cost for everyone else.
3Adapters grow to 20k.
20k × 84 MB ≈ 1.7 TB no longer fits host RAM: tier to local NVMe, shrink rank or target modules (attention-only rank 8 ≈ 7 MB), and lean harder on affinity routing.
Red flags interviewers listen for
- One dedicated deployment per tenant.
- Merges adapters into weights and calls it multi-tenant.
- No plan for base-model upgrades.
- Random routing that loads adapters on every replica.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Mechanics | Explains W + BA, batched heterogeneous kernels and their overhead. |
| Memory | Computes adapter size and tiers GPU, host and object storage. |
| Routing | Adapter affinity with bounded load; cold-start mitigation. |
| Lifecycle | Registry, eval gate and a base-upgrade migration plan. |
Sources
- LoRA: Low-Rank Adaptation of Large Language Models — Hu et al., 2021paper
- S-LoRA: Serving Thousands of Concurrent LoRA Adapters — Sheng et al., 2023paper
- Punica: Multi-Tenant LoRA Serving — Chen et al., 2023paper
- vLLM documentation — vLLM project, 2025docs