Open-weights model or a hosted API — how do you decide?
- 1 min — the two options and the managed middle
- 3 min — break-even math with utilisation
- 2 min — non-cost constraints: residency, pinning, licence
- 1 min — gateway-based rollout
Hosted API: frontier capability, no infra, fast iteration, pay per token; you accept provider deprecation schedules, data-handling terms, regions and rate limits. Open weights (Llama, Qwen, Mistral, DeepSeek, gpt-oss): control of version, data residency, fine-tuning and serving stack; you own GPUs, inference tuning, autoscaling, safety tuning and on-call. Middle option: open weights on a managed host (Bedrock, Vertex, Together, Fireworks) — version control without running GPUs. Downloadable weights are not automatically open source; read the licence.
A healthcare document pipeline must keep data in-region (EU) and runs 40M narrow extractions/month. A fine-tuned 8B open-weights model self-hosted in-region fits: steady volume, high utilisation, pinned version for audit. The same company's internal assistant stays on a hosted frontier API: volume is 200k requests/month, quality matters more, and a GPU pool would sit idle 70% of the day.
- Compare total cost, not token price: GPU-hours at real utilisation, engineers, on-call, evals. Self-hosting wins at sustained high utilisation or hard constraints (residency, version pinning, air-gap).
- Break-even (illustrative): one H100 at $2.50/hr serving an 8B model at ~2,500 output tok/s at full batch; at 50% average utilisation → 4.5M output tok/hr → $0.56 / M output. A hosted small model at $0.40–0.60 / M output is equal or cheaper before you add a 0.5 FTE on-call (~$8k/month ≈ 4.4 H100s running 24/7).
- Put both behind an LLM gateway so routing, fallback and cost tracking don't depend on the choice.
- Org scope: the decision is the platform team's with finance and security; product teams consume an endpoint. Roll out self-hosting route by route via gateway weights, hosted as fallback.
- Where it breaks: bursty traffic on under-provisioned self-hosting — idle GPUs at night, queueing at peak, and cold starts of 2–5 minutes to load 70B weights.
Assume H100 $2.50/hr, 8B model in vLLM, 2,500 output tok/s at full batch. 100% utilisation: 9M tok/hr → $0.28 / M. 50%: $0.56 / M. 25%: $1.11 / M. 40M extractions × 300 output tokens = 12B output tok/month ÷ 2,500 tok/s ≈ 1,330 GPU-hours → 2 H100s at 90% utilisation, 3 with N+1 redundancy ≈ $5.5k/month (prefill of the inputs adds headroom needs) vs ~$6k output-only at a hosted $0.50 / M. Near parity before a 0.5 FTE on-call, so the case rests on EU residency and version pinning, not price.
What the answer sounds like at each level
Lists control vs convenience trade-offs.
Computes $/M tokens for self-hosting at a stated utilisation and compares to API prices.
Decides with TCO including people, residency and deprecation risk; uses managed open-weights as a middle path; rolls out via gateway with fallback.
Follow-ups the interviewer will push
1Traffic is spiky: 10× at business hours. Does self-hosting still win?
Average utilisation falls to ~25%, doubling effective $/M. Keep a self-hosted baseline sized to the trough and burst to a hosted endpoint for the same open model.
2Now the hosted provider deprecates your model in 6 months.
Hosted: run the upgrade playbook (eval gate, shadow, canary). Self-hosted open weights: you may keep the pinned version indefinitely — one reason regulated workloads choose it.
Red flags interviewers listen for
- Compares GPU hourly price to token price without utilisation.
- Ignores on-call and engineering cost.
- Assumes 'open weights' means unrestricted licence.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Economics | Break-even with explicit utilisation and people cost. |
| Constraints | Residency, deprecation, licence, burstiness. |
| Architecture | Gateway abstraction with fallback and staged migration. |
Sources
- vLLM documentation — vLLM project, 2026docs
- Efficient Memory Management for Large Language Model Serving with PagedAttention — Kwon et al., 2023paper