Estimate $/1M tokens for self-hosting a 70B open-weights model versus calling an API, and the monthly volume at which self-hosting breaks even.
- 1 min — define the alternatives (frontier API, hosted open-weights, self-hosted)
- 3 min — fixed floor and marginal cost for self-hosting, with utilisation
- 2 min — blended API price, break-even volume for each comparison
- 2 min — non-price factors, quality gate, reversibility via gateway
API cost is linear in tokens. Self-hosting is a step function: a fixed floor (at least two replicas for availability, plus the people who run them) and then near-linear GPU cost above it. Break-even volume = monthly fixed cost ÷ (API price per token − self-hosted marginal price per token). Utilisation is the hidden variable: a GPU at 30% average load costs 3× per token what it does at 90%.
An insurance-claims triage pipeline processing ~4B tokens/month (80% input), currently on a frontier API. Finance asks whether moving to a self-hosted 70B saves money. The honest answer depends on which API you compare against and on whether the 70B passes your golden set at all.
Price is rarely the deciding factor at this volume. Against hosted open-weights providers (same model, someone else's GPUs), self-hosting almost never wins on price below ~20B tokens/month; against frontier-API prices it can, but only if quality holds on your evals. The real reasons to self-host are data residency, latency control, fine-tuned variants, and freedom from deprecation schedules. Build-vs-buy: route through an LLM gateway so the choice is reversible per use case, and re-run the comparison every quarter because API prices fall faster than GPU leases.
Assumptions (illustrative, Oct 2026): H100 at $2.50/GPU-hour; 70B FP8 on TP2 replicas sustaining ~3,000 output tok/s and ~15,000 prompt tok/s; 40% average utilisation; minimum 2 replicas for availability; 0.5 FTE to operate at $10k/month loaded; traffic 4:1 input:output. Frontier API at $3 / M input, $15 / M output; hosted open-weights 70B at ~$0.80 / M blended.
- Fixed floor: 2 replicas × 2 GPUs × $2.50 × 730 h = $7,300 + $10,000 people = $17,300/month.
- Self-hosted marginal: a replica costs $5/hour and at 40% utilisation serves ~0.4 × 3,600 × (3,000 + 12,000 matched input) ≈ 21.6M blended tokens/hour → ~$0.23 / M blended above the floor.
- Frontier API blended: (4 × $3 + 1 × $15) ÷ 5 = $5.40 / M. Break-even = 17,300 ÷ (5.40 − 0.23) ≈ 3.3B tokens/month (~1,300 tok/s average) — two replicas carry it.
- Hosted open-weights: 17,300 ÷ (0.80 − 0.23) ≈ 30B tokens/month — above which you need ~6+ replicas anyway.
- This workload (4B/month): frontier API ≈ $21.6k; self-hosted ≈ $17.3k + 4,000 × $0.23 ≈ $18.2k — a 16% saving if the 70B matches quality; vs hosted open-weights at $3.2k, self-hosting loses by 5×.
- Discounts change the answer: batch APIs (~50% off) and prompt-cache reads (0.1× input) can halve the frontier bill; recompute with your cache-hit rate.
What the answer sounds like at each level
Computes GPU $/hour ÷ tokens/hour and compares it to an API list price.
Adds the availability floor, utilisation, input/output price asymmetry and engineering headcount, and computes a break-even volume.
Separates the comparison by alternative (hosted open-weights vs frontier), makes quality parity a precondition, names the non-price reasons, and keeps the decision reversible behind a gateway.
Follow-ups the interviewer will push
1Now 60% of input tokens are a cacheable shared prefix.
On a frontier API, cache reads at 0.1× cut input cost ~54% (writes cost 1.25× for 5-minute TTL); blended drops to ~$3.30/M and break-even rises to ~5.5B tokens/month. Self-hosting gets the same benefit from prefix caching, but it shows up as fewer GPUs, not lower price per token.
2Now a regulated tenant requires in-region processing in the EU.
That's a non-price constraint that may decide it regardless: check whether your API provider offers EU data residency first. If not, self-host in-region for that tenant only, routed by the gateway, and keep everyone else on the cheaper option.
3Now the API provider is down for two hours.
Self-hosting a fallback 70B gives you degraded-mode continuity; whether that's worth $7,300/month of idle floor is an availability-SLO question. A second API provider is usually the cheaper fallback.
Red flags interviewers listen for
- Compares GPU price at 100% utilisation with API price — self-hosted fleets average 30–50%.
- Ignores the people cost and the two-replica availability floor.
- Treats the 70B and the frontier model as interchangeable without an eval.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Cost model | Fixed floor + marginal, with utilisation explicit. |
| Break-even | Computes a volume for each alternative and places the actual workload on it. |
| Judgement | Makes quality parity a precondition and names residency/deprecation as the real drivers. |
| Org | Proposes a gateway and a quarterly re-check, not a one-time migration. |
Sources
- Claude pricing — Anthropic, 2026docs
- Efficient Memory Management for Large Language Model Serving with PagedAttention — Kwon et al., 2023paper
- vLLM documentation — vLLM project, 2026docs