Self-host or API? A break-even memo with the maths shown

“Should we self-host?” usually arrives as a GPU price compared with an API price per million tokens. That comparison is wrong in three ways: it ignores utilisation, it ignores the people who keep the cluster alive, and it compares against list API prices instead of what you actually pay after caching and batch discounts. This memo fixes all three and shows where the lines cross.

Decision

Default to the API. Revisit self-hosting when sustained volume on a single model passes the break-even below, demand is steady enough to keep GPUs above 50% utilisation, and you have, or will fund, at least 1.5 engineers who own serving. Self-host earlier only for a non-cost reason that survives scrutiny: data that contractually cannot leave your network, a fine-tuned model no provider will host, or latency your SLO cannot get from a shared endpoint.

Assumptions (all illustrative, October 2026)

Input	Value	Why this value
GPU price	$2.50 / H100-hour	Committed cloud; on-demand is higher
Replica	2× H100, 70B-class, FP8	Weights ≈70 GB fit with KV headroom
Peak throughput	6,000 tok/s per replica	Blended in+out at full batch, vLLM
Utilisation	50% average of peak	Diurnal traffic plus burst headroom
Replica floor	2 always on	One can fail or be upgraded
Ops	1.5 FTE × $20k/month	Serving, upgrades, on-call, evals
Token mix	75% input / 25% output	RAG and agent workloads skew to input
Frontier API	$3 / M in, $15 / M out	Illustrative list prices
Caching	50% reads at 0.1×, 5% writes at 1.25×	Stable system prompt + docs
Batch	30% of traffic at 50% off	Evals, backfills, nightly jobs
Hosted open-weights	$0.90 / M flat	Same 70B model, provider-hosted

Change any row and rerun the calculator. The conclusion is most sensitive to utilisation, ops headcount and GPU price.

Two API comparisons matter, and confusing them is the usual mistake. A frontier model at $3/$15 is a different product from a 70B Open-weights vs closed models model: if the open model does not pass your eval gate, its cost is irrelevant. The honest like-for-like comparison is your self-hosted 70B against the same 70B served by a provider.

The formulas

Replica capacity (tokens/month) = peak tok/s × utilisation × 3,600 × 730 = 6,000 × 0.5 × 3,600 × 730 ≈ 7.9 B tokens/month.
Self-host cost = max(floor, ⌈volume ÷ capacity⌉) × GPUs × $/GPU-hour × 730 + ops FTE × $/FTE-month. With the floor of 2: 2 × 2 × $2.50 × 730 = $7,300 of GPUs plus $30,000 of people = $37,300/month before serving a single token.
Marginal self-host cost = $3,650 per replica-month ÷ 7,884 M tokens ≈ $0.46 / M tokens.
Blended API price = [input share × input price × (uncached + 0.1 × cache reads + 1.25 × cache writes) + output share × output price] × (1 − batch share × 0.5). Frontier: [0.75 × $3 × 0.6125 + 0.25 × $15] × 0.85 ≈ $4.26 / M.
Break-even volume is where self-host cost = blended API price × volume.

Self-host 70B (2×H100 replicas)Frontier API, cached + batchedSame 70B, hosted API

Computed by the calculator below. Self-hosting starts as a $37k fixed cost and only beats the same model hosted elsewhere near 70 B tokens/month.

The calculator

Save as breakeven.py and run python3 breakeven.py. Standard library only. Every constant at the top is an assumption from the table above.

example.pypython74 lines

"""Self-host vs API break-even calculator. Every number below is an assumption: replace it with your own."""
import math

HOURS_PER_MONTH = 730

# ---- Self-host assumptions (illustrative, Oct 2026) ----
GPU_PER_HOUR = 2.50          # $ per H100-hour, committed cloud pricing
GPUS_PER_REPLICA = 2         # a 70B-class model in FP8 on 2x H100
PEAK_TOKENS_PER_S = 6000     # blended input+output tokens/s per replica at full batch
UTILISATION = 0.50           # average load / peak capacity (diurnal traffic, headroom)
MIN_REPLICAS = 2             # always-on floor for availability
OPS_FTE = 1.5                # engineers owning serving, upgrades, on-call
FTE_PER_MONTH = 20_000       # $ fully loaded per engineer-month

# ---- API assumptions (illustrative list prices, Oct 2026) ----
INPUT_SHARE = 0.75           # 3:1 input:output tokens
CACHE_READ_MULT = 0.10       # cache reads billed at 0.1x base input
CACHE_WRITE_MULT = 1.25      # cache writes billed at 1.25x (5-minute TTL)
BATCH_DISCOUNT = 0.50


def api_price_per_m(input_price, output_price, cache_hit=0.0, cache_write=0.0, batch_share=0.0):
    """Blended $ per million tokens after caching and batch discounts.

    cache_hit / cache_write: share of input tokens read from / written to the cache.
    batch_share: share of all traffic sent through the batch API.
    """
    plain = 1 - cache_hit - cache_write
    effective_input = input_price * (plain + cache_hit * CACHE_READ_MULT + cache_write * CACHE_WRITE_MULT)
    blended = INPUT_SHARE * effective_input + (1 - INPUT_SHARE) * output_price
    return blended * (1 - batch_share * BATCH_DISCOUNT)


def replica_capacity_m():
    """Million tokens one replica serves per month at the stated utilisation."""
    return PEAK_TOKENS_PER_S * UTILISATION * 3600 * HOURS_PER_MONTH / 1e6


def self_host_monthly(volume_m):
    replicas = max(MIN_REPLICAS, math.ceil(volume_m / replica_capacity_m()))
    gpu = replicas * GPUS_PER_REPLICA * GPU_PER_HOUR * HOURS_PER_MONTH
    return gpu + OPS_FTE * FTE_PER_MONTH, replicas


def break_even(api_per_m, hi=1e7):
    """Smallest monthly volume (M tokens) at which self-hosting is cheaper."""
    v = 1.0
    while v < hi:
        if self_host_monthly(v)[0] <= api_per_m * v:
            return v
        v *= 1.01
    return None


if __name__ == "__main__":
    apis = {
        # frontier-class model: 50% of input from cache, 5% cache writes, 30% of traffic batched
        "frontier API ($3/$15)": api_price_per_m(3.00, 15.00, cache_hit=0.50, cache_write=0.05, batch_share=0.30),
        # the same 70B open-weights model, hosted by a provider: flat price, no cache or batch
        "hosted open-weights API ($0.90/$0.90)": api_price_per_m(0.90, 0.90),
    }
    cap = replica_capacity_m()
    print(f"replica capacity: {cap:,.0f} M tokens/month at {UTILISATION:.0%} utilisation")
    print(f"self-host marginal cost: {GPUS_PER_REPLICA * GPU_PER_HOUR * HOURS_PER_MONTH / cap:.2f} USD / M tokens")
    for name, price in apis.items():
        be = break_even(price)
        where = "never, at these assumptions" if be is None else f"~{be / 1000:,.1f} B tokens/month"
        print(f"{name}: blended {price:.2f} USD / M tokens; break-even {where}")
    print()
    print(f"{'B tok/mo':>8} {'self-host':>10} {'replicas':>8} {'frontier':>10} {'open API':>10}   (USD/month)")
    for b in [1, 5, 10, 25, 50, 100, 150, 200]:
        cost, replicas = self_host_monthly(b * 1000)
        row = [api * b * 1000 for api in apis.values()]
        print(f"{b:>8} {cost / 1000:>9.1f}k {replicas:>8} {row[0] / 1000:>9.1f}k {row[1] / 1000:>9.1f}k")

Output:

texttext14 lines

replica capacity: 7,884 M tokens/month at 50% utilisation
self-host marginal cost: 0.46 USD / M tokens
frontier API ($3/$15): blended 4.26 USD / M tokens; break-even ~8.8 B tokens/month
hosted open-weights API ($0.90/$0.90): blended 0.90 USD / M tokens; break-even ~69.9 B tokens/month

B tok/mo  self-host replicas   frontier   open API   (USD/month)
       1      37.3k        2       4.3k       0.9k
       5      37.3k        2      21.3k       4.5k
      10      37.3k        2      42.6k       9.0k
      25      44.6k        4     106.6k      22.5k
      50      55.5k        7     213.2k      45.0k
     100      77.5k       13     426.3k      90.0k
     150     103.0k       20     639.5k     135.0k
     200     124.9k       26     852.7k     180.0k

Reading the result

Against a frontier API, self-hosting looks cheaper above about 8.8 B tokens/month (≈290 M tokens/day, ≈3,400 tok/s average). That number is real but usually irrelevant, because it compares two different models. Use it only if your evals show the open model is good enough.
Against the same model hosted by a provider, break-even is about 70 B tokens/month (≈2.3 B/day, ≈27,000 tok/s average, nine replicas). Providers win below that because they run their GPUs hotter than you will and spread the on-call across thousands of customers.
The fixed cost is mostly people. At low volume, 80% of the self-host bill is the 1.5 FTE. Cutting GPU price does little; owning serving with half an engineer is the real lever, and also the one that fails first.

Sensitivity: what moves break-even

Change (one at a time)	Marginal $/M	vs frontier	vs same model hosted
Base case	0.46	8.8 B/mo	70 B/mo
Utilisation 25%	0.93	9.6 B/mo	never
Utilisation 70%	0.33	8.8 B/mo	54 B/mo
Ops 0.5 FTE	0.46	4.1 B/mo	23 B/mo
Ops 3 FTE	0.46	16.7 B/mo	140 B/mo
GPU $4.00/hour	0.74	9.8 B/mo	197 B/mo
Peak 3,000 tok/s	0.93	9.6 B/mo	never

Rerun of the calculator with one constant changed. “Never” means marginal self-host cost exceeds the hosted price.

Two rows deserve attention. At 25% utilisation, or at half the assumed throughput, your marginal cost per token is above the hosted price and self-hosting never breaks even against the same model. Both are common: spiky traffic, long contexts that eat KV cache and shrink batch size, or a serving stack left at default settings. Measure your real tokens/s at your real context lengths with vLLM or SGLang before trusting the 6,000 figure. FP8 or INT4 quantisation and prefix caching move it; nothing moves it as much as keeping the batch full.

Consequences and how to stay reversible

Put every call behind an LLM gateway from day one, so routing a model to your own endpoint is a config change, not a migration.
Track cost per query and tokens per model per month. When one model's steady volume crosses half the break-even, start a measured pilot: one replica pair, shadow traffic, real tok/s at real context lengths.
Keep the API as overflow and fallback even after self-hosting. Hybrid (self-host the base load, burst to the API) beats either pure option for spiky traffic.

Key takeaways

Compare like with like: your self-hosted model against the same model hosted by a provider. Frontier-API comparisons only matter if the open model passes your evals.
Self-hosting starts as a fixed cost (here $37k/month, 80% of it people) and wins only on marginal cost: ≈$0.46/M at 50% utilisation versus $0.90/M hosted, crossing near 70 B tokens/month.
Utilisation and real throughput decide everything. At 25% utilisation, or half the throughput, self-hosting never breaks even against the same model hosted.
Use blended API prices: cache reads at 0.1×, writes at 1.25×, batch at 50% off. List prices overstate the API bill.
Stay reversible: route through a gateway, measure Cost per query, pilot with shadow traffic before committing hardware.

Sources

Efficient Memory Management for Large Language Model Serving with PagedAttention — Kwon et al., 2023paper
vLLM documentation — vLLM project, 2026docs
Prompt caching — Anthropic, 2026docs
Batch API — OpenAI, 2026docs

#Decision

#Assumptions (all illustrative, October 2026)

#The formulas

#The calculator

#Reading the result

#Sensitivity: what moves break-even

#Consequences and how to stay reversible