How do you build a cost model for an LLM feature before launch?
- 1 min — unit: cost per successful task
- 3 min — per-call formula incl. cache and reasoning, per-task distribution
- 2 min — levers ranked
- 1 min — attribution, budgets, rollout
Model cost per successful user task, then multiply by volume. Per call: input_tokens × input_price + cache_write_tokens × 1.25 × input_price + cache_read_tokens × 0.1 × input_price + (output + reasoning tokens) × output_price. Per task: sum over calls, plus retrieval, reranking, tool and judge calls, plus retries and fallbacks. For agents, calls per task is a distribution, and because each step re-sends the growing context, input tokens grow quadratically with step count.
A research agent demo averaged 4 calls/task. Each call sends 3k base tokens plus 1.5k per prior tool result, outputs 400 tokens. 4 steps: 21k input + 1.6k output = $0.087/task. Production p95 is 22 steps: 412k input + 8.8k output = $1.37/task — 16× the demo for 5.5× the steps. Modelled as a distribution (p50 5, p90 14, p95 22) with a step cap of 12 and tool-result summarisation, mean cost lands at $0.21.
- Unit: cost per successful task; failures and retries count against it.
- Levers in order of typical size: shorten and cache context, route easy steps to a smaller model, cap output and thinking tokens, summarise tool results, batch APIs (~50% off) for offline work.
- Forecast p50/p95/p99 per task and per tenant, not one mean; heavy tenants and runaway loops dominate.
- Org scope: finance and product own the price; the platform team owns per-feature, per-tenant cost attribution in the gateway and budget enforcement. Agent cost needs per-step attribution in traces. Rollout: beta with per-tenant budgets and alerts at 50/80/100%.
- Where it breaks: pricing a plan on average usage. 5% of tenants can drive 40% of spend.
assume $3 / M input and $15 / M output tokens (illustrative frontier list prices, Oct 2026); cache write 1.25× input for a 5-minute TTL, cache read 0.1×. Single call 3,000 in / 400 out: $0.009 + $0.006 = $0.015 → $15 per 1k. With 2,500 of the 3,000 as a cached prefix (reads): 2,500 × $0.30/M + 500 × $3/M + $0.006 = $0.00825 → $8.25 per 1k, plus ~one write per 5 min (2,500 × $3.75/M ≈ $0.0094). Add 1,500 reasoning tokens: +$0.0225 → $30.75 per 1k cached. Agent of k steps with +1.5k tokens per step: input = 3k·k + 1.5k·k(k−1)/2.
| Setup | $ / request | $ per 1k |
|---|---|---|
| 3k in / 400 out | 0.0150 | 15.00 |
| + 2.5k cached prefix | 0.0083 | 8.25 |
| + 1.5k reasoning | 0.0308 | 30.75 |
| Agent p95 (22 steps) | 1.37 | 1,370 |
What the answer sounds like at each level
Multiplies tokens by price per call and by volume.
Models a distribution of calls per task, includes cache, retries and judge calls, and computes cost per successful task.
Owns cost attribution and budgets as platform features, prices plans from p95 tenants, and ties levers to measured quality.
Follow-ups the interviewer will push
1Now 10× traffic. What changes in the model?
Volume discounts and committed-use pricing appear; rate limits (tokens/minute) may force multi-region or multi-provider routing; batch APIs become worth it for offline steps.
2The provider changes prices mid-quarter.
Prices are config in the gateway's rate card, not constants in code; re-run the forecast from stored token counts per feature in minutes.
3How do caching costs work out with a 5-minute TTL?
Each cache write costs 1.25× input; reads 0.1×. Caching pays when a prefix is reused ≥2 times within the TTL; below that, writes cost more than they save.
Red flags interviewers listen for
- Uses a single mean calls-per-task from a demo.
- Forgets retries, judges and reasoning tokens.
- No per-tenant budgets or alerts.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Formula | Complete per-call formula including cache write/read and reasoning tokens. |
| Distribution | Shows quadratic context growth and p95 tasks. |
| Org | Attribution, budgets and pricing from tails. |
The demo cost and the real cost
Illustrative exampleOur research-agent feature (beta, 120 tenants, ~38k tasks/week) had been priced from a demo averaging 4 calls per task at $0.09. Two weeks into beta, spend was running at $0.41 per task — 4.6× forecast — and the GA price was already announced.
I owned the feature's economics as staff engineer on a 6-person team (4 engineers, 1 PM, 1 data scientist). I had to produce an accurate model and bring mean cost under $0.15 without dropping task success below 85%. Our PM argued for simply raising the price; sales opposed that because 30 tenants were mid-negotiation.
I exported 9,400 task traces from the gateway and plotted calls and tokens per task: p50 5 calls, p95 22, and the top 6% of tasks consumed 41% of tokens because re-sent context grew every step. I wrote the per-task cost model and the budget design; the data scientist built the 300-task golden set; two engineers implemented stop conditions (12-step cap, 150k-token budget) and tool-result summarisation; one moved the stable 5k-token system prompt to prompt caching and routed extraction steps to a smaller model. I persuaded the PM by showing the price rise needed to cover p95 tenants would be 3×, which sales confirmed would lose the 30 deals.
On the golden set (n=300) task success went from 87% to 86%; in production mean cost fell to $0.13 and p95 to $0.48 per task over the next 4 weeks (n≈150k tasks). Per-tenant budgets with 80% alerts went into the launch checklist, and the forecast template now models a step distribution. What I'd do differently: instrument per-step cost before beta — we lost two weeks reconstructing it from logs.
Sources
- Prompt caching — Anthropic, 2025docs
- Building effective agents — Anthropic, 2024blog