The real cost of an LLM feature: latency, tokens and the bill
Five terms make up the bill and two make up the wait. Reasoning tokens, cache writes, batch discounts and the self-host break-even, worked with stated assumptions.
A research-assistant feature moves to a reasoning model. The estimate spreadsheet says about 2 cents per answer: 6,000 input tokens and 400 visible output tokens. The first invoice works out at about 7 cents, and the median user waits 30 seconds before the first word appears. The spreadsheet counted what users see. The model was generating around 3,000 reasoning tokens per answer, billed at the output rate and produced before anything streams.
That gap is typical of how LLM cost estimates fail. The formula is simple, but it has more terms than the obvious two. Every price and speed below is an illustrative assumption (Oct 2026): $3 per million input tokens, $15 per million output tokens, 0.5 s of queueing and prefill, 100 output tokens per second. Substitute your provider's sheet and your own measurements.
The wait: two terms, one of them invisible
first visible token ≈ queue + prefill + reasoning_tokens / decode_speed
total time ≈ first visible token + visible_output_tokens / decode_speedWithout reasoning, time to first token is mostly prefill and grows with prompt length. With reasoning, the thinking term dominates: 3,000 tokens at 100 tokens/s is 30 seconds of apparent silence. Two consequences follow. Show progress (“Thinking…” plus the steps being taken) for any surface backed by a reasoning model, and set the reasoning budget per feature: reasoning_effort (low / medium / high) on OpenAI, budget_tokens on Anthropic's extended thinking. Track p95, not the mean: reasoning length varies 5–10× across prompts, so the tail is far from the median.
The bill: five terms
cost = uncached_input x P_in
+ cache_write_tokens x P_in x 1.25 (Anthropic 5-minute TTL; 2.0 for 1-hour)
+ cache_read_tokens x P_in x 0.10
+ (reasoning + visible) x P_out (thinking bills as output)
then x 0.5 if submitted through a batch API (results within 24 h)The cache multipliers are Anthropic's published structure; OpenAI caches long prefixes automatically with no write premium and discounts cached reads. Either way, prompt caching is a bet on hit rate. At a 95% hit rate a 4,000-token stable prefix costs about 16% of list price. If something in that prefix changes on every request, such as a timestamp, a request ID or a shuffled tool list, every request pays the 1.25× write premium and the cache costs more than having no cache at all.
from dataclasses import dataclass
# Illustrative list prices and speeds, Oct 2026 -- replace with your provider's sheet.
P_IN, P_OUT = 3.00 / 1e6, 15.00 / 1e6 # $ per token
CACHE_WRITE, CACHE_READ = 1.25, 0.10 # x input price (Anthropic, 5-minute TTL)
BATCH = 0.50 # batch APIs: ~50% off, results within 24 h
QUEUE_PREFILL_S, TOKENS_PER_S = 0.5, 100 # time before first token; decode speed
REQUESTS_PER_MONTH = 600_000
@dataclass
class Shape:
name: str
tokens_in: int
tokens_out: int
reasoning: int = 0
cached_prefix: int = 0
hit_rate: float = 0.0
batched: bool = False
def cost(self) -> float:
reads = self.cached_prefix * self.hit_rate
writes = self.cached_prefix - reads
fresh = self.tokens_in - self.cached_prefix
dollars = (fresh + writes * CACHE_WRITE + reads * CACHE_READ) * P_IN
dollars += (self.reasoning + self.tokens_out) * P_OUT # thinking bills as output
return dollars * (BATCH if self.batched else 1.0)
def first_visible_s(self) -> float:
return QUEUE_PREFILL_S + self.reasoning / TOKENS_PER_S
def main() -> None:
shapes = [
Shape("RAG, 8 chunks", 6_000, 300),
Shape("RAG + 4k cached prefix", 6_000, 300, cached_prefix=4_000, hit_rate=0.95),
Shape("cache busted (0% hits)", 6_000, 300, cached_prefix=4_000, hit_rate=0.0),
Shape("reasoning, 3k thinking", 6_000, 300, reasoning=3_000),
Shape("reasoning + cache", 6_000, 300, reasoning=3_000, cached_prefix=4_000, hit_rate=0.95),
Shape("RAG via batch API", 6_000, 300, batched=True),
]
print(f"{'shape':<26}{'$/request':>10}{'$/month':>10}{'first token':>13}")
for s in shapes:
print(f"{s.name:<26}{s.cost():>10.4f}{s.cost() * REQUESTS_PER_MONTH:>10,.0f}{s.first_visible_s():>12.1f}s")
if __name__ == "__main__":
main()shape $/request $/month first token
RAG, 8 chunks 0.0225 13,500 0.5s
RAG + 4k cached prefix 0.0124 7,434 0.5s
cache busted (0% hits) 0.0255 15,300 0.5s
reasoning, 3k thinking 0.0675 40,500 30.5s
reasoning + cache 0.0574 34,434 30.5s
RAG via batch API 0.0113 6,750 0.5sRead the table by line items, not totals. Reasoning tripled the per-request cost, and caching barely dents it, because caching only discounts input while thinking is output. Caching nearly halves a plain RAG request. The busted cache costs $1,800 a month more than no caching. Batch halves everything that can wait a day: nightly summaries, backfills, eval runs, document enrichment.
Retries and agents multiply the row
Every retry, validation failure and timed-out stream is billed. A 5% retry rate is a 5% surcharge on every row above, and a retry storm during a provider incident can multiply it; cap retries and use exponential backoff. An agent loop is the table's row times the number of steps, with context growing on each step, so five steps typically cost 6–10× a single call, not 5×. Agent cost needs a hard step cap and a per-task budget enforced in code.
When self-hosting breaks even
Self-hosting turns a per-token price into a fixed cost: GPUs that bill whether busy or idle, at least two replicas for availability, and engineers on call. The model below uses a 70B-class open model in FP8 on 2× H100 per replica under vLLM, an assumed 1,200 output tokens/s sustained per replica for this request shape at p95 under 3 s, a 3× peak-to-average traffic ratio, replicas planned to 60% utilisation at peak, and $10,000 a month of engineering time. Measure your own throughput before trusting any of it; continuous batching and the KV-cache efficiency of paged attention are what make the per-replica number possible.
import math
# Illustrative assumptions -- every one of these should be replaced with a measurement.
GPU_HOUR = 3.00 # $ per H100-hour, on-demand
GPUS_PER_REPLICA = 2 # 70B-class open model, FP8, vLLM
REPLICA_OUT_TOKENS_S = 1_200 # sustained output tokens/s for a 6k-in / 300-out shape at p95 < 3 s
OUT_TOKENS_PER_REQ = 300
PEAK_TO_AVERAGE = 3.0 # diurnal traffic
TARGET_UTIL = 0.6 # never plan replicas past 60% busy at peak
MIN_REPLICAS = 2 # one can fail or be upgraded
ON_CALL_MONTHLY = 10_000 # share of engineer time to run it
API_COST_PER_REQ = 0.0225 # the hosted RAG shape from the cost table
HOURS_PER_MONTH = 730
def self_host_monthly(requests_per_month: int) -> tuple[int, float]:
avg_rps = requests_per_month / (HOURS_PER_MONTH * 3600)
replica_rps = REPLICA_OUT_TOKENS_S / OUT_TOKENS_PER_REQ * TARGET_UTIL
replicas = max(MIN_REPLICAS, math.ceil(avg_rps * PEAK_TO_AVERAGE / replica_rps))
gpu_cost = replicas * GPUS_PER_REPLICA * GPU_HOUR * HOURS_PER_MONTH
return replicas, gpu_cost + ON_CALL_MONTHLY
def main() -> None:
print(f"{'requests/month':>15}{'API $':>10}{'replicas':>10}{'self-host $':>13}")
for volume in (600_000, 2_000_000, 10_000_000, 50_000_000):
replicas, own = self_host_monthly(volume)
print(f"{volume:>15,}{volume * API_COST_PER_REQ:>10,.0f}{replicas:>10}{own:>13,.0f}")
if __name__ == "__main__":
main() requests/month API $ replicas self-host $
600,000 13,500 2 18,760
2,000,000 45,000 2 18,760
10,000,000 225,000 5 31,900
50,000,000 1,125,000 24 115,120Under these assumptions the break-even sits around 830,000 requests a month, set almost entirely by the two-replica floor plus on-call. Below it, the API is cheaper and someone else holds the pager. Above it, the gap widens fast. Two conditions gate the decision before the arithmetic does: the open model must pass the same golden set as the hosted one, and the team must accept owning upgrades, capacity and incidents. If either fails, the volume does not matter.
Levers, in order of effort
| Lever | Effort | Typical effect |
|---|---|---|
| Move volatile text out of the cached prefix | Hours | Hit rate 0% → 90%+; −45% cost |
| Cap visible output and reasoning effort | Hours | −20–60% on reasoning features |
| Rerank, send 4 chunks instead of 8 | Days | −25–35% input tokens |
| Batch everything that can wait 24 h | Days | −50% on those jobs |
| Route easy requests to a small model | Weeks | −50–80% on the routed share |
| Self-host | Months | Wins only past break-even |
What goes on the dashboard
- Cost per successful request by feature and tenant. Total spend rising with usage is good news; cost per success rising is a regression.
- Reasoning tokens, p50 and p95, per feature. Alert when p95 doubles week on week; a prompt change can do that silently.
- Cache hit rate. Alert below 80% where you expect 95%. The usual cause is a deploy that put a timestamp or request ID into the system prompt.
- Retry rate and batch share. Retries above 3%, or interactive traffic doing work that could have been batched.
Log tokens by type (uncached, cache-write, cache-read, reasoning, visible output) for every call, centrally, in an LLM gateway. Provider invoices arrive monthly and in aggregate; a regression found on the invoice has already cost a month.
Key takeaways
- Reasoning tokens are the line item estimates miss: invisible in the UI, billed at the output rate, and paid for in latency before the first word.
- Caching is a hit-rate bet. It cuts a stable prefix to about a sixth of list price, and costs more than no cache when one volatile field busts it.
- Batch is the only lever that halves cost without touching quality; anything that can wait a day should be using it.
- Self-hosting is a fixed-cost bet with a volume threshold and a quality gate; below either, the API is cheaper once on-call is counted.
Sources
- Prompt caching — Anthropic, 2024docs
- Batch processing (Message Batches API) — Anthropic, 2024docs
- Reasoning models — OpenAI, 2024docs
- Batch API — OpenAI, 2024docs
- Efficient Memory Management for Large Language Model Serving with PagedAttention — Kwon et al., 2023paper