Cost per query
Total spend to answer one request: input, cache writes and reads, output and reasoning tokens, retrieval, reranking, tool calls and retries.
Cost per query is how much it costs, on average and at the expensive tail, to answer one user request. Most of it is Token charges: providers bill input tokens (what you send, including the system prompt and retrieved documents) and output tokens (what the model writes, including hidden reasoning for Reasoning models & extended thinking) at different rates, with discounts for cached input (Prompt caching).
Add smaller costs like embeddings, a Reranker, vector database queries and tool calls, then multiply by retries and agent steps.
A taxi fare: a fixed charge to start (the system prompt), a per-kilometre rate (input tokens), a higher per-minute rate while waiting in traffic (output and reasoning tokens), and a discount for regular routes (caching).
Estimating cost from visible output only. Reasoning tokens are billed as output and can be 3–10× the visible answer; teams that switch on extended thinking without budgeting it see the bill jump with no change in what users see.
Per-call cost = uncached input × p_in + cache writes × p_in × 1.25 (5-minute TTL; 2× for 1-hour) + cache reads × p_in × 0.1 + (visible output + reasoning tokens) × p_out, using Anthropic's multipliers; other providers have similar but not identical discounts. Per-query cost = Σ calls (an agent may make 5–30) + retrieval + reranking + retries. Output is typically 4–5× the input price, so reasoning tokens often dominate: a model thinking for 2,000 tokens to write 300 visible ones spends most of its output budget where the user cannot see it.
Levers, in order of usual impact: cap or disable extended thinking where evals show no gain; cache stable prefixes (system prompt, tool schemas, few-shots) with cache breakpoints placed before anything variable; cut retrieved context with a reranker (send top 5, not top 20); route easy intents to a smaller model (Intent classification & routing); batch offline work (50% off); and set max_tokens. Report p50 and p95 cost per query, not just the mean: long documents and agent loops make the distribution heavy-tailed. Track cost per successful task, since a cheaper configuration that needs a retry is not cheaper.
Gateway logs per call: input, cache-write, cache-read, output and reasoning tokens, model, and computed cost; dashboards of p50/p95 cost per query and per successful task by route; prompt caching on stable prefixes; thinking budgets set per route from evals; max_tokens set; reranker to top 5; Batch API for offline jobs.
Support RAG answer, Sonnet 4.6-class list prices (assumptions, Oct 2026): $3 / M input, $15 / M output, cache write 1.25×, cache read 0.1×. Request: 4k-token system prompt + tool schemas (cacheable), 6k tokens of retrieved context, 200-token question, 1,500 reasoning tokens, 400-token answer. No caching: 10.2k × $3 / M + 1.9k × $15 / M = $0.0306 + $0.0285 = $0.059. With the 4k prefix cached (read): $0.0012 + $0.0186 + $0.0285 = $0.048. With caching and thinking off (evals show no loss on this route): $0.026. At 1M queries/month: $59k → $48k → $26k.
Say: "I break cost per query into cached input, uncached input, reasoning and visible output, because the fix is different for each; then I check p95, not just the mean, and cost per successful task." Expect: "how would you halve it?": turn off reasoning where evals allow, cache the prefix, rerank to shrink context, route easy intents to a smaller model.
Numbers worth knowing
- Cache multipliers (Anthropic)
- write 1.25× (5 min) or 2× (1 h); read 0.1× base inputOct 2026 pricing docs
- Batch discount
- 50% on input and outputasynchronous Message Batches API
- Worked example
- $0.059 → $0.048 → $0.026 per queryno cache → cached prefix → cached prefix + thinking off; assumptions in example
- Cache break-even
- ≥2 requests per prefix within the TTLone write (1.25×) + one read (0.1×) = 1.35× vs 2× uncached
Try the numbers
Illustrative list prices — check your provider. Assumes cache reads at 0.1× input and misses re-writing the prefix at 1.25× (a 5-minute TTL); batch APIs typically take ~50% off on top.
How to decide
| When | Do this |
|---|---|
| Reasoning tokens exceed 50% of output spend | Eval the route with thinking off or a lower budget; keep it only where scores drop beyond the CI. |
| A stable prefix of ≥1k tokens is reused at least twice per 5 minutes | Cache it; at lower reuse a 5-minute write costs more than it saves. |
| p95 cost is more than 5× p50 | Find the tail (long documents, agent loops) and cap it with context limits or per-task budgets. |
Cost maths
Per query = Σ_calls [uncached_in × p_in + write × 1.25 p_in + read × 0.1 p_in + (out + reasoning) × p_out] + retrieval + rerank. With $3 / $15 per M: 4k cached prefix read = $0.0012 vs $0.012 uncached; but at one request every 6 minutes each request re-writes the cache at $0.015, more than not caching. Embedding the query (~$0.00001) and a self-hosted reranker over 50 candidates (~$0.0002) are rounding errors next to 1,500 reasoning tokens ($0.0225).
Operating it
- Monthly LLM spend passes ~$10k: add per-route cost dashboards and a weekly cost review.
- One route exceeds 40% of spend: run a dedicated cost-reduction eval (model routing, thinking budget, context size).
- Agent tasks' p95 cost exceeds 10× p50: add per-task token and step budgets.
- Cost per query p50 and p95 per route: alert on +25% week over week.
- Cache hit rate (cache-read tokens / cacheable prefix tokens): alert below 70% on routes designed for caching.
- Reasoning-to-visible output ratio: alert when it doubles after any model or prompt change.
When it goes wrong
Monthly spend on an internal code-search assistant tripled after a model upgrade, while traffic was flat.
Finance flagged the invoice; the gateway dashboard showed output tokens per query up 4.2× with visible answer length unchanged.
The new model enabled extended thinking by default on the route; reasoning tokens (billed as output) averaged 2,100 per query, and the cached prefix had been invalidated because a timestamp was inserted at the top of the system prompt.
Set a thinking budget per route from evals (off for lookup questions, 1k for cross-repo questions), moved the timestamp after the cache breakpoint, and added alerts on output-to-visible-token ratio and cache hit rate.
Related terms
Where it shows up
Sources
- Pricing — Anthropic, 2026docs
- Prompt caching — Anthropic, 2026docs
Check yourself
Three questions. Every option is explained once you answer.