How do you measure and control cost per query for an LLM product?
- 2 min — what goes into cost per query
- 4 min — worked cost per query with and without caching
- 4 min — levers ranked, each with its quality check
- 3 min — budgets, ceilings, pricing on the tail, ownership
Cost per query is the full cost of serving one user request: input and output tokens for every model call it triggers (retries, agent loop steps, judge and guardrail calls), reasoning tokens billed as output, embeddings, retrieval and reranking. Output costs ~5× input at typical list prices, cache reads ~0.1× input, cache writes 1.25× (5-minute TTL) or 2× (1-hour) on Anthropic, batch ~50% off. Measure the distribution, not the mean: in agentic products the p99 request often costs 10× the median.
E-commerce catalog assistant, 1M queries/month. Instrument every call through the gateway with feature and tenant tags. Levers in order of usual payoff: prompt caching on the 3k-token stable system prompt and tool definitions; capping agent steps with stop conditions; routing simple lookups to a smaller model via an intent classifier; trimming retrieved context from 8 chunks to the 4 a reranker keeps; capping max_tokens; and moving offline jobs (catalog enrichment, nightly summaries) to batch APIs.
Every lever has a quality cost, so each one ships through the eval gate — cheaper routing that drops quality on the 'size and fit' slice is not a saving. Budgets: per tenant and per feature monthly budgets with alerts at 80%, plus a hard per-request ceiling (e.g. $0.50 or 25 calls) so a looping agent can't burn unbounded spend (agent cost). Unit economics must use the tail: price plans on p90 usage per tenant, not on mean tokens. Ownership: the gateway team owns metering and the ledger; finance gets showback dashboards by feature; feature teams own their cost-per-successful-task target. Where it breaks: 'unlimited' plan tiers on features whose marginal cost isn't small, cache hit rates silently collapsing after someone adds a timestamp to the system prompt, and judge/guardrail spend nobody attributed.
Assume $3 / $15 per M tokens (illustrative, Oct 2026). Per call 4k in + 350 out = $0.012 + $0.00525 ≈ $0.017. Average 2.3 calls per query ≈ $0.040; plus embeddings, rerank and guardrail ≈ $0.0015 → ~$0.041/query, ~$41k/month at 1M queries. p99 query at 25 calls ≈ $0.43. Caching 3k of the 4k input at 0.1× read: input per call = $0.003 + $0.0009 ≈ $0.0039 instead of $0.012 → saves ~$0.0186/query ≈ 45%, ~$18k/month, minus cache-write premiums on misses.
What the answer sounds like at each level
Track token usage and costs, and use cheaper models or caching where you can.
Meter every call by feature and tenant, look at the cost distribution, apply caching, step caps, routing and context trimming, each validated against the eval set.
I own cost as a product metric: cost per successful task per feature, budgets and hard per-request ceilings, pricing on tail usage, and a ledger reconciled to invoices that finance trusts.
Follow-ups the interviewer will push
1Cache hit rate dropped from 70% to 5% overnight. Why?
Something dynamic entered the cached prefix — a timestamp, user name or reordered tool list. Caching matches exact prefixes, so put stable content first, dynamic content last, and alert on hit rate below 50%.
2One tenant is 30% of spend on 5% of queries. What do you do?
Check whether they're on agentic workflows with long loops, then apply per-tenant budgets and step caps, and talk to sales about a usage-based tier — don't silently degrade them.
Red flags interviewers listen for
- Computes unit economics from average tokens per query.
- Forgets retries, judge calls, guardrails or reasoning tokens in the cost.
- Applies cost cuts without re-running evals.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Completeness | Counts every call type, reasoning tokens, cache write premiums and batch discounts. |
| Quantification | Works a per-query cost and the saving from at least one lever. |
| Governance | Budgets, hard ceilings, tail-based pricing, eval-validated cuts. |
Sources
- Prompt caching — Anthropic, 2025docs
- Batch processing — Anthropic, 2025docs