How do you control agent cost and token budgets?
- 2 min — quadratic growth formula
- 3 min — worked example uncached vs cached
- 2 min — cost per successful task; routing
- 3 min — tail spend, budgets, caching pitfalls
- 2 min — FinOps/platform ownership and gateway choice
Agent cost grows faster than step count because each step re-sends the transcript. With prefix P and growth g per step, input tokens over n steps ≈ n·P + g·n(n−1)/2 — quadratic in n if nothing is trimmed. Levers, in order of impact: fewer steps (better tools), prompt caching of the stable prefix (reads at 0.1× base input), smaller context per step (compaction), routing easy work to cheaper models, and hard per-run budgets.
The metric is cost per successful task, not per call. Model A at $0.40/run and 80% success costs $0.50 per success; model B at $0.25/run and 45% success costs $0.56 — the “cheaper” model is more expensive.
Internal-wiki research agent: 20 steps, 8k prefix (system + tools), each step adds 1.5k tokens of tool output and reasoning, 300 output tokens per step. Uncached: 445k input → ~$1.43 per run. Caching the prefix and incrementally caching the transcript brings it to ~$0.35. A triage call on a small model routes 55% of questions to a single retrieval call (~$0.01) without entering the loop. Per-run budget: $1.00 hard stop, with partial results returned. Spend is attributed per team via the LLM gateway.
Watch the tail, not the mean: if 3% of runs are runaway loops at 10× the median cost, they are ~24% of spend. Set budgets from the observed distribution of successful runs and alert on runs above p99. Price long tasks honestly — a 60-step task costs ~7× a 20-step one under quadratic growth, not 3×. Caching has rules: content must be a byte-identical prefix, writes cost 1.25× (5-min TTL) or 2× (1-hour), and anything that changes early in the prompt (a timestamp in the system prompt) silently disables it — monitor cache-hit ratio per route. Use per-tenant limits against noisy neighbours.
Org scope. FinOps and the platform team co-own the cost model: the gateway attributes spend per tenant, team and feature; budgets are policy defaults with override-on-evidence; a monthly review flags routes with falling cache-hit ratio or rising cost per success. Build vs buy: buy or adopt a gateway (LiteLLM, Portkey, cloud-native gateways) for attribution and limits; build the per-run budget logic in your loop. Failure mode: aggressive compaction drops a constraint, the task fails, and a failed task is the most expensive outcome of all — always tune cost with success on the same dashboard.
Assumptions: n=20 steps, prefix P=8k, growth g=1.5k/step, 300 output tokens/step. Input = 20×8k + 1.5k×190 = 445k. assume $3 / M input and $15 / M output tokens (illustrative list prices, Oct 2026); cache writes 1.25× input (5-minute TTL), cache reads 0.1×. Uncached: 445k×$3/M = $1.34 + 6k×$15/M = $0.09 → $1.43. Cached: ≈36.5k written at $3.75/M ($0.14) + ≈408k read at $0.30/M ($0.12) + output $0.09 → ≈ $0.35 (≈4× cheaper). At 50k runs/day: $71.5k/day uncached vs ≈$17.5k/day cached.
What the answer sounds like at each level
Limit the number of steps, cache the system prompt, and use cheaper models where possible.
Models quadratic transcript growth, quantifies caching savings, reports cost per successful task, and budgets from the success distribution with tail alerts.
Builds a cost model with FinOps — attribution per tenant and feature, policy budgets, cache-hit monitoring — and makes the gateway build-vs-buy call.
Follow-ups the interviewer will push
1Your cache-hit ratio fell from 85% to 20% overnight. Likely cause?
Something early in the prefix started changing per request — a timestamp, a user name in the system prompt, reordered tool definitions. Diff the first 4k tokens of two consecutive requests; move volatile content after the cache breakpoint.
2Now a 10× traffic spike from one tenant. What protects everyone else?
Per-tenant token-rate limits and concurrency caps at the gateway, a per-tenant monthly budget with soft and hard thresholds, and queueing of that tenant's background runs rather than letting them consume shared rate limits.
3Would batch APIs help?
For non-interactive runs, yes: ~50% discount, but each step waits for batch completion, so only for offline agent workloads such as nightly catalog enrichment — not interactive loops.
Red flags interviewers listen for
- Reports cost per call instead of cost per successful task.
- Assumes cost scales linearly with steps.
- Cuts cost without tracking success on the same eval.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Cost model | Correct quadratic formula with a worked, priced example. |
| Metric choice | Cost per successful task, tail share of spend. |
| Caching fluency | Prefix rules, write premiums, hit-ratio monitoring. |
| Org scope | Attribution, policy budgets, review cadence with FinOps. |
Sources
- Prompt caching — Anthropic, 2025docs
- How we built our multi-agent research system — Anthropic, 2025blog