Prompt caching
Reusing the model's computed state for an identical prompt prefix across requests: cheaper, faster input for repeated context, minus a cache-write premium.
Many requests start with the same long text: the same instructions, the same tool list, sometimes the same big document. Prompt caching lets the provider keep the work it already did on that shared beginning, so the next request starting the same way is faster and cheaper.
It only works if the beginning of the Prompt & system prompt is exactly the same, character for character.
A chef who preps the base sauce once in the morning and finishes each order from it, instead of starting every dish from scratch. Making the big batch costs a little extra; every plate after that is cheap.
Putting a timestamp, user id or retrieved context at the top of the system prompt, silently busting the cache on every request and paying the 1.25× write premium for nothing.
The server stores the KV cache (per-layer key/value tensors from Attention) for a prompt prefix; a later request with a byte-identical prefix skips prefill for those tokens. Self-hosted, this is vLLM automatic prefix caching or SGLang's RadixAttention.
Hosted pricing as of Oct 2026, Anthropic: explicit cache_control breakpoints (up to 4 per request); cache writes cost 1.25× base input for the default 5-minute TTL or 2× for the 1-hour TTL; reads cost 0.1× on most models (some newer models discount further); each hit refreshes the TTL; prompts below a model-specific minimum (512–4,096 tokens) are not cached. OpenAI caches automatically for prompts of 1,024+ tokens with a model-dependent discount and no write premium. Because writes cost more than uncached input, caching only pays above a break-even hit rate: ~22% for 5-minute writes, ~53% for 1-hour writes.
Order by volatility: system prompt, tool schemas, reference documents, examples, then the breakpoint, then retrieved chunks and the question. Any change early in the prefix (a reordered tool list, an injected timestamp) invalidates everything after it. This is different from a Semantic cache, which reuses whole answers for similar questions.
One breakpoint after the static prefix (system + sorted tool schemas + reference docs); a second after conversation history for long agent loops; 5-minute TTL by default, 1-hour only when measured hit rate on it exceeds ~60%; a CI test that asserts the prefix bytes are identical across two renders; log cache_read_input_tokens and cache_creation_input_tokens per request.
A coding agent sends a 14k-token prefix (instructions + 30 tool schemas) every turn, plus ~2k tokens of turn-specific content. In base-input token-equivalents: uncached = 14k + 2k = 16k. Cache hit = 14k × 0.1 + 2k = 3.4k. Cache write = 14k × 1.25 + 2k = 19.5k. With 90% hits: 0.9 × 3.4k + 0.1 × 19.5k = 3.06k + 1.95k = 5.01k, a saving of 1 − 5.01/16 ≈ 69%, not the 75% you get by ignoring the write premium. TTFT drops from ~1.2 s to ~0.4 s on hits.
Commit: cache any static prefix over ~1k tokens that is reused within the TTL, and treat hit rate as an SLI. Show the break-even arithmetic including the write premium. The follow-up is 'traffic arrives every 10 minutes per tenant, now what?': 5-minute entries will expire between requests, so either the 1-hour TTL (needs >53% hits to pay) or accept no caching for that tenant.
Cache-friendly
- System prompt
- Tool schemas (sorted)
- Reference docs
- — breakpoint —
- Retrieved chunks + question
Cache-busting
- Timestamp / user id
- System prompt
- Retrieved chunks
- Tool schemas (unsorted)
- Question
Numbers worth knowing
- Anthropic cache write
- 1.25× base input (5-min TTL), 2× (1-hour TTL)as of Oct 2026
- Anthropic cache read
- 0.1× base inputmost models as of Oct 2026; some newer models lower
- Break-even hit rate
- ≈22% (5-min), ≈53% (1-hour)solve 1.25 − 1.15h < 1 and 2 − 1.9h < 1 for prefix-only cost
- Saving at 90% hits
- ≈69%14k cached prefix + 2k volatile, 5-minute writes
Try the numbers
Illustrative list prices — check your provider. Assumes cache reads at 0.1× input and misses re-writing the prefix at 1.25× (a 5-minute TTL); batch APIs typically take ~50% off on top.
How to decide
| When | Do this |
|---|---|
| Static prefix ≥ the model's minimum and requests per prefix arrive within 5 minutes | Cache with the default 5-minute TTL. |
| Requests per prefix arrive every 5–60 minutes | Use the 1-hour TTL only if the measured hit rate is above ~53%; otherwise don't cache. |
| Measured hit rate below ~22% | Remove the breakpoint; the write premium is costing you money. |
| Whole answers repeat for near-identical questions | That is a Semantic cache problem, not prompt caching. |
Cost maths
Expected billed prefix tokens per call = P × [h × r + (1 − h) × w], where r = 0.1 (read) and w = 1.25 (5-min write). P = 14k, V = 2k, h = 0.9: 14k × (0.09 + 0.125) = 3.01k; add V = 5.01k vs 16k uncached → 69% saving. At $3 / M input (illustrative): $0.048 → $0.015 per turn; at 2M turns/month, $96k → $30k. Drop h to 0.5 and the blend is 14k × (0.05 + 0.625) + 2k = 11.45k, only a 28% saving; at h = 0.2 it costs more than not caching.
Operating it
- Hit rate on the main prefix below 70% for a day
cache_creation_input_tokensper hour spikes without a deploy- Static prefix grows past ~20k tokens: split into two breakpoints by change frequency
- Cache hit ratio = read ÷ (read + creation + uncached input): alert below 70%
- Cache-creation tokens per hour: alert on 3× the 7-day baseline
- TTFT p95 on cache hits vs misses: alert if the gap closes (cache not being used)
When it goes wrong
Input spend on a support agent tripled overnight with flat traffic.
Cost-per-1k-requests alarm, then a dashboard showing cache_read_input_tokens near zero and cache_creation_input_tokens on every call.
A change added 'Current time: {now}' to the first line of the system prompt so the agent could reason about business hours; every request now had a unique prefix and paid the 1.25× write.
Moved the timestamp below the breakpoint, added a CI test that renders the prompt twice and diffs the cached prefix, and an alert on hit rate below 70%.
Where it shows up
Sources
- Prompt caching — Anthropic, 2026docs
- Prompt caching — OpenAI, 2026docs
- SGLang: Efficient Execution of Structured Language Model Programs — Zheng et al., 2023paper
Check yourself
Three questions. Every option is explained once you answer.