How would you design caching for a RAG application?
- 2 min — layers and key/TTL for each.
- 3 min — correctness: tenant keys, invalidation, false-hit eval.
- 3 min — cost maths: prompt cache vs semantic cache.
- 2 min — rollout order and ownership.
Cache at every layer, with different keys and TTLs. Embeddings keyed by content hash — permanent, model-versioned. Retrieval results keyed by normalised query plus filters plus tenant, short TTL, invalidated on index change. Reranker scores keyed by query-document pair. LLM prompt cache for stable prefixes (system prompt, boilerplate). Semantic answer cache for near-duplicate questions.
Support traffic is extremely head-heavy — a small number of questions covers most volume. A semantic cache there pays for itself quickly. Set the similarity threshold conservatively: "how do I reset my password" and "how do I reset my PIN" are close in embedding space and must not share an answer.
Caching is where correctness bugs hide. Three rules to state: tenant must be in every key; a stale cached answer after a document update is a correctness failure, so wire index changes to invalidation; and a semantic cache needs its own eval measuring false-hit rate, because it can only be wrong silently. Also note prompt caching often gives a bigger latency win than retrieval caching, for less risk.
Org scope: prompt caching is a provider feature you turn on (buy); the semantic cache is something you build and therefore own its eval, invalidation and on-call. Ship prompt caching first across all teams via the LLM gateway, then add the semantic cache only for products whose traffic is head-heavy enough to justify it.
Assumptions: 1M queries/day; per query 3,000 input tokens (2,000-token stable system prompt + 1,000 tokens of retrieved context and question) and 250 output tokens; illustrative list prices, Oct 2026: $3/M input, $15/M output; cache reads 0.1× input, 5-minute cache writes 1.25×.
- No caching: 3,000 × $3/M + 250 × $15/M = $0.009 + $0.00375 = $0.01275/query → $12,750/day.
- Prompt caching on the 2,000-token prefix: 2,000 × $0.30/M + 1,000 × $3/M = $0.0036 input + $0.00375 output = $0.00735/query (−42%). At 1M/day (~12 queries/s) the 5-minute prefix stays warm, so write premiums are negligible.
- Semantic cache at 25% hit rate on top: 0.75 × $7,350 ≈ $5,510/day. Lookup cost: one ~20-token query embedding plus a Redis/vector lookup, well under $0.0001/query.
- Risk budget: at a 0.5% false-hit rate, 250k hits/day → 1,250 wrong answers/day. That number, not the savings, sets the threshold.
What the answer sounds like at each level
Cache repeated questions and their answers, and cache embeddings so you don't recompute them.
Layers caches with distinct keys and TTLs, includes tenant in every key, invalidates on index change, and turns on provider prompt caching for stable prefixes.
Quantifies savings per layer, sequences buy (prompt caching) before build (semantic cache), and makes false-hit rate an evaluated, owned metric.
Follow-ups the interviewer will push
1A policy document changes. How long can the old answer be served?
Zero if you can help it: on document update, publish an invalidation event listing affected chunk ids; evict retrieval and semantic-cache entries that cited them (store citations with each cached answer). Fall back to a short TTL (e.g. 1 hour) as a safety net, not the primary mechanism.
2How do you pick the semantic-cache similarity threshold?
Label ~1,000 near-duplicate query pairs as same-answer or different-answer, sweep the threshold, and pick the point where false-hit rate is under your budget (e.g. <0.5%). Expect a high cosine (0.95+) for most general-purpose embedding models; re-tune whenever the embedding model changes.
Red flags interviewers listen for
- Semantic cache keyed only on query text, without tenant.
- No invalidation when source documents change.
- No measurement of false hits.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Layering | Distinct key and TTL per layer, model-versioned embedding cache. |
| Correctness | Tenant in every key, event-driven invalidation, false-hit eval. |
| Economics | Computes savings including cache-write premiums and hit rates. |
Sources
- Prompt caching — Anthropic, 2024docs