Design a semantic cache for a multi-tenant LLM product. How do you keep it correct?
- 5 min — scope: traffic shape, tenants, correctness bar
- 10 min — exact vs semantic vs prompt caching; key design
- 10 min — thresholds, guards, cacheability, invalidation
- 10 min — false-hit measurement, poisoning, isolation
- 5 min — estimate and rollout
Two caches with different risk: an exact cache (hash of the normalised prompt plus the full key) and a semantic cache (nearest-neighbour on a query embedding above a cosine threshold). Provider prompt caching is a third, unrelated thing — prefix KV reuse that is always correct.
The key must include everything that changes the right answer: tenant, model and version, prompt version, retrieval index version, the user's ACL scope, locale and tool availability — and the user for personalised answers. Each entry stores the source document ids it was built from, for invalidation.
A developer-docs assistant with 500 tenants. 'How do I rotate an API key in the Python SDK?' → normalise → exact lookup in a tenant-scoped Redis key (~1 ms) → miss → embed (~5 ms) → ANN search in the tenant's namespace at cosine ≥ 0.95 (calibrated per tenant) plus guard checks that versions, languages, numbers and entities match ('v2' vs 'v3', 'Python' vs 'Go') → hit returns the stored answer with its original citations; miss runs full RAG and writes back only if the answer is cacheable (grounded, no personal data, not low-confidence). A docs-update event invalidates every entry citing that document.
- False hits are silent. Sample hits daily and judge whether the cached answer is right for the new query (LLM judge + human spot-checks); budget ≤0.5%. Compare thumbs-down rate on hits vs misses — if hits are worse by >1 point, raise the threshold.
- Isolation: a namespace per tenant; never share across tenants unless content is tenant-agnostic and explicitly flagged. Cache entries count as customer data for deletion requests.
- Failure modes: stale answers after document updates; version mismatch; cache poisoning (a crafted query produces a bad answer that is then served to others — only cache answers from trusted paths, or cache per user).
- Where it pays: 10–30% hit rates on support and docs traffic, under 5% on open chat, near zero for agents. Org scope: the platform ships it as an opt-in gateway feature that requires a passing false-hit eval per product; tenants can disable it. It is thin enough to build on Redis/Valkey plus a vector index.
Assumptions: 500 tenants, 20M queries/day; full pipeline per miss ≈ $0.003 (2k input / 200 output at illustrative $1/M input, $5/M output); 20% hit rate (5% exact, 15% semantic); 30-day TTL; 1,024-d int8 embeddings + ~2 KB payload.
- QPS: 230 average, ~700 peak; one embedding per query (~5 ms on a shared GPU, or under $10/day via API).
- Savings: 20M × 20% × $0.003 ≈ $12k/day ≈ $360k/month, and hits return in ~30 ms instead of ~2 s.
- Storage: ~100M live entries × ~3 KB ≈ 300 GB → a six-node Redis/Valkey + vector cluster ≈ $15k/month.
- Risk budget: 4M hits/day at a 0.5% false-hit rate = 20k wrong answers/day; at 0.1%, 4k/day. That number — not the hit rate — sets the threshold.
What the answer sounds like at each level
Embeds queries and returns a stored answer when similarity exceeds a threshold.
Puts tenant, model and prompt version in the key, invalidates on document change, and calibrates the threshold.
Treats false-hit rate as the governing metric with a budget, defines cacheability rules and poisoning defences, and ships it as an opt-in, eval-gated platform feature.
Follow-ups the interviewer will push
1A regulated tenant joins.
Disable the semantic tier or scope it per user; encrypt entries with the tenant's key; TTL ≤ 24 h; deletion requests purge cache entries; no cross-tenant sharing under any flag.
2Traffic grows 10×.
Hit rate improves on head queries; shard the index per tenant, give hot tenants dedicated shards, and enforce per-tenant memory quotas so a noisy neighbour cannot evict everyone else.
3How do you detect false hits in production?
Daily sampled judging of hits against fresh pipeline answers, plus feedback on cached vs uncached responses; alert when estimated false-hit rate exceeds 0.5%.
Red flags interviewers listen for
- Cache key is the query text or embedding alone, with no tenant.
- No invalidation when source documents change.
- Picks a threshold like 0.9 without measuring false hits.
- Caches answers containing personal data or user-specific tool results.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Key design | Includes tenant, model/prompt/index versions and ACL scope; stores source ids. |
| Correctness | False-hit budget with sampled judging and feedback comparison. |
| Security | Per-tenant namespaces, poisoning defence, deletion coverage. |
| Economics | Weighs savings against the absolute number of wrong answers served. |
Sources
- Prompt caching — Anthropic, 2025docs
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Lewis et al., 2020paper