Semantic cache
Reusing a stored answer when a new query's embedding is close enough to a past one. Saves cost and latency; risks returning the wrong answer.
A normal cache only helps when the exact same request repeats. A semantic cache stores past questions as embeddings and, when a new question is very similar in meaning, returns the saved answer without calling the Large Language Model (LLM). It is fast and cheap, but 'similar' is not 'the same', so a careless threshold can hand someone an answer to a different question.
A help-desk worker who recognises 'how do I reset my password?' and 'forgot my password, what now?' as the same question and reads the standard reply, but must notice when someone asks 'how do I stop password resets?'.
Tuning the threshold for hit rate. Every point of hit rate bought by lowering the threshold is paid for in wrong answers served at full confidence.
Flow: embed the normalised query → Approximate nearest neighbour (ANN) lookup in a cache index scoped by tenant, locale, user permissions and prompt/model version → if top cosine ≥ threshold (commonly 0.92–0.97), return the stored answer; else generate, then store. The hard parts are correctness, not speed. Embeddings are weak on negation, numbers and entities ('cancel order 1182' vs 'cancel order 1183' score ~0.98), so production systems add guards: exact match on extracted entities, a cross-encoder or small-LLM equivalence check on hits, and exclusion of personalised or time-sensitive intents. Invalidate on corpus change (key by document-version hash), on prompt or model change, and with a TTL & cache invalidation. Do not confuse it with Prompt caching, which caches the KV state of an exact prefix at the provider and is always correct; prompt caching is the safer first win.
Provider prompt caching first. Semantic cache (Redis vector search, GPTCache, or a gateway feature) only for an allow-list of non-personalised intents; threshold 0.95; key scope = tenant + locale + prompt version + model + corpus version; TTL 24 h; entity-equality check on hits; 1% of hits regenerated and judged.
Customer-support bot, 2M queries/month, ~35% are FAQ-style ('where is my invoice', 'change billing email'). Cache only intents tagged faq, threshold 0.95, plus a check that extracted entities (order ids, plan names) match. Hit rate 24% of total traffic; p50 latency on hits 40 ms vs 2.1 s; sampled precision of hits 99.2% by LLM judge. Personalised intents ('why was I charged twice') are never cached.
Say: I start with provider prompt caching because it is always correct, then add a semantic cache only for non-personalised intents, with a high threshold, entity matching, and a sampled correctness audit on hits. Expect the follow-up: 'how do you measure false hits?' Sample 1% of hits, regenerate fresh, and have a judge compare; alert above 1% disagreement.
Numbers worth knowing
- Threshold
- 0.92–0.97 cosinestart at 0.95; depends on embedding model
- Realistic hit rate
- 10–30% of trafficsupport or FAQ-heavy workloads; < 5% for open-ended assistants
- Hit latency
- 20–60 msembedding + ANN lookup + checker skipped on exact hash match
How to decide
| When | Do this |
|---|---|
| Answers depend on user, account state or time | Do not semantic-cache; use prompt caching only |
| High repeat FAQ traffic in a narrow domain | Semantic cache on allow-listed intents with entity check |
| Corpus updates hourly | Key by corpus version or keep TTL ≤ 1 h |
Cost maths
Savings = hit rate × queries × LLM cost per query − cache cost. 2M queries/month × 24% hits × $0.012 = $5,760/month saved. Cache cost: embeddings (2M × 20 tokens at $0.02/M ≈ $0.80), a small Redis node (~$200/month), checker calls on hits (480k × $0.0005 = $240). Net ≈ $5,300/month, provided false hits stay < 1%.
Operating it
- FAQ-style intents > 20% of traffic and stable: worth adding
- Sampled false-hit rate > 1%: raise threshold or add a cross-encoder check
- Cache index > 10M entries: add TTL eviction and per-tenant caps
- Hit rate by intent; trend, alert on a sudden > 2× jump (threshold or embedding bug)
- Sampled false-hit rate; alert > 1%
- 'Didn't answer my question' feedback on cached vs fresh answers; alert if cached is > 1.5× fresh
When it goes wrong
Users asking how to keep auto-renew on were told how to turn it off
Spike in 'didn't answer my question' feedback on a single intent; trace review showed cache hits
'How do I turn off auto-renew' and 'how do I keep auto-renew on' embedded at cosine 0.95; the threshold was 0.93 and there was no checker
Threshold to 0.96, small-LLM equivalence check on every hit, negation-pair test cases added to the cache regression suite
Related terms
Where it shows up
Sources
- GPTCache: semantic cache for LLMs — Zilliz, 2023docs
- Prompt caching — Anthropic, 2025docs
Check yourself
Three questions. Every option is explained once you answer.