Debugging drill: semantic cache hit rate fell from 35% to 4% over a day. Walk through it.
- 1 min — quantify impact (cost, latency)
- 2 min — ranked hypotheses
- 3 min — miss-similarity distribution, key cardinality, eviction counters
- 2 min — fix, model-namespaced cache, alerts and ownership
Symptoms: hit rate collapse, p50 latency up from ~80 ms to ~1.8 s, LLM spend up ~50%, no errors. Hypotheses ranked: (1) the embedding model or its version changed for queries while cached entries were embedded with the old one, so similarities fell below threshold; (2) the cache key gained a high-cardinality component (session id, timestamp, user id instead of tenant); (3) capacity or TTL change causing eviction; (4) the similarity threshold changed in config; (5) a genuine query distribution shift (product launch, new language).
The one metric that splits them: the distribution of nearest-neighbour similarity for misses. Whole distribution shifted down (median 0.94 → 0.72) → (1) or (4); check the embedding model version and threshold config. Nearest neighbours still at 0.95+ but rejected → key mismatch (2). Cache entry count dropping and eviction counter spiking → (3). Similarities normal for old query types but new topics have no neighbours → (5). Fix for (1): pin model version in the cache key namespace, rebuild or warm the cache under the new model, and never mix vectors from two models in one index.
Prevention: the cache's index namespace includes the embedding model id; a deploy that changes the encoder must change the namespace and warm it before cutover. Alert on hit-rate drop >30% relative over an hour and on similarity-distribution shift. Track false-hit rate on a sampled, judged set too — a cache can also fail by hitting too much after a threshold change, which is silent and worse. Ownership: one platform team owns embedding versions for search and cache, so a model change can't be shipped to one and not the other.
Impact: 1M queries/day, uncached cost ~$0.004 per query (illustrative), cache lookup ~$0.0001. At 35% hits: 650k × $0.004 = $2,600/day. At 4%: 960k × $0.004 = $3,840/day → +$1,240/day (~$450k/year) and median latency up ~20× for the 31% of traffic that used to hit. Rebuild cost after a model change: 2M cached entries × 30 tokens ≈ 60M tokens ≈ $1.20 on a $0.02 / M embedding API — trivial; the risk is entirely in the cutover, not the compute.
Whole curve shifts down
- median 0.94 → 0.72
- embedding model mismatch
- or threshold changed
Curve unchanged, still misses
- neighbours at 0.95+
- key cardinality exploded
- or entries evicted
What the answer sounds like at each level
Checks whether the cache is up and whether the threshold changed.
Plots nearest-neighbour similarity for misses to split model mismatch, key cardinality and eviction.
Namespaces the cache by model version, alerts on similarity distribution, owns false-hit rate as well as hit rate, and unifies embedding version ownership.
Follow-ups the interviewer will push
1Now the hit rate is fine but users report wrong answers.
False hits: threshold too loose or the key lacks tenant/locale. Sample hits, judge whether the cached answer fits the new query, and measure false-hit rate; tighten the threshold per intent rather than globally.
2Now it's a multi-tenant system.
Tenant must be in every key — a shared semantic cache across tenants leaks one customer's grounded answer to another. Global caching only for ungrounded, non-tenant content.
Red flags interviewers listen for
- Lowers the similarity threshold to restore hit rate — trades a visible problem for silent wrong answers.
- Doesn't consider that vectors from two embedding models aren't comparable.
- Monitors hit rate but not false-hit rate.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Diagnosis | Uses the similarity distribution of misses as the splitting metric. |
| Correctness awareness | Refuses to lower the threshold blindly; tracks false hits and tenant scoping. |
| Prevention | Model version in the cache namespace, warm-before-cutover, joint ownership of embedding versions. |
Sources
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks — Reimers & Gurevych, 2019paper
- MTEB: Massive Text Embedding Benchmark — Muennighoff et al., 2022paper
- Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs — Malkov & Yashunin, 2016paper