Semantic search vs similarity search
Search that matches what the query means rather than the exact words, usually built from embeddings plus a reranker.
Semantic search is the user-facing idea: type a question in your own words and get results that answer it, even if they use different words. Under the hood it is usually Dense retrieval with embeddings, often combined with keyword search (Hybrid search) and a Reranker that reads the top results more carefully.
A shop assistant who understands 'something to keep my coffee hot on the commute' and walks you to travel mugs, even though you never said 'mug'.
Assuming 'semantic' means 'understands constraints'. Embeddings blur numbers, negation and attributes: 'laptop under $800, not Chromebook' needs a parser that turns price and exclusion into filters. Without it, the top results include $1,400 laptops and Chromebooks.
The term names an outcome, not an algorithm. A production semantic-search stack has four stages: query understanding (spell-fix, Query rewriting, Intent classification & routing), candidate retrieval (dense + BM25, 100–200 candidates), reranking (Cross-encoder on 50–100), and presentation (snippets, highlighting, facets). Most of the 'semantic' quality comes from the reranker, which reads query and document together: on MS MARCO, BERT reranking of BM25 candidates lifted MRR@10 from 0.187 to 0.365 (Nogueira & Cho 2019). Highlighting is harder than in keyword search because the matching words may not be in the text; production systems use the reranker's best-scoring sentence or an LLM-extracted span.
Hybrid retrieval (BM25 + bge-m3 or text-embedding-3-small) to 200 candidates, structured filters from a lightweight query parser, bge-reranker-v2-m3 on the top 50, sentence-level highlighting from the reranker's best sentence.
E-commerce catalog, 800k products. Query 'gift for a 10-year-old who likes space'. Keyword search returns 'Space heater 1500W' first. Semantic stack: dense retrieval pulls telescopes, planetarium projectors and astronaut LEGO (200 candidates, 12 ms); a filter removes out-of-stock items; a cross-encoder reorders the top 50 (140 ms) and puts a kids' telescope first. The space heater falls to rank 180.
Clarify which layer they mean: "Semantic search to me is dense + BM25 recall with a cross-encoder for precision; the embedding alone gets the right items into the top 100, the reranker gets them into the top 5." The follow-up is evaluation: report nDCG@10 and zero-result rate on a sampled query log, not anecdotes.
Numbers worth knowing
- BERT rerank gain over BM25
- MRR@10 0.187 → 0.365MS MARCO passage dev (Nogueira & Cho 2019)
- Typical end-to-end budget
- 150–300 ms p95hybrid recall + cross-encoder on 50 candidates, one L4
How to decide
| When | Do this |
|---|---|
| Queries carry hard constraints (price, date, size, 'not X') | Parse into filters before retrieval; do not rely on embeddings. |
| Latency budget < 80 ms | Skip the cross-encoder; use a distilled reranker or a Bi-encoder fine-tuned on clicks. |
Cost maths
Per query: embed (~$0.0000004 at $0.02/M tokens × 20 tokens) + ANN (fixed infra) + rerank. Self-hosted reranker on one L4 (~$0.80/hr) at ~8 queries/s with 50 candidates → $0.000028/query, ~$28 per million queries. A hosted rerank API at an assumed ~$2 per 1k searches (illustrative, Oct 2026) is ~$2,000 per million — self-host above ~1M queries/month.
Operating it
- Zero-result or low-CTR rate > 10% on a query class — add rewriting or synonyms for that class
- Rerank stage > 60% of latency at p95 — distil or cut candidates to 30
- Query refinement rate within 30 s — alert if > 25% for any query class
- Zero-result rate — alert above 2%
- Click-through at rank 1–3 vs last month — alert on −10% relative
When it goes wrong
A product search returned premium items for 'cheap running shoes under 50'.
Session-level refinement rate (user re-types within 30 s) was 41% on queries containing price words vs 18% overall.
Price constraints were left to the embedding; 'cheap' and 'premium' sit close together in vector space because both describe price.
Added a rule + small-model parser that extracts price, size and exclusions into filters before retrieval; refinement rate on price queries dropped to 21%.
Related terms
Sources
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks — Reimers & Gurevych, 2019paper
- Passage Re-ranking with BERT — Nogueira & Cho, 2019paper
Check yourself
Three questions. Every option is explained once you answer.