Dense retrieval
Search by meaning: embed the query and every chunk as vectors, then return the chunks whose vectors are nearest to the query's.
Dense retrieval turns every chunk of text into an Embedding, a list of a few hundred to a few thousand numbers that captures its meaning. Your question becomes the same kind of list, and the system returns the chunks whose lists are most similar, usually by Cosine similarity. It is called dense because every number is used, unlike Sparse retrieval where almost all are zero. It finds paraphrases that share no words with the query.
A map where every document is a pin and similar topics sit near each other. You drop a pin for your question and pick up the closest pins.
Picking the embedding model from the MTEB leaderboard average. Leaderboard gains of 1–2 points routinely vanish or reverse on a private corpus; a 300-query in-domain eval costs a day and is the only number that matters. Also: changing models means re-embedding everything — vectors from two models are not comparable.
A Bi-encoder maps query and passage independently into one Vector space (384–3072 dims), trained contrastively with in-batch and hard negatives. Passages are embedded offline; a query costs one encoder call (5–20 ms self-hosted, 50–200 ms via an API) plus an Approximate nearest neighbour (ANN) lookup (HNSW, IVF (inverted file index)) of 1–10 ms. The whole passage is compressed into one vector before the query is seen, so term-level interactions (negation, numbers, exact IDs) are lost — the reason a Cross-encoder Reranker follows. Asymmetric models (E5, bge, Qwen3-Embedding) need the right Instruction prefix (query: / passage:) on queries; forgetting it costs 2–10 recall points. DPR beat BM25 on in-domain Natural Questions (78.4% vs 59.1% top-20 accuracy) but fell behind it out of domain on BEIR — domain fit matters more than the MTEB leaderboard leaderboard.
Open-weights bge-m3 (568M, 1024-d, multilingual) or Qwen3-Embedding-0.6B self-hosted on one L4; or an API model (OpenAI text-embedding-3-small, Cohere Embed, Voyage) when you cannot host. Store in pgvector HNSW (m=16, ef_construction=64, ef_search 100) under ~10M vectors; normalise vectors and use inner product.
Insurance claims, 4M adjuster notes, bge-m3 (1024-d). Query: 'water came through the ceiling after the storm'. Top hit (cosine 0.81): 'roof leak following hail event, interior drywall damage' — zero shared content words, BM25 ranks it 212th. Failure on the same index: 'claims NOT involving water damage' returns water-damage claims at ranks 1–10, because the vector for 'not water damage' sits next to 'water damage'.
Default answer for a RAG retriever: "dense for paraphrase recall, BM25 for identifiers, fused with RRF, cross-encoder on the top 50." If the interviewer pushes on domain shift, say you would measure recall@20 on 200 in-domain queries before picking a model, and fine-tune (Fine-tuning an embedding model) only when the best off-the-shelf model is > 10 points below the target.
Numbers worth knowing
- Query embed latency
- 5–20 ms self-hosted, 50–200 ms APIsingle query, ≤64 tokens, 0.5B-class model on L4
- Raw vector memory
- 3 KB per chunk768-d float32; 4 KB at 1024-d
- DPR vs BM25, NQ top-20 accuracy
- 78.4% vs 59.1%in-domain (Karpukhin et al. 2020); reverses on many BEIR datasets
How to decide
| When | Do this |
|---|---|
| Natural-language questions over prose (wikis, policies, support) | Dense is the lead leg; add BM25 via Hybrid search. |
| In-domain recall@20 is > 10 points below target with the best off-the-shelf model | Fine-tune on 5–50k mined pairs (Fine-tuning an embedding model). |
| Queries contain negation, numbers or date ranges | Push those constraints to Metadata filtering (pre, post, filtered ANN) or a reranker; vectors will not honour them. |
Cost maths
Embedding cost = tokens × price. 10M chunks × 400 tokens = 4B tokens; at an assumed $0.02 per M tokens (OpenAI text-embedding-3-small list price, Oct 2026) = $80 per full embed; at $0.13/M (3-large) = $520. Self-hosted bge-m3 on one L4 (~$0.80/hr, ~2k chunks/s) = 1.4 hours ≈ $1.10. Storage: 10M × 1024-d × 4 B = 41 GB float32 — halve with float16, quarter with int8 (Scalar & binary quantization).
Operating it
- Vector count passes ~10–20M or index RAM > 50% of the Postgres instance — move off pgvector or quantize
- Recall@20 drops > 3 points after a corpus shift — re-evaluate model or fine-tune
- Embedding API p99 > 300 ms — self-host the query encoder
- Top-1 similarity distribution (daily median) — alert on a shift > 0.1
- Recall@20 on the golden set per release — alert on a drop > 3 points
- Embedding model version in query path vs index metadata — page on mismatch
When it goes wrong
After a quiet model swap in the developer-docs assistant, answers cited unrelated pages for two days.
Groundedness score from the nightly Eval harness fell from 0.91 to 0.62; retrieval logs showed top-1 cosine dropping from ~0.8 to ~0.3.
The query encoder was upgraded to a new model version while the index still held vectors from the old one; the two spaces were incompatible.
Pinned model version as index metadata, refused queries when versions mismatched (Fail closed), and moved model changes to a full re-embed behind an Index alias swap.
Related terms
Where it shows up
Sources
- Dense Passage Retrieval for Open-Domain Question Answering — Karpukhin et al., 2020paper
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks — Reimers & Gurevych, 2019paper
- BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models — Thakur et al., 2021paper
Check yourself
Three questions. Every option is explained once you answer.