Bi-encoder
Encodes query and documents separately into vectors and compares them: fast and indexable, the backbone of dense retrieval.
A bi-encoder turns the question into a vector and each document into a vector separately, then compares the vectors to find the closest matches. Because documents are encoded without knowing the question, that work happens once, ahead of time, and is stored.
At query time only the question is encoded and its nearest neighbours looked up, which takes milliseconds even across millions of documents.
Giving every book in a library a map coordinate in advance, so a new question just needs its own coordinate to find the nearest shelves.
Expecting a bi-encoder alone to put the best result at position 1. It is a recall tool; precision at the top needs reranking.
One encoder (or twin encoders) maps query q and document d to vectors independently; relevance is s(q, d) = sim(E(q), E(d)), usually Cosine similarity or Dot product. Independence is what makes it scale: documents are embedded offline into an Approximate nearest neighbour (ANN) index, so query cost is one encoder pass plus a sublinear lookup.
The price is that query and document never see each other. Negation, exact constraints and which entity a number belongs to get compressed away. So bi-encoders do high-recall Candidate generation & the retrieval funnel (top 50–200), and a Cross-encoder Reranker restores precision. Quality depends on training pairs and hard negatives (DPR showed in-batch plus BM25 hard negatives matter); Fine-tuning an embedding model on 5k–50k in-domain pairs often adds 5–15 points of recall. Asymmetric models need the right Instruction prefix (query: / passage:): E5 expects 'query: ' and 'passage: ', and omitting them costs recall silently.
bge-m3 or a hosted embedding model; documents embedded at ingest, queries at request time with the model's documented prefix; HNSW (M = 16, ef_construction = 200, ef_search tuned to recall ≥ 0.95 vs exact); top 50–100 candidates handed to a reranker.
Developer-support search over 5M doc and forum chunks, embedded offline in ~3 hours on 4 GPUs. Query 'webhook signature mismatch after rotating secret': encode 8 ms, HNSW search 12 ms, top 100 with recall@100 = 0.93 on the golden set. The right answer sits at rank 9; a cross-encoder over the 100 moves it to rank 1.
Commit: bi-encoder for recall, cross-encoder for precision, and measure recall@k at the depth you hand the reranker. Follow-up: 'why not cross-encode everything?': 5M pairwise passes per query is ~hours of GPU time; the bi-encoder precomputes the document side so a query costs one forward pass.
Numbers worth knowing
- Query encode
- 5–15 ms100M–600M encoder on a GPU
- ANN lookup
- 2–20 msHNSW over 1–10M vectors, recall ≥ 0.95 vs exact
- Fine-tuning gain
- +5–15 recall points5k–50k in-domain pairs with hard negatives (range to verify on your golden set)
How to decide
| When | Do this |
|---|---|
| Corpus above ~10k documents | Bi-encoder first stage; cross-encoders cannot scan the corpus. |
| Recall@100 below 0.85 on in-domain queries | Fine-tune the bi-encoder on mined pairs before adding more stages. |
| Queries are short keywords or identifiers | Add BM25 and fuse; don't rely on dense alone. |
Cost maths
Offline: 5M chunks × 300 tokens; at an assumed ~500 chunks/s per GPU, 4 GPUs ≈ 42 minutes, ~$7 at ~$2.50/GPU-hour (illustrative). Online: one query encode ≈ 1/1,000 of a GPU-second; 100 QPS needs a fraction of one GPU. The cross-encoder alternative over all 5M chunks would need ~5M forward passes per query: not a cost, an impossibility.
Operating it
- Recall@100 below 0.9 at the depth passed to the reranker
- Index RAM above one node
- Query volume above ~500 QPS on one encoder replica
- Recall@k at reranker depth on the golden set: alert at −3 points
- HNSW recall vs exact on 500 sampled queries: alert below 0.95
- Query encode + search p95: alert above 50 ms
When it goes wrong
After switching to an E5-family model, developer-docs recall@20 dropped from 0.82 to 0.71 despite the model's higher MTEB score.
Pre-launch golden-set comparison, luckily before cut-over.
Queries and passages were embedded without the required 'query: ' / 'passage: ' prefixes.
Prefixes added in the embedding client keyed by model name, with a unit test asserting them; recall rose to 0.87.
Sources
- Dense Passage Retrieval for Open-Domain Question Answering — Karpukhin et al., 2020paper
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks — Reimers & Gurevych, 2019paper
Check yourself
Three questions. Every option is explained once you answer.