Cross-encoder
Reads the query and one document together and outputs a relevance score: far more precise than vector similarity, too slow to run over a whole corpus.
A cross-encoder looks at the question and one candidate document at the same time, word against word, and outputs a single score for how well the document answers the question.
Because it reads them together it catches details a vector comparison misses. But it runs once per document per question, so it is used as a Reranker on a shortlist (say 50 candidates), never to search millions.
A specialist who reads each shortlisted CV carefully against the job description, after a recruiter has quickly filtered thousands down to a shortlist.
Reranking only the top 5 or 10. If the right chunk is at rank 30 in the first stage, the reranker never sees it; set depth from recall@k, not habit.
Input is [query ; SEP ; document] through a transformer Encoder; full Attention across both texts allows token-level interaction, and a head outputs one scalar (a logit, not a calibrated probability). Nogueira & Cho (2019) showed a BERT cross-encoder reranking BM25 candidates jumped MS MARCO MRR@10 well beyond prior methods; BEIR later showed rerankers generalise across domains better than most dense retrievers.
Cost is one forward pass per (query, candidate) pair with nothing precomputed, so latency scales with candidate count × length. Rough numbers for 300-token chunks on one A10G: 100–300 ms for 100 candidates with MiniLM-L6 (22M), 3–5× that with a 568M model such as bge-reranker-v2-m3. LLM rerankers (listwise prompting) are stronger still and cost more; small cross-encoders distilled from them are the usual production compromise. Scores are not comparable across queries or languages, so order by them and be careful with fixed thresholds.
bge-reranker-v2-m3 (568M, multilingual) self-hosted on one GPU, or a hosted rerank API; rerank the top 50 from hybrid search, keep 5–8; max length 512 tokens per pair; batch all candidates in one call; MiniLM-L6 cross-encoder when CPU-only or under 100 ms total budget.
Insurance policy Q&A: 'Is hail damage to a detached carport covered?' The bi-encoder's top 50 puts 'Carport construction specifications' at rank 1 (cosine 0.81) and 'Other structures: wind and hail covered, subject to deductible' at rank 11. bge-reranker-v2-m3 scores the coverage clause 8.7 and the specs page 0.9, moving the right clause to rank 1. Reranking 50 candidates adds ~250 ms at p95 on one A10G.
Commit: rerank the top 50 by default, tune depth with recall@k (the reranker can't promote what it never sees) and latency. Say the bi-encoder decides what is possible, the cross-encoder decides what is first. Follow-up: 'latency budget is 100 ms total?': MiniLM-class reranker over 30 candidates, or a distilled model, measured on nDCG@10.
Numbers worth knowing
- Rerank latency, small model
- 100–300 ms per 100 candidatesMiniLM-L6 (22M) on one A10G, 300-token chunks
- Rerank latency, 568M model
- 3–5× the small modelbge-reranker-v2-m3 on the same hardware
- Default depth
- top 50 → keep 5–8first-stage recall@50 ≥ 0.9
How to decide
| When | Do this |
|---|---|
| First-stage recall@50 ≥ 0.9 but nDCG@10 is low | Add a cross-encoder over the top 50. |
| Latency budget for reranking under 100 ms | Small cross-encoder over 30–50, or a distilled model. |
| Need to abstain on weak evidence | Calibrate thresholds per language and domain, or use the margin between top scores. |
Cost maths
Rerank cost = queries × candidates × pair tokens. 1M queries/month × 50 × 400 tokens = 20B tokens of encoder work. Self-hosted bge-reranker-v2-m3 at ~250 ms per query on one A10G (~$1.00/hour, illustrative) handles ~4 QPS per GPU; 1M queries/month ≈ 0.4 QPS average, so one GPU ≈ $730/month covers it with headroom. A hosted rerank API priced per search is often cheaper below ~1M searches/month.
Operating it
- Rerank p95 above its share of the latency budget
- Sustained QPS above ~70% of one GPU's rerank capacity
- nDCG@10 gains from reranking below 3 points (reconsider the stage)
- nDCG@10 with vs without reranker on the golden set: alert if the gain falls below 3 points
- Rerank p95 latency: alert above 300 ms
- Abstention rate by language: alert on 2× baseline
When it goes wrong
After adding Spanish policy documents, the assistant's 'I couldn't find that' rate rose from 3% to 22% for Spanish questions.
Abstention-rate alarm broken down by language.
Abstention used a fixed reranker score threshold tuned on English; the model's score distribution for Spanish pairs was systematically lower.
Per-language thresholds calibrated on 300 labelled pairs each, and abstention switched to a combination of score and top-1/top-2 margin.
Related terms
Where it shows up
Sources
- Passage Re-ranking with BERT — Nogueira & Cho, 2019paper
- BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models — Thakur et al., 2021paper
Check yourself
Three questions. Every option is explained once you answer.