Reranker
A slower, more accurate model that re-scores the top 50–200 first-stage candidates so only the best 5–10 reach the LLM.
Search usually runs in two steps. A fast first step (Candidate generation & the retrieval funnel) pulls the 100 or so passages that might be relevant out of millions. A reranker then reads the question and each of those 100 passages together and gives each a fresh relevance score. You keep the top few and pass them to the Large Language Model (LLM). It cannot find passages the first step missed; it only puts the ones you have in a better order.
A recruiter skims 1,000 CVs to pick 100, then the hiring manager reads those 100 closely to choose 5 for interview.
Reranking 1,000 candidates to fix bad recall. A reranker raises precision at the top of what you hand it; if the answer is not in the candidate set, no depth helps. Fix Recall@k in first-stage retrieval first.
First-stage retrieval (BM25, Dense retrieval, Hybrid search) uses a Bi-encoder: documents are embedded once, so a query costs one embedding plus an Approximate nearest neighbour (ANN) lookup. A reranker is usually a Cross-encoder that runs full Attention over the concatenated (query, passage) pair, so query terms can attend to passage terms directly. That buys 5–15 nDCG@10 points on BEIR-style benchmarks over a bi-encoder alone, and costs one forward pass per candidate, so latency scales linearly with rerank depth.
The practical spectrum as of Oct 2026: cross-encoder/ms-marco-MiniLM-L6-v2 (22M params, English, 512-token limit) runs on CPU; BAAI/bge-reranker-v2-m3 (568M, multilingual, 8k tokens) needs a GPU; hosted APIs (Cohere Rerank, Voyage rerank, Jina) remove the ops; listwise LLM rerankers (RankGPT-style) are the most accurate and 10–50× the cost. Late-interaction models (ColBERT) sit between bi- and cross-encoders in cost and quality. Scores are model logits, not probabilities, so any cutoff threshold must be tuned on your own Golden set.
Hybrid retrieval → top 100 → bge-reranker-v2-m3 (FP16, max length 512, batch 32) on a GPU behind Hugging Face TEI or Infinity → keep top 5–8. If no GPU: ms-marco-MiniLM-L6-v2 exported to ONNX INT8 on CPU, depth 50. Truncate passages to 256–384 tokens for scoring even if you send longer text to the LLM. Log the reranker score of every chunk you send.
Developer-docs search, 4M chunks. Hybrid retrieval returns 100 candidates in 40 ms. Without reranking, the chunk that answers 'how do I rotate an API key without downtime' sits at rank 23 and never reaches the 6-chunk prompt. bge-reranker-v2-m3 on one L4 GPU scores all 100 pairs (≤384 tokens each) in ~250 ms and moves it to rank 2. Across a 400-query golden set, answer-bearing chunk in final top-6 goes from 71% to 86%; end-to-end p95 rises from 1.9 s to 2.2 s.
Say: default to a cross-encoder over the top 100 from hybrid retrieval, MiniLM-L6 on CPU if you are latency- or cost-bound, a 500M-class reranker on GPU if quality pays. Then show the knee: measure recall of the answer chunk in the final top-k at depths 25, 50, 100, 200 and stop where the curve flattens. Expect the follow-up 'what if p95 must stay under 300 ms?' Answer: drop depth to 50, quantise to INT8 ONNX, or distil the big reranker into the small one on your own click or judge labels.
Numbers worth knowing
- MiniLM-L6 cross-encoder latency
- 300–600 ms CPU, 30–60 ms GPU100 pairs × 256 tokens; 8 vCPU ONNX vs one L4/A10G
- bge-reranker-v2-m3 latency
- 200–400 ms100 pairs × 256 tokens, one L4/A10G, FP16; 3–8 s on CPU
- Typical quality gain
- +5 to +15 nDCG@10cross-encoder over bi-encoder top-100 on BEIR-style English sets
- Useful rerank depth
- 50–150 candidatesbeyond ~100 the curve is usually flat; measure your knee
How to decide
| When | Do this |
|---|---|
| p95 budget for retrieval < 150 ms and no GPU | MiniLM-L6 ONNX INT8 on CPU, depth 30–50 |
| Multilingual corpus or passages > 512 tokens | bge-reranker-v2-m3 on GPU, or a hosted multilingual rerank API |
| < ~600k searches/month and no ML platform team | Hosted rerank API; self-hosting does not pay for the on-call |
| Hard reasoning relevance (legal holdings, code semantics) and latency is not critical | LLM listwise rerank of the top 20 after a cross-encoder cut |
Cost maths
Hosted: assume ~$2 per 1,000 searches of up to 100 docs (illustrative list price, Oct 2026) → 1M searches/month = $2,000. Self-hosted: bge-v2-m3 at ~300 ms per search ≈ 3 searches/s per L4 ≈ 7.7M/month capacity. Two L4s for HA at ~$0.80/h each (illustrative on-demand) = ~$1,170/month flat. Break-even ≈ 600k searches/month, before counting engineer time.
Operating it
- Reranker p95 > 40% of the retrieval latency budget
- GPU utilisation > 70% at peak on the rerank pool
- Answer chunk present in top-100 but missing from final top-k on > 10% of failed golden-set queries
- Hosted rerank bill exceeds the cost of two always-on GPUs
- Reranker p95 latency; alert > 400 ms for 10 min
- Mean top-1 reranker score per day; alert on a > 20% drop (corpus or query drift)
- Rank displacement: share of final top-k that came from first-stage ranks > 50; alert if > 30% (first stage degrading)
- Timeout/fallback rate to first-stage order; alert > 1%
When it goes wrong
Answer quality on a multilingual support corpus dropped for German and Japanese queries only, a week after a 'latency win' release
Per-language slice of the nightly golden-set run: answer-chunk-in-top-6 fell from 84% to 61% for non-English, flat for English
The team swapped bge-reranker-v2-m3 for the English-only ms-marco MiniLM to save 250 ms; it scored non-English passages near-randomly
Route by detected language: MiniLM for English, bge-v2-m3 for everything else; added per-language slices to the release gate
Related terms
Where it shows up
Sources
- Passage Re-ranking with BERT — Nogueira & Cho, 2019paper
- cross-encoder/ms-marco-MiniLM-L6-v2 model card — Sentence Transformers, 2021docs
- BAAI/bge-reranker-v2-m3 model card — BAAI, 2024docs
- Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents — Sun et al., 2023paper
- BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models — Thakur et al., 2021paper
Check yourself
Three questions. Every option is explained once you answer.