When should you use a reranker after vector search?
- 1 min — bi-encoder vs cross-encoder: separate vs joint encoding.
- 2 min — diagnostic: Recall@50 vs Recall@5 gap.
- 2 min — latency budget, candidate-count sweep, model size choice.
- 1 min — build vs buy and caching.
Bi-encoders embed query and document separately, so the model never sees them together — fast, but it can't reason about the interaction. A cross-encoder reranker reads query and document jointly and scores relevance directly. Much more accurate, far too slow to run over the whole corpus. So: retrieve 50–200 candidates cheaply, rerank them expensively, keep the top 5.
The diagnostic is the Recall@50-vs-Recall@5 gap. A large gap means the right answers are in the pool but badly ordered — exactly what a reranker fixes. A small gap means retrieval isn't finding them at all, and a reranker will do nothing.
Budget it: on one GPU, a cross-encoder over 100 candidates of ~256 tokens costs roughly 50–300ms — the low end for a small model like ms-marco-MiniLM-L6 (22M), the high end for bge-reranker-v2-m3 (568M). If you're at a 2-second SLA with a 1-second LLM call, that's affordable. If you're doing agentic multi-hop retrieval with five rounds, it isn't. Options: rerank fewer candidates, use a smaller distilled reranker, or cache by query-document pair.
Org scope: buy before build — a hosted rerank API (Cohere Rerank, Voyage) gets a team to a measured answer in a day; self-host on the shared inference platform once volume makes the per-query fee exceed a GPU's monthly cost.
Assumptions: 2.0s p95 end-to-end SLO; LLM generation 1.0s p95; retrieval (hybrid) 120ms; network and assembly 150ms.
- Remaining budget for reranking: 2000 − 1000 − 120 − 150 ≈ 730ms, but keep 40% headroom for p99 tails → ~440ms.
- bge-reranker-v2-m3 (568M) on one L4/A10G: ~2–3ms per 256-token pair batched → 100 candidates ≈ 200–300ms. Fits.
- Five-hop agentic loop: 5 × 250ms = 1.25s of reranking alone. Does not fit → MiniLM-L6 at ~40ms/100 pairs, or rerank only the final hop.
- Throughput: 100 QPS × 64 candidates = 6,400 pairs/s; at ~2,500 pairs/s per GPU for MiniLM-class models → 3 GPUs + 1 for headroom (~$1/GPU-hour illustrative on-demand A10G, Oct 2026 → ~$2.9k/month).
What the answer sounds like at each level
Use a cross-encoder reranker on the top 50–100 vector results to improve ordering, then pass the top 5 to the LLM.
Decides from the Recall@50 vs Recall@5 gap, sweeps candidate count to find the quality plateau, and states the latency cost per model size against the SLO.
Frames reranking as a latency-budget allocation across the whole request, decides hosted vs self-hosted from volume, and makes the candidate-count sweep a shared platform default.
Follow-ups the interviewer will push
1Traffic grows 10×. Does the reranker still fit?
Rerank cost scales with QPS × candidates. At 500 QPS × 64 candidates = 32k pairs/s; a MiniLM-L6 cross-encoder sustains a few thousand pairs/s per A10G with batching, so ~10 GPUs. Cut candidates to the plateau, distil, cache query-document pairs, and shed the reranker first under overload.
2Recall@50 is 0.92 but Recall@5 is 0.55. What's your first move?
That 37-point gap is exactly the reranker's job — add a cross-encoder over the top 50 and expect most of the gap back. If it's 0.60 vs 0.55 instead, the reranker won't help; fix retrieval (hybrid, chunking, fine-tuning).
3Could you use the LLM itself as the reranker?
Yes (listwise LLM reranking), and it can beat cross-encoders on reasoning-heavy queries, but it's 5–20× the latency and token cost. Use it for low-QPS, high-value queries; keep a cross-encoder for the default path.
Red flags interviewers listen for
- Proposes running a cross-encoder over the whole corpus.
- Adds a reranker without checking whether the right document is in the candidate pool.
- Gives no latency number or model size.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Diagnosis | Uses the recall gap to decide whether a reranker can help at all. |
| Latency | Gives ms per candidate set by model size and fits it into an SLO. |
| Tuning | Sweeps candidate count to the quality plateau rather than using a default. |
Buying quality back from latency
Illustrative exampleWe added a cross-encoder reranker to our search assistant. Answer quality improved clearly on the eval set, but p95 latency went from about 1.4s to 2.1s and we breached the SLA the product team had committed to.
Keep the quality gain, get back under budget. Rolling back was the easy option and I didn't want to take it.
I profiled the stage and found two things: we were reranking 200 candidates when a sweep showed quality plateaued around 60, and reranking ran strictly after retrieval completed on both indexes even though the dense results arrived much earlier. I cut the candidate pool to 64, started reranking dense results while the lexical leg was still running and merged late arrivals, and added a semantic cache on query-document pairs that hit on roughly 30% of traffic from repeated support questions. I also distilled a smaller reranker and kept the large one behind a flag for a quality-sensitive tenant.
Sources
- Passage Re-ranking with BERT — Nogueira & Cho, 2019paper
- ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT — Khattab & Zaharia, 2020paper