Candidate generation & the retrieval funnel
The cheap, high-recall first stage that narrows millions of items to a few hundred for an expensive reranker to order.
Candidate generation is the first step of a multi-stage search: quickly pull a few hundred possibly-relevant items out of millions. A slower, more accurate model (a Reranker) then reorders just those. The first stage's job is not to be precise — it is to make sure the right answers are somewhere in the shortlist.
A recruiter scanning 10,000 CVs for the right keywords and experience to make a shortlist of 50, so the hiring manager only interviews those.
Evaluating only the final ranking. If end-to-end nDCG is low, you can't tell whether the reranker is ordering badly or never saw the right documents. Always log and measure recall at the handoff.
The pipeline is a funnel with a cost per item rising 100–1000× at each stage: BM25/ANN (microseconds per item over millions) → Cross-encoder (milliseconds per item over 50–200) → optionally an LLM judge (hundreds of ms over 5–20). The first stage is judged by recall@N at the handoff size, because the reranker can never recover a document that was not passed to it. Candidate sources are usually several, unioned or fused: BM25, dense, recency, popularity, a 'same author/same SKU family' source; YouTube's recommender (Covington et al. 2016) is the classic description of generating hundreds of candidates from millions before ranking. Picking N: plot reranked nDCG@10 against N; it rises then flattens (often by N = 50–100) while rerank latency grows linearly.
Hybrid (BM25 + dense) to 100–200 fused candidates; rerank 50–100 with a cross-encoder; pass 5–10 to the LLM. Log candidate ids at each stage so recall can be computed offline.
Insurance-claims triage search, 18M claim notes. Stage 1: BM25 top 150 + dense top 150 → RRF → 200 unique (14 ms). Recall@200 on 400 labelled queries = 0.94. Stage 2: bge-reranker-v2-m3 on 200 pairs = 410 ms on one L4 — too slow. Sweep: N=50 → nDCG@10 0.71, 110 ms; N=100 → 0.73, 210 ms; N=200 → 0.735. Ship N=75 at 160 ms.
Frame it as a recall contract: "Stage 1 owns recall@100, stage 2 owns precision@10; I measure each separately so I know which one to fix." Expect: how do you choose 100 vs 500? Show the nDCG-vs-N curve and the latency line, and pick the knee under the latency budget.
Numbers worth knowing
- Stage-1 recall target
- ≥ 0.9 at the handoff Nlabelled golden set; below this the reranker is starved
- Cross-encoder cost
- ~2–4 ms per pair568M reranker, 512 tokens, batched on one L4
- Typical knee
- N = 50–100reranked nDCG@10 flattens; latency keeps rising linearly
How to decide
| When | Do this |
|---|---|
| Recall@N < 0.9 | Fix stage 1 (add a leg, rewrite queries, widen N) before touching the reranker. |
| Recall@N ≥ 0.9 but nDCG@10 low | Improve the reranker or its inputs (titles, metadata in the pair text). |
| Latency budget exceeded | Shrink N to the knee or use a distilled reranker. |
Cost maths
Rerank compute = N × per-pair cost. N=100 × 3 ms = 300 ms of L4 time per query; at ~$0.80/hr per L4 (illustrative, Oct 2026) that is $0.000067/query, ~$67 per million queries. Halving N to 50 halves the bill and usually costs < 1 nDCG point.
Operating it
- Reranker GPU utilisation > 70% at peak — cut N or add GPUs
- Recall@N falls > 3 points after a corpus change — retune stage 1
- A third candidate source (recency, popularity) appears in user complaints — add it as a fused leg
- Recall@N at the handoff on the golden set — alert below 0.9
- Rerank latency p95 — alert above budget (e.g. 200 ms)
- Share of final top-10 from each candidate source — alert if any source drops to 0
When it goes wrong
Switching to a stronger reranker produced no improvement in answer quality, despite +6 nDCG points on a public benchmark.
Offline breakdown: recall@50 at the handoff was 0.68; the reranker was ordering a list that lacked the answer a third of the time.
Stage 1 was dense-only with a general model; financial-filing queries with ticker symbols and line-item names were missed.
Added a BM25 leg with RRF and raised N to 100; recall@100 rose to 0.93, then the new reranker delivered +5 nDCG@10.
Where it shows up
Sources
- Deep Neural Networks for YouTube Recommendations — Covington, Adams & Sargin, 2016paper
- Passage Re-ranking with BERT — Nogueira & Cho, 2019paper
Check yourself
Three questions. Every option is explained once you answer.