Recall@k
Of all the relevant documents for a query, the fraction that appear in the top k results. The ceiling on what a RAG system can answer.
Recall@k asks: of everything that should have been found, how much made it into the first k results? If 4 documents answer a question and 3 are in your top 10, recall@10 is 0.75. In Retrieval-Augmented Generation (RAG) it matters most because the Large Language Model (LLM) can only use what retrieval hands it; anything missed is invisible.
Fishing with a net: recall is how many of the fish in the pond ended up in the net, regardless of how much seaweed came with them.
Raising k until recall looks good. Recall@200 is meaningless if only 8 chunks reach the prompt; measure at the k you actually use.
recall@k = |relevant ∩ top-k| / |relevant|, averaged over queries. For single-answer Q&A, people often use hit rate@k (or success@k): 1 if any relevant item is in the top k. Recall is monotonically non-decreasing in k, so it says nothing about order; pair it with nDCG or MRR (mean reciprocal rank) for that. In two-stage systems measure it at the stage boundary: recall@100 for Candidate generation & the retrieval funnel (what the Reranker can work with) and recall@5–10 after reranking (what the LLM sees). It also needs complete relevance labels; with sparse labels, unlabelled relevant hits count as misses, which deflates absolute numbers but usually still ranks systems correctly. BEIR reports recall@100 alongside nDCG@10 for exactly this reason.
Report recall@100 (first stage) and recall@k_final (what the prompt sees) on the golden set, per slice. Use ranx or pytrec_eval rather than hand-rolled code. Pair with nDCG@10.
Internal wiki, golden set of 300 queries, 1–5 relevant pages each. First stage (hybrid): recall@10 = 0.62, recall@50 = 0.86, recall@100 = 0.91. After reranking the top 100 down to 8: recall@8 = 0.81. So 9% of relevant pages are never retrieved (fix in chunking or query rewriting), and 10% are retrieved but cut by the reranker (fix in reranker depth or model). The two gaps need different fixes, which is why you measure both.
Say: I report recall@100 at the first stage and recall@final-k after reranking, because they localise failures to different components. Then I add nDCG@10 for ordering. Expect: 'recall@100 is 0.95 but answers are bad': the problem is reranking, context assembly or generation, so look at recall@final-k and run an Oracle experiment.
Numbers worth knowing
- First-stage target
- recall@100 ≥ 0.90before investing in reranking or prompts
- Final-k target
- recall@5–10 ≥ 0.80single-hop Q&A; multi-hop needs per-hop recall
- Rerank loss
- recall@100 − recall@k_final> 10 points means depth or reranker model is the bottleneck
How to decide
| When | Do this |
|---|---|
| recall@100 < 0.85 | Fix first stage: hybrid search, query rewriting, chunking, metadata filters |
| recall@100 high, recall@8 low | Better reranker or larger final k within token budget |
| Both high but answers wrong | Context assembly or generation; run an oracle experiment |
Cost maths
Recall is cheap to compute once labels exist; the cost is labels (see golden set). Raising final k from 5 to 10 at 400 tokens/chunk adds 2k input tokens ≈ $0.006/query at $3/M; at 1M queries/month that is $6,000, so buy recall in retrieval before buying it with tokens.
Operating it
- recall@100 drops > 3 points after a corpus growth step (ANN parameters or sharding)
- Rerank loss > 10 points
- A new slice (language, doc type) under 0.75
- Nightly recall@100 and recall@k_final; alert on a > 3-point drop
- Canary-document retrieval probe; alert on any miss
- Recall by slice (age, language, doc type); alert if any slice < 0.75
When it goes wrong
Answers about recently added runbooks were 'not found' while older runbooks worked
Golden-set slice by document age: recall@100 0.93 for pages older than 30 days, 0.41 for newer
Incremental indexing wrote new vectors but the HNSW index was not refreshed on two of six shards
Fixed refresh job, added a freshness probe that queries a canary document every 5 minutes, and a per-age recall slice in nightly evals
Related terms
Where it shows up
Sources
- Introduction to Information Retrieval (ch. 8, Evaluation) — Manning, Raghavan & Schütze, 2008book
- BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models — Thakur et al., 2021paper
Check yourself
Three questions. Every option is explained once you answer.