Top-k (retrieval k vs context k)
How many results each stage returns. Recall-heavy stages want large k (100+); the context sent to the model wants small k (5–10).
Top-k is simply the number of results you ask a search to return: the k best matches. There isn't one k in a RAG system — there's one per stage. Early stages return many results (say 100) so nothing important is missed, and the final step passes only the best few (say 5–8) into the Large Language Model (LLM)'s prompt so it isn't buried in noise.
A playlist shortlist: grab 100 songs that might fit the party, then actually play the best 10.
One global k for everything. A single k=5 starves comparison questions; k=20 pollutes lookups. Use a large retrieval k, a reranker threshold, and a per-intent cap.
Each k trades recall against cost or noise. Retrieval k (50–200) is limited by latency and rerank cost. Context k (3–10 chunks, 2–8k tokens) is limited by model attention: Liu et al. (2023) showed accuracy drops by 20+ points when the answer sits in the middle of 20 documents versus at the start, so more chunks can lower quality. Practical refinements: a score threshold after reranking (drop chunks with cross-encoder probability < 0.2) so k adapts per query; dynamic k by intent (lookup questions 3, comparison questions 8–12); deduplication of near-identical chunks before counting; MMR (maximal marginal relevance, λ ≈ 0.7) for diversity. In pgvector the HNSW ef_search must be ≥ k or the index returns fewer rows.
Retrieval k = 100 (each hybrid leg), rerank top 50, reranker score threshold tuned on the golden set (start at 0.2–0.3), cap 8 chunks / ~6k tokens in context, near-duplicate removal before the cap.
Developer-docs assistant. Retrieval k=100 (hybrid), rerank to 20, threshold 0.25 → a lookup like 'what's the default timeout for the HTTP client?' keeps 2 chunks (one answer, one changelog note). 'Compare v2 and v3 auth flows' keeps 9. Before thresholding, a fixed k=10 put 8 unrelated chunks next to the 2 relevant ones and the model blended a deprecated v2 default into the answer.
Separate the ks: "k=100 into the reranker for recall, then a threshold plus a cap of ~8 into the prompt for precision; I tune the cap on answer faithfulness, not on retrieval metrics." Expect: why not stuff 50 chunks into a 200k context? Cost scales linearly, and lost-in-the-middle plus distractors lower accuracy.
Numbers worth knowing
- Mid-context accuracy drop
- 20+ pointsanswer in the middle of 20 docs vs first position (Liu et al. 2023)
- Context k sweet spot
- 5–8 chunksreranked, 300–800-token chunks, typical QA
- Prompt cost per extra chunk
- ~500 tokens × input pricee.g. $0.0015 per chunk at $3/M input
How to decide
| When | Do this |
|---|---|
| Lookup / factoid intent | Cap at 3–5 after threshold. |
| Comparison, summary or multi-hop intent | Cap at 8–12 and enforce diversity (MMR or per-source quotas). |
| Reranker scores all below threshold | Return zero chunks and let the model abstain. |
Cost maths
Context cost = k × tokens per chunk × input price × queries. 1M queries/month × 10 chunks × 500 tokens = 5B input tokens; at an assumed $3 per M input tokens (illustrative Oct 2026) = $15,000/month. Dropping to 6 chunks saves $6,000/month; Prompt caching does not help here because retrieved chunks differ per query.
Operating it
- Answer faithfulness flat while k rises — you're adding distractors; lower the cap
- Input tokens per query > 8k p95 — enforce a token budget, not a chunk count
- Abstention rate on in-scope questions > 10% — threshold too high or retrieval k too low
- Chunks and tokens in context per query (p50/p95) — alert if p95 tokens > budget
- Share of queries with zero chunks above threshold — alert above 10%
- Faithfulness on golden set by k bucket — alert on −3 points
When it goes wrong
After raising context k from 5 to 20 to 'give the model more to work with', a policy assistant began citing superseded policies.
Faithfulness eval flagged 14% of answers citing documents marked superseded, up from 2%.
The extra 15 chunks included older policy versions with similar wording; the model merged them.
Reverted to a cap of 6 with a reranker threshold, filtered status = current in metadata, and added superseded-document citations as a tracked eval metric.
Related terms
Where it shows up
Sources
- Lost in the Middle: How Language Models Use Long Contexts — Liu et al., 2023paper
Check yourself
Three questions. Every option is explained once you answer.