How would you choose the value of top-k?
- 1 min — the two k's and why they differ.
- 2 min — sweep context k against faithfulness; show the peak.
- 2 min — dynamic k with a calibrated threshold and cap.
There are two different k's and conflating them is the common mistake. Retrieval k (candidates into the reranker) should be generous — 50 to 200 — because recall lost here is unrecoverable. Context k (chunks into the LLM) should be small — typically 3 to 8 — because extra weak chunks dilute attention and cost tokens.
Tune context k empirically against answer quality, not retrieval metrics; they peak at different values. Recall keeps rising with k while answer faithfulness starts falling, and the right choice is at the answer-quality peak.
Make k dynamic rather than fixed. A score-threshold cutoff — include chunks until the reranker score drops below a calibrated bar, capped at a maximum — adapts to query difficulty. A factual lookup needs one chunk; a comparison question needs six. Fixed k gives both the same treatment and serves neither well.
Org scope: k is a per-product config owned by the product team, but the sweep harness and the calibrated threshold belong to the platform, re-run whenever the reranker or LLM changes.
What the answer sounds like at each level
Pick k around 5 for the LLM context and adjust based on answer quality.
Separates retrieval k (50–200) from context k (3–8) and sweeps context k against answer faithfulness, not recall.
Makes k adaptive via a calibrated reranker threshold, prices it in tokens, and owns re-tuning when models change.
Follow-ups the interviewer will push
1You move to a 1M-token context model. Should context k go to 100?
No. Long-context models still lose accuracy on evidence in the middle of long inputs (lost in the middle), and 100 × 500 tokens = 50k input tokens per query is ~17× the cost of k = 6. Re-run the sweep; you may gain a little at k = 10–15, rarely beyond.
2What does k cost in money?
Each 500-token chunk at an assumed $3/M input tokens is $0.0015 per query. Going from k = 5 to k = 15 adds $0.015/query — $15k/month at 1M queries/month — for a gain you have to show on the eval.
Red flags interviewers listen for
- Uses one k for both retrieval and context.
- Tunes context k on Recall@k instead of answer quality.
- Assumes bigger context windows make k irrelevant.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Distinction | Clearly separates retrieval k from context k with ranges. |
| Method | Tunes on answer quality and explains why recall and quality peak differently. |
| Cost | Translates k into tokens and money per query. |
Sources
- Lost in the Middle: How Language Models Use Long Contexts — Liu et al., 2023paper