Context assembly
Turning ranked chunks into the prompt: dedupe, fit a token budget, order, label with source ids, and add the instructions around them.
After retrieval you have a ranked list of text pieces (chunks). Context assembly decides which ones actually go into the Prompt & system prompt, in what order, with what labels, and how much room each gets. It removes duplicates, stops before the token budget runs out, and tags each chunk with an id so the model can cite it.
Packing a carry-on: you can only fit so much, so you drop duplicates, put the essentials where you can reach them, and label the bags.
Treating 'top-k = 10' as the design. Ten 80-token chunks and ten 2,000-token chunks are different prompts; budget in tokens and let scores decide how many fit.
Steps, in order: (1) dedupe near-identical chunks (same content_hash, or MinHash/Jaccard > 0.9) so three copies of a policy do not take three slots; (2) merge adjacent chunks from the same document and optionally expand to the parent section (Parent-document retrieval); (3) cut to a token budget by reranker score, not by a fixed k; (4) order: highest-scoring first, second-best last, weakest in the middle, which hedges against Lost in the middle position effects; (5) wrap each chunk with a stable id, title, date and source (<doc id="S3" title="…" date="2026-08-01">) so Attribution & citations and Grounding & groundedness checks checks can work; (6) put long static instructions and few-shot examples before the retrieved block so Prompt caching can reuse the prefix. Prepending a one-line document context to each chunk at index time (Contextual enrichment (breadcrumbs)) cut retrieval failures by 35–49% in Anthropic's 2024 contextual retrieval tests.
Budget 3–6k tokens of retrieved context for single-hop Q&A, 8–15k for multi-document synthesis. Dedupe by content_hash then MinHash. Order best-first, second-best-last. Wrap chunks in XML-style tags with id, title, source, date. Static system prompt and examples first, retrieved context next, user question last; mark the static prefix for prompt caching.
Clinical intake assistant answering 'which anticoagulant was the patient on before admission?'. Reranker returns 20 chunks; 4 are duplicate copies of the same medication list from three systems. After dedupe, 16 remain; budget is 4,000 tokens; the top 7 fit. Order: med-reconciliation note (score 0.94) first, discharge summary (0.88) last, five weaker chunks between. Each is wrapped with source system, note type and timestamp, so the answer can say 'apixaban 5 mg BID per med-rec note S1, 2026-09-14'.
Say: I budget by tokens, not by k, dedupe before cutting, order best-first and second-best-last, and label every chunk with an id and date. Then I prove each step with an ablation on the golden set. Expect the follow-up 'why not send all 20?' Answer: cost scales linearly, distractors measurably lower accuracy on multi-hop questions, and p95 time-to-first-token grows with input length.
Numbers worth knowing
- Retrieved-context budget
- 3–6k tokenssingle-hop Q&A; 8–15k for multi-document synthesis
- Contextual retrieval gain
- 35–49% fewer retrieval failuresAnthropic 2024 tests; 49% with BM25 + contextual embeddings, 67% with reranking added
- Duplicate share in enterprise corpora
- 10–30% of retrieved slotswikis and document stores without dedupe at ingestion
How to decide
| When | Do this |
|---|---|
| Chunks often cut mid-table or mid-clause | Expand to parent section or neighbours before budgeting |
| Same instructions on every request | Put them first and cache the prefix; retrieved text goes after |
| Answers must cite sources | Stable short ids per chunk (S1…S8) plus a citation check |
Cost maths
Input cost per query = (static + retrieved + question) tokens × price. Assume $3/M input, cache reads 0.1×, writes 1.25× (Anthropic 5-min TTL, illustrative Oct 2026). Naive: 1.5k static + 10k retrieved = 11.5k × $3/M = $0.0345. Assembled with cached prefix: 1.5k × $0.30/M + 4k × $3/M = $0.0125. At 1M queries/month: $34.5k → $12.5k.
Operating it
- Mean retrieved tokens per prompt > 2× the budget you designed for
- Duplicate rate in assembled context > 10%
- Cache hit rate on the static prefix < 70% (ordering is breaking the prefix)
- Retrieved tokens per prompt p50/p95; alert if p95 > 1.5× budget
- Duplicate-chunk rate in assembled context; alert > 10%
- Prompt cache hit rate on the static prefix; alert < 70%
- Share of answers citing a chunk outside the top 3; trend, not alert
When it goes wrong
An HR policy bot quoted a superseded 2023 parental-leave policy about one time in five
Spot-check of answers citing policy documents found mismatched effective dates; confirmed by a 'which version' slice in the golden set
Both versions were retrieved; context assembly stripped dates and ordered by score, and the older doc's wording happened to match the query better
Effective date and superseded_by added to chunk metadata; superseded versions filtered pre-retrieval; dates rendered in every chunk header
Related terms
Where it shows up
Sources
- Introducing Contextual Retrieval — Anthropic, 2024blog
- Lost in the Middle: How Language Models Use Long Contexts — Liu et al., 2023paper
- Prompt caching — Anthropic, 2025docs
Check yourself
Three questions. Every option is explained once you answer.