Parent-document retrieval
Search over small chunks for precise matching, then hand the LLM the larger parent section that contains them. Retrieve small, return big.
Small chunks are good for finding the exact sentence that matches a question, but on their own they often lack the surrounding explanation. Parent-document retrieval keeps two sizes: small 'child' chunks that get searched, and bigger 'parent' sections that get sent to the model.
When a child chunk matches, the system looks up its parent and passes that instead, so the answer comes with its context.
Finding a quote with a book's index, then reading the whole page around it before you cite it.
Returning parents without a size cap. One mis-parsed 'section' becomes 30k tokens, blows the context budget and buries the answer in the middle.
Index children (100–300 tokens) with a parent_id in Chunk metadata; store parents (1–3k tokens, usually a section from the heading tree) in a document store keyed by id. At query time retrieve top-k children (k = 20–50), group by parent, rank parents by best or summed child score, and return the top 3–5 parents. Variants: sentence-window retrieval returns ±N sentences around the hit; auto-merging returns the parent only when enough of its children match (e.g. 3 of 8).
Trade-offs: parents multiply context tokens (5 parents × 2k = 10k tokens vs 1.5k for children), which raises LLM cost and invites Lost in the middle effects; so cap parents and order the best first. It also requires parents defined by structure, which depends on Layout-aware parsing. It is the main alternative to large chunks and to Chunk overlap, and pairs well with a Reranker that scores children before grouping.
Children of 150–250 tokens indexed with parent_id; parents = heading sections capped at 2,000 tokens (split larger ones); retrieve 30 children, group, return top 3 parents ordered best-first. LangChain ParentDocumentRetriever or LlamaIndex AutoMergingRetriever if you use those frameworks, otherwise ~40 lines over any vector DB plus a key-value store.
Financial filings (10-K), 50k filings. Question: 'What drove the change in gross margin in fiscal 2025?'
Children (200 tokens) find the sentence 'Gross margin decreased 140 basis points…' in Item 7. Alone it lacks the causes listed two paragraphs later. Grouping the top-30 children by parent returns the full 1,800-token 'Gross Margin' subsection of MD&A. Answer completeness on 150 analyst questions rose from 58% to 81%, while context grew from 1.2k to 6.5k tokens per answer.
Show the trade-off: "I retrieve 200-token children for precision and return up to three ~1.5k-token parents, grouped by best child score, so completeness goes up while context stays under ~6k tokens. Auto-merging only expands when several children of one parent hit." Expect: "what if the parent is a 40-page section?" Cap parent size, fall back to a sentence window, and fix the heading tree in parsing.
Numbers worth knowing
- Context per answer
- ~1.5k (children) → 4–6k tokens (3 parents)200-token children, 1.5–2k parents
- Children retrieved before grouping
- 20–50to fill 3–5 distinct parents
- Parent size cap
- 1.5–2k tokenskeeps 3 parents under ~6k context
How to decide
| When | Do this |
|---|---|
| Answers need surrounding explanation (procedures, filings, policies) | Parent-document retrieval with section parents. |
| Answers are single facts | Plain small chunks; parents only add tokens. |
| No reliable section structure | Sentence-window expansion (±3 sentences) instead of parents. |
Cost maths
Extra cost is LLM input tokens. 500k answers/month × (6k − 1.5k) extra tokens = 2.25B tokens × $3/M (assumed input price, Oct 2026) = $6,750/month. With prompt caching that does not apply here (contexts differ per query), so the cap on parent size is the main lever; 3 parents at 1.2k instead of 2k saves ~$2,400/month.
Operating it
- Golden-set answers judged incomplete >20% while the right chunk is retrieved: add parents
- Context tokens per answer >8k: cap parent size or count
- Input tokens per answer p50/p95: alert on p95 above 8k
- Parent size distribution: alert on any parent above the cap
- Distinct parents per answer: alert if median falls to 1 on multi-part questions
When it goes wrong
An HR-policy assistant's latency and cost tripled overnight with no traffic change.
Cost-per-query dashboard: median input tokens jumped from 5k to 21k.
A re-exported handbook lost its heading styles, so the parser treated the whole 60-page document as one parent section.
Hard 2k-token parent cap with fallback to a sentence window, and a parse check that alerts when a document yields fewer than 3 sections per 10 pages.
Where it shows up
Sources
- Lost in the Middle: How Language Models Use Long Contexts — Liu et al., 2023paper
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Lewis et al., 2020paper
Check yourself
Three questions. Every option is explained once you answer.