What is the impact of chunk overlap on retrieval?
- 1 min — what overlap insures against
- 2 min — costs and the 10–20% default
- 1 min — dedupe before assembly
- 1 min — structural chunking reduces the need
Heavy overlap on a 50-million-chunk corpus is a real cost line. On a 50-thousand-chunk corpus it's free. Scale the decision to the corpus.
Overlap is a patch for bad boundaries. If you chunk semantically — on headings, clauses, sentence boundaries — you need far less of it. And always deduplicate overlapping results before building context, or you'll spend tokens on the same paragraph three times and squeeze out a genuinely different source. Org scope: overlap is a corpus-scale cost decision — at tens of millions of chunks it belongs in the platform's capacity plan, not in each team's notebook.
Assumptions: 50M base chunks of 512 tokens, 1024-dim float32 vectors (4 KB each), $0.02 / M embedding tokens (illustrative, Oct 2026).
- 20% overlap → stride 0.8 → ~1.25× chunks = 62.5M vectors.
- Extra vectors: 12.5M × 4 KB ≈ 51 GB RAM (before HNSW graph overhead).
- Extra embedding: 12.5M × 512 = 6.4B tokens ≈ $128 one-off, repeated on every re-embed.
The embedding bill is noise; the RAM is the real cost.
What the answer sounds like at each level
Overlap helps when the answer is split across two chunks.
Quantifies the cost (storage, embedding, duplicate results), recommends 10–20%, and deduplicates before context assembly.
Treats overlap as a patch for bad boundaries, prefers structural chunking, and scales the decision to corpus size and memory budget.
Follow-ups the interviewer will push
1Now the corpus is 50M chunks and memory is the constraint.
Drop overlap toward 0–10% and switch to boundary-aware chunking. At 20% overlap you carry ~25% more vectors — for 1024-dim float32 that is ~50 GB extra RAM in the HNSW tier.
2Now top-5 often returns three near-identical chunks.
Deduplicate by source span or by MMR before assembly, and collapse adjacent hits into one window. That recovers two slots for genuinely different evidence.
Red flags interviewers listen for
- Recommends 50% overlap by default.
- Doesn't deduplicate overlapping hits.
- No mention of storage or embedding cost.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Trade-off | States recall gain vs storage, cost and duplicate-slot loss. |
| Scale | Ties the decision to corpus size with numbers. |
| Root cause | Identifies overlap as a patch for bad boundaries. |
Sources
- Dense Passage Retrieval for Open-Domain Question Answering — Karpukhin et al., 2020paper