Chunk overlap
Repeating the last N tokens of one chunk at the start of the next so ideas cut at a boundary survive. Often unnecessary with structural splits.
When a document is cut into chunks, a sentence or idea can be split across two pieces. Overlap copies a little text from the end of one chunk into the start of the next, so an idea near a boundary appears whole in at least one chunk.
It costs extra storage and can return near-duplicate chunks, so it helps most when you cut at fixed lengths and least when you already cut along paragraphs and headings.
Overlapping roof tiles: each tile covers the top edge of the one below so rain cannot slip through the gap.
Treating overlap as free insurance. It multiplies vectors by 1/(1 − overlap fraction) and fills top-k with near-duplicates, which can lower answer quality.
Overlap is a patch for boundary loss from fixed-length splitting. With 400-token chunks and 200 overlap, storage and embedding cost double (each token lives in ~2 chunks) and top-k results often contain adjacent near-duplicates, which wastes context slots and can push distinct evidence out.
Evidence as of Oct 2026 is mixed and model-dependent. Chroma's 2024 evaluation with text-embedding-3-large found removing overlap raised recall at 400 tokens (88.1% → 89.5%) and nearly doubled precision; with the smaller all-MiniLM-L6-v2, 250-token chunks with 125 overlap reached 82.4% recall vs 77.1% without. So overlap matters more for weak embedders and fixed-length splits. Better alternatives: split on structure so boundaries fall between ideas, use sentence-window or Parent-document retrieval to expand context at read time, or Contextual enrichment (breadcrumbs). If you keep overlap, deduplicate adjacent chunks in Context assembly by merging consecutive chunk ids from the same document.
0 overlap with structure-aware splitting plus read-time expansion (neighbouring chunks or parent section). 10–15% overlap (e.g. 40 of 300 tokens) only with fixed-length splits or a ≤100M-parameter embedder. Merge adjacent chunk ids in context assembly either way.
Insurance claims notes, 2M chunks, MiniLM embedder on CPU. A clause like 'Water damage is excluded unless caused by a sudden pipe burst' straddled a 256-token boundary in 9% of policies.
With 0 overlap: recall@5 on 300 coverage questions = 0.74. With 64-token overlap (25%): 0.79, storage +25%. After moving to sentence-aware splitting with 0 overlap: 0.81, storage back to baseline.
Answer with a default and a test: "With a structure-aware splitter I default to zero overlap and expand context at read time. With fixed-length splits or a small embedder I use 10–20% overlap and dedupe adjacent hits in assembly. I verify with recall and with context tokens per answer." Expect: "what does 50% overlap cost?" Double the vectors, double the embedding bill, and frequent duplicate hits in top-k.
| Setting (tokens) | Recall | Precision | Vectors |
|---|---|---|---|
| 400, 200 overlap | 88.1% | 3.3% | 2.0× |
| 400, 0 overlap | 89.5% | 3.6% | 1.0× |
| 250, 125 (MiniLM) | 82.4% | n/a | 2.0× |
| 250, 0 (MiniLM) | 77.1% | n/a | 1.0× |
Numbers worth knowing
- Storage multiplier
- 1 / (1 − overlap fraction): 1.25× at 20%, 2× at 50%fixed-length windows
- Recall effect, strong embedder
- 89.5% (0 overlap) vs 88.1% (50%)Chroma 2024, 400 tokens, text-embedding-3-large
- Recall effect, small embedder
- 82.4% (50%) vs 77.1% (0)Chroma 2024, 250 tokens, all-MiniLM-L6-v2
How to decide
| When | Do this |
|---|---|
| Splitting on headings and paragraphs | No overlap; expand neighbours at read time if needed. |
| Fixed-length splitting of unstructured text (transcripts, OCR) | 10–20% overlap and adjacent-chunk dedupe. |
| Top-k shows >30% adjacent duplicates | Cut overlap or merge adjacent hits before the LLM sees them. |
Cost maths
Vectors = tokens / (size − overlap). 2B corpus tokens at 300/0 = 6.7M vectors; at 300/150 = 13.3M vectors. At 1024-dim int8 plus graph ≈ 1.2 KB/vector, that is 8 GB vs 16 GB RAM, and embedding cost $40 vs $80 at $0.02/M tokens (text-embedding-3-small, Oct 2026).
Operating it
- Boundary-split answers exceed 5% of golden-set misses: add overlap or switch to structural splits
- Duplicate-adjacent rate in top-10 above 30%: reduce overlap
- Adjacent-duplicate share in top-k: alert above 30%
- Distinct source documents per answer: alert if median falls below 2 on multi-source questions
When it goes wrong
A call-centre transcript search returned five nearly identical snippets for every query, and answers missed the second half of the call.
Context review: 4 of 5 retrieved chunks were consecutive windows of the same 90 seconds.
50% overlap on 200-token windows; the top-5 was saturated by one passage's neighbours.
Overlap to 15%, merge adjacent chunk ids per call in assembly, cap 2 chunks per call in top-k; distinct-evidence rate rose from 1.4 to 3.6 sources per answer.
Sources
- Evaluating Chunking Strategies for Retrieval — Smith & Troynikov (Chroma), 2024blog
Check yourself
Three questions. Every option is explained once you answer.