Chunking
Splitting documents into passages small enough to embed and retrieve. Default: ~200–400 tokens on structural boundaries, no overlap.
Search systems rarely store whole documents. They cut each document into smaller passages called chunks, turn each chunk into an Embedding, and later fetch only the few chunks that match a question.
If chunks are too big, the important sentence gets diluted by everything around it. If they are too small, a chunk loses the context that makes it make sense. Good chunking cuts along the document's own structure, like headings and paragraphs, at a size you have tested on real questions.
Cutting a textbook into index cards for revision. One card per idea works; a card per chapter is too vague, and a card per sentence loses the thread.
Labelling the golden set by chunk id. The moment you change the chunker every label is wrong; label answer spans or documents instead.
Three numbers define a chunker: target size (tokens, measured with the embedding model's tokenizer), overlap, and the split hierarchy. The common default is a recursive splitter that tries headings, then paragraphs, then sentences, then characters until each piece fits. Chroma's 2024 evaluation (text-embedding-3-large, 5 retrieved chunks) found RecursiveCharacterTextSplitter at 200 tokens with no overlap consistently strong: recall 88.1%, token-level precision 7.0%, versus the 800-token/400-overlap setting at 85.4% recall and 1.5% precision. Recall sat in an 85–90% band across sizes; precision and context efficiency changed 4–5×.
Smaller chunks sharpen the embedding (one idea per vector) and cut tokens sent to the LLM; larger ones keep context but blur the vector and waste prompt budget. Structure matters more than size: tables, code blocks, list items and clause numbers should never be split mid-unit, which is why Layout-aware parsing comes first. When small chunks lose context, fix it with Parent-document retrieval (retrieve small, return big) or Contextual enrichment (breadcrumbs) rather than by growing the chunk. Every chunk carries Chunk metadata: doc id, section path, page, position, and the chunker version.
Parse to Markdown first (Docling or Unstructured), then a recursive splitter (LangChain RecursiveCharacterTextSplitter.from_tiktoken_encoder or LlamaIndex SentenceSplitter) at 256–400 tokens of the embedding model's tokenizer, 0 overlap, headings prepended as a breadcrumb, tables and code blocks kept whole up to 800–1,000 tokens. Store chunker name and version on every chunk.
Developer docs for an SDK, 12k pages. Fixed 1,000-character splits cut a code sample from its explanation; questions like 'how do I paginate list_users' retrieved prose without the snippet.
Switching to a Markdown-aware recursive splitter (H2 → H3 → paragraph → sentence, 300-token target, code fences kept whole up to 800 tokens) raised recall@5 on a 250-query golden set from 0.71 to 0.83 and cut average context from 4,100 to 1,900 tokens per answer.
Give a default and the evidence you'd use to move it: "I start with a structure-aware recursive splitter at ~300 tokens, no overlap, tables and code kept whole, and sweep 150/300/600 on the golden set. If small chunks lack context I add parent-document retrieval rather than making chunks bigger." The usual follow-up is "how do you evaluate chunking when chunk ids change every run?" Label relevance at the document-span level, not chunk id, so different chunkers can be scored on the same set.
Numbers worth knowing
- Recall vs size
- 88.1% @200 · 89.5% @400 · 85.4% @800(+400 overlap)Chroma 2024, text-embedding-3-large, 5 chunks
- Token precision vs size
- 7.0% @200 vs 1.5% @800same study; context efficiency differs ~5×
- Embedding model input limit
- 512 tokens (bge-base, e5) to 8,192 (bge-m3, text-embedding-3)chunks beyond it are truncated silently
How to decide
| When | Do this |
|---|---|
| Questions are factoid (a number, a clause, a parameter) | 200–300-token chunks; precision wins. |
| Answers need a full section (procedures, reasoning in filings) | Small chunks for search plus Parent-document retrieval to return the section. |
| Documents are tables or forms | One row or record per chunk with the header repeated; never split a row. |
Cost maths
Chunk size barely moves embedding cost (tokens are the same) but drives LLM cost. 1M answers/month × 5 chunks: at 300 tokens = 1,500 context tokens per answer; at 800 tokens = 4,000. At an assumed $3/M input tokens (illustrative, Oct 2026) that is $4,500 vs $12,000/month for the same recall.
Operating it
- Context tokens per answer above ~4k while recall is flat: shrink chunks or add a reranker
- Golden-set misses traced to split tables or code: move to layout-aware parsing
- Answers need adjacent chunks >30% of the time: add parent-document retrieval
- Chunk size distribution (p5/p50/p95 tokens): alert if p95 exceeds the embedding model's input limit
- Share of chunks that start mid-sentence or mid-table: alert above 5%
- Golden-set recall@k after any chunker change: gate rollout on no drop >1 point
When it goes wrong
A benefits-policy assistant quoted the wrong deductible for 'family plan' questions.
Error analysis of 40 flagged answers: 31 cited a chunk that began mid-table without the header row.
A 512-token fixed splitter cut a 3-page premium table into pieces; lower pieces lost the 'Individual / Family' column headers.
Layout-aware parsing to Markdown tables, one chunk per row group with headers repeated; recall@5 on table questions went from 0.52 to 0.89.
Related terms
Where it shows up
Sources
- Evaluating Chunking Strategies for Retrieval — Smith & Troynikov (Chroma), 2024blog
- Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models — Günther et al., 2024paper
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Lewis et al., 2020paper
Check yourself
Three questions. Every option is explained once you answer.