Structural & semantic chunking
Cutting where the topic changes, detected by a drop in embedding similarity between neighbouring sentences, instead of at fixed lengths.
Instead of cutting text every N words, semantic chunking reads it sentence by sentence and starts a new chunk when the topic shifts. It detects the shift by comparing each sentence's Embedding with the next one: a sharp drop in similarity suggests a new subject.
The result is chunks that each hold one idea. It costs more compute at ingestion, and on well-structured documents with clear headings it often performs no better than simply cutting at those headings.
A good editor splitting a long speech into paragraphs by listening for where the speaker changes subject, not by counting words.
Adopting semantic chunking on well-structured docs and calling the gain obvious. Without a structural baseline on the same golden set, you often pay 10× ingest compute for zero recall.
Breakpoint method (popularised by LangChain's SemanticChunker): split into sentences, embed each (often with a window of ±1 sentence), compute Cosine similarity between neighbours, and cut where distance exceeds a percentile threshold (95th by default) or a standard-deviation rule. Add min and max size guards so you never emit a 20-token or a 2,000-token chunk. Chroma's ClusterSemanticChunker instead optimises a global objective over sentence similarities and scored the best precision in their 2024 study (8.0% at 200 tokens vs 7.0% for a recursive splitter); an LLM-driven chunker scored the best recall (91.9%).
Costs: one embedding call per sentence at ingest (5–20× more embedding calls than chunks), and thresholds that need tuning per corpus. Gains are largest on unstructured text: transcripts, chat logs, OCR without headings, long emails. On Markdown or HTML with real headings, structural splitting usually matches it for free. An alternative for long-context embedders is Contextual enrichment (breadcrumbs) or late chunking (embed the whole document, then pool token vectors per chunk), which keeps cross-chunk context without guessing breakpoints.
LangChain SemanticChunker or LlamaIndex SemanticSplitterNodeParser with a small fast embedder (bge-small-en-v1.5 or all-MiniLM-L6-v2) for breakpoint detection, buffer of 1 sentence each side, 90–95th percentile threshold, 100-token minimum and 500-token maximum; then embed the final chunks with your main model. Only for corpora without reliable headings.
Clinical intake transcripts (doctor–patient audio → text), no headings, 45 minutes each. Fixed 300-token chunks mixed medication history with family history in 40% of chunks.
Semantic chunking (bge-small sentence embeddings, 3-sentence window, 92nd-percentile breakpoint, 100–450-token guards) produced topic-pure chunks; recall@5 on 200 clinician questions went 0.66 → 0.77. Ingest cost: 18k sentence embeddings per 1k transcripts instead of 1.2k chunk embeddings, about 2 extra GPU-minutes per 1k transcripts.
Position it as a tool for unstructured text: "If the document has headings, I split on them. For transcripts and OCR I use semantic breakpoints with min/max guards, and I compare against a recursive baseline on the golden set before paying the ingest cost." Expect: "how do you set the threshold?" Sweep the percentile (85–97) and pick by recall@k and chunk-size spread, not by eyeballing.
| s1 | s2 | s3 | s4 | s5 | s6 | |
|---|---|---|---|---|---|---|
| s1 meds | 1.00 | 0.82 | 0.78 | 0.31 | 0.28 | 0.25 |
| s2 meds | 0.82 | 1.00 | 0.85 | 0.34 | 0.30 | 0.27 |
| s3 meds | 0.78 | 0.85 | 1.00 | 0.38 | 0.33 | 0.30 |
| s4 family | 0.31 | 0.34 | 0.38 | 1.00 | 0.81 | 0.77 |
| s5 family | 0.28 | 0.30 | 0.33 | 0.81 | 1.00 | 0.84 |
| s6 family | 0.25 | 0.27 | 0.30 | 0.77 | 0.84 | 1.00 |
Numbers worth knowing
- Extra ingest embedding calls
- 5–20× chunk countone call per sentence (window)
- Best precision in Chroma study
- 8.0% (ClusterSemanticChunker, 200 tok) vs 7.0% recursivetext-embedding-3-large, 2024
- LLM chunker recall
- 91.9% vs 88–89.5% recursivesame study; far higher ingest cost
How to decide
| When | Do this |
|---|---|
| Documents have real headings (Markdown, HTML, well-tagged PDFs) | Structural splitting; skip semantic. |
| Transcripts, chats, OCR output, long emails | Semantic breakpoints with min/max guards. |
| Embedder accepts 8k+ tokens and docs fit | Try late chunking or contextual enrichment as an alternative. |
Cost maths
Extra ingest = sentences × tokens per window × price. 100k transcripts × 600 sentences × 60 tokens = 3.6B tokens. Self-hosted bge-small on one L4 at ~5k windows/s (estimate) ≈ 3.3 hours ≈ $3; via text-embedding-3-small at $0.02/M ≈ $72. The main chunk embeddings are a separate, unchanged cost.
Operating it
- More than ~20% of chunks mix topics on a manual sample of 50: consider semantic chunking
- Ingest throughput below source arrival rate: move breakpoint detection to a smaller model or batch GPU job
- Chunk length p95 vs embedder input limit: alert if p95 exceeds it
- Breakpoints per 1k tokens per source: alert on a 2× shift (format change upstream)
When it goes wrong
After enabling semantic chunking on a legal corpus, some chunks were 4,000 tokens and their answers were vague.
Chunk-size histogram showed a long tail; those chunks were truncated at the embedder's 512-token limit.
Uniform boilerplate (recitals, definitions) never crossed the similarity threshold, so no breakpoint fired; no max-size guard was set.
Added a 450-token max with recursive fallback, and switched clause-numbered contracts back to structural splitting.
Sources
- Evaluating Chunking Strategies for Retrieval — Smith & Troynikov (Chroma), 2024blog
- Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models — Günther et al., 2024paper
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks — Reimers & Gurevych, 2019paper
Check yourself
Three questions. Every option is explained once you answer.