Choose your chunk size by measurement, not folklore
A reproducible method: build a small golden set on a public corpus, sweep chunk sizes, and read recall@k, evidence coverage and context cost off one table.
“Use 512 tokens with 50 overlap” is the most copied line in RAG tutorials, and it is a guess. The right chunk size depends on how long your answers are, how your documents are structured, which retriever you use, and how many chunks you can afford to put in the prompt. You can measure all of that in an afternoon.
The method has four steps: pick a corpus, write a golden set with evidence spans, sweep sizes with everything else frozen, and read three numbers together. The harness below runs on a toy inline corpus so you can see every moving part; the same code points at a real corpus by swapping one dictionary.
Step 1: a corpus you can share
Start on a public corpus that looks like yours, so the experiment is reproducible and reviewable: a BEIR dataset such as SciFact or FiQA for short-answer factoid content, the Python or Kubernetes docs for developer documentation, or SEC 10-K filings from EDGAR for long financial prose. Then rerun on a 200-document sample of your own data. If the two disagree, trust yours; the public run exists to debug the harness.
Step 2: a golden set with evidence spans
Write 50–100 questions. For each, record the exact sentence or span that contains the answer, not the document id. Document-level labels cannot tell chunk sizes apart: every size “finds the document”. Span labels can, because a chunk either contains the whole span or it does not. Spread the questions across short facts (a limit, a version number), multi-sentence explanations, and table lookups, roughly in the proportion your users ask them. See golden sets.
Step 3: sweep one variable
Freeze the retriever, the embedding model, k and the overlap ratio; vary only size. The harness uses stdlib BM25 so it runs anywhere; replace bm25_index with your real retriever when you move to your corpus. Overlap is held at 25% of size so it scales with the chunk.
"""Chunk-size sweep harness: BM25 retrieval, recall@k and evidence coverage per chunk size."""
import math
import re
from collections import Counter
CORPUS = {
"install.md": (
"Installing the CLI. The tool ships as a single static binary for Linux, macOS and Windows. "
"Download the archive for your platform and place the binary on your PATH. "
"On macOS the first launch may be blocked by Gatekeeper; allow it under Privacy and Security. "
"The minimum supported glibc version on Linux is 2.31. "
"To verify the download, compare the SHA-256 checksum with the value published on the releases page. "
"Upgrades replace the binary in place and keep your configuration file untouched."
),
"config.md": (
"Configuration lives in a TOML file at ~/.config/tool/config.toml. "
"Every key can be overridden by an environment variable with the TOOL_ prefix. "
"The sync interval defaults to 300 seconds and cannot be set below 30 seconds. "
"Set log_level to debug to print every HTTP request the client makes. "
"Proxy settings are read from HTTPS_PROXY; the tool does not read the lowercase https_proxy variable. "
"Profiles let you keep separate credentials per workspace; switch with the --profile flag."
),
"limits.md": (
"Rate limits and quotas. Each workspace may make 600 API requests per minute. "
"Requests above the limit receive HTTP 429 with a Retry-After header in seconds. "
"The client retries 429 responses up to five times with exponential backoff starting at one second. "
"Uploads larger than 5 GB must use the multipart command, which splits files into 64 MB parts. "
"A single multipart upload may contain at most 10,000 parts. "
"Deleted files stay recoverable for 30 days before they are purged."
),
}
# Golden set: question -> evidence span that must be retrieved intact to answer.
GOLDEN = [
("What is the default sync interval and its minimum?", "defaults to 300 seconds and cannot be set below 30 seconds"),
("Why is my lowercase https_proxy ignored?", "does not read the lowercase https_proxy variable"),
("How many requests per minute can a workspace make?", "600 API requests per minute"),
("How does the client handle HTTP 429?", "retries 429 responses up to five times with exponential backoff"),
("What size part does multipart upload use?", "splits files into 64 MB parts"),
("Which glibc version is required on Linux?", "minimum supported glibc version on Linux is 2.31"),
("How long can deleted files be recovered?", "recoverable for 30 days"),
("How do I check the download is genuine?", "compare the SHA-256 checksum"),
]
def words(text):
return re.findall(r"[a-z0-9_.~/-]+", text.lower())
def chunk(text, size, overlap):
toks = text.split()
step = max(1, size - overlap)
return [" ".join(toks[i:i + size]) for i in range(0, max(1, len(toks) - overlap), step)]
def bm25_index(chunks, k1=1.2, b=0.75):
docs = [Counter(words(c)) for c in chunks]
n = len(docs)
avg = sum(sum(d.values()) for d in docs) / n
df = Counter(t for d in docs for t in d)
def score(query):
out = []
for i, d in enumerate(docs):
dl = sum(d.values())
s = 0.0
for t in set(words(query)):
if t in d:
idf = math.log(1 + (n - df[t] + 0.5) / (df[t] + 0.5))
s += idf * d[t] * (k1 + 1) / (d[t] + k1 * (1 - b + b * dl / avg))
out.append((s, i))
return [i for s, i in sorted(out, reverse=True)]
return score
def norm(s):
return " ".join(words(s))
def evaluate(size, overlap, k):
chunks = [c for text in CORPUS.values() for c in chunk(text, size, overlap)]
rank = bm25_index(chunks)
hits, coverage = 0, 0.0
for question, evidence in GOLDEN:
top = [norm(chunks[i]) for i in rank(question)[:k]]
ev = norm(evidence)
if any(ev in c for c in top):
hits += 1 # the full evidence span sits inside one retrieved chunk
coverage += 1.0
else: # partial credit: share of evidence words present in the top-k
got = set(w for c in top for w in c.split())
ev_words = ev.split()
coverage += sum(w in got for w in ev_words) / len(ev_words)
ctx_words = k * size
return hits / len(GOLDEN), coverage / len(GOLDEN), len(chunks), ctx_words
if __name__ == "__main__":
K = 2
print(f"{'size':>5} {'overlap':>7} {'chunks':>6} {'recall@2':>8} {'coverage':>8} {'ctx words':>9}")
for size in (8, 12, 16, 24, 32, 48, 64):
overlap = size // 4
r, cov, n, ctx = evaluate(size, overlap, K)
print(f"{size:>5} {overlap:>7} {n:>6} {r:>8.2f} {cov:>8.2f} {ctx:>9}")Output from python3 -I sweep.py:
size overlap chunks recall@2 coverage ctx words
8 2 40 0.38 0.69 16
12 3 26 0.38 0.79 24
16 4 20 0.75 0.93 32
24 6 14 1.00 1.00 48
32 8 10 0.75 0.93 64
48 12 6 1.00 1.00 96
64 16 6 1.00 1.00 128Step 4: read three numbers together
- recall@k on spans: share of questions where one retrieved chunk contains the whole evidence span. This is the number that predicts whether the generator can answer.
- Coverage: partial credit for split evidence. When coverage is high but recall is low, answers are being cut at chunk boundaries: raise overlap or switch to structure-aware splitting before raising size.
- Context cost: k × size. This is what you pay per query and what crowds the context window.
On the toy corpus, 8- and 12-word chunks cut most answers in half (recall 0.38, coverage 0.69–0.79). Recall reaches 1.00 at 24 words for 48 words of context. 32 words dips back to 0.75 because a boundary lands mid-sentence on two questions, which is exactly the boundary noise a small golden set exposes. 48 and 64 also score 1.00, but only because each “chunk” is now a whole toy document: with three documents and k=2, retrieval barely has to work. On a real corpus that effect disappears and large chunks start losing recall to dilution.
Add answer quality last
Retrieval metrics are cheap and deterministic, so sweep them first. For the two or three best sizes, run the full pipeline and grade the answers with an LLM judge against reference answers, binary pass/fail per question. Answer quality can disagree with recall: a larger chunk sometimes carries the context the model needs to interpret a number (units, the table header), and a smaller chunk can leave a pronoun with no antecedent. If the judge prefers a larger size at equal recall, prefer it, and check whether contextual enrichment (prefixing each chunk with its document and section title) gets the same benefit at the smaller size.
Decision rules
- Pick the smallest size within 0.02 recall of the best, then confirm with answer quality. Smaller is cheaper per chunk and lets you raise k.
- If coverage − recall > 0.15, boundaries are the problem: sentence- or heading-aware splitting first, more overlap second, bigger chunks last.
- If the best size differs by question type (facts vs explanations), consider parent-document retrieval: match on small chunks, return their parent section.
- Re-run the sweep when the embedding model, the document mix or the retriever changes. Chunk size is coupled to all three.
Key takeaways
- Chunk size is a measured parameter: golden set with evidence spans, one variable swept, everything else frozen.
- Label evidence spans, not documents; document-level labels cannot distinguish chunk sizes.
- Read span recall, coverage and context cost together; the gap between coverage and recall diagnoses boundary damage.
- Pick the knee, not the peak, and confirm the top candidates with judged answer quality.
- A toy corpus validates the harness; only your own documents validate the number.
Sources
- BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models — Thakur et al., 2021paper
- Lost in the Middle: How Language Models Use Long Contexts — Liu et al., 2023paper
- The Probabilistic Relevance Framework: BM25 and Beyond — Robertson & Zaragoza, 2009paper
- Introducing Contextual Retrieval — Anthropic, 2024blog