How would you choose the optimal chunk size?
- 1 min — precision/recall trade-off
- 2 min — natural unit per document type
- 2 min — sweep on golden set, two metric families
- 1 min — small-to-big decoupling
It's a precision/recall trade-off against the natural unit of meaning in your corpus. Small chunks embed precisely but lose context and fragment answers. Large chunks keep context but dilute the embedding — one vector averaging five topics matches none of them well. There is no universal number; there is a measurement procedure.
Start from document structure rather than token counts. API reference docs chunk cleanly at the endpoint; legal contracts at the clause; support tickets as whole tickets; transcripts by speaker turn or topic shift. Then sweep sizes around that natural unit against a golden set and measure Recall@k and end-to-end answer quality — not just retrieval metrics, because the two diverge.
Decouple the retrieval unit from the generation unit. Embed a small precise chunk, but return an expanded window (parent document or neighbouring chunks) to the LLM. That gets you precision in search and context in generation without compromising either. Mention that this makes chunk size much less critical — which is the real answer. Org scope: chunking config belongs to the platform with per-source-type overrides owned by the teams who know the documents; version it (chunker_version) so a change is a tracked re-index, not a silent drift.
What the answer sounds like at each level
Pick something like 512 tokens with some overlap and test it.
Starts from the document's natural unit, sweeps sizes on a golden set and measures both Recall@k and answer quality.
Decouples retrieval unit from generation unit (small-to-big), which makes size less critical, and treats chunker config as a versioned, per-source contract.
Follow-ups the interviewer will push
1Now the corpus mixes code, contracts and chat transcripts.
Route by source type: code by function/class via a syntax-tree splitter, contracts by clause, transcripts by speaker turn with a window. One global size is wrong for at least two of the three.
2Now you must cut embedding storage by half.
Larger child chunks (e.g. 256 → 512 tokens) halve vector count; or keep size and use Matryoshka truncation 1024 → 512 dims. Measure Recall@10 for both on the golden set and pick the smaller loss.
Red flags interviewers listen for
- Gives a single magic number with no measurement.
- Evaluates only retrieval metrics, never end-to-end answers.
- Ignores document structure.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Method | Proposes a measured sweep, not a number. |
| Structure awareness | Maps document types to natural units. |
| Design insight | Separates retrieval unit from generation unit. |
Sources
- Dense Passage Retrieval for Open-Domain Question Answering — Karpukhin et al., 2020paper
- Lost in the Middle: How Language Models Use Long Contexts — Liu et al., 2023paper