Content hashing
A fingerprint (e.g. SHA-256) of a document's normalised text, used to skip unchanged content, dedupe copies and make re-processing idempotent.
A content hash is a short code computed from a document's text. The same text always gives the same code, and any change, even one character, gives a different one.
Ingestion stores the hash for each document. Next time the document arrives, if the hash matches, nothing changed and the expensive steps (parsing, Embedding) can be skipped. Identical hashes across documents also reveal exact duplicates.
A wax seal on a letter: if the seal is unbroken and matches, you know nobody changed the contents without opening it.
Hashing raw bytes. Timestamps, tracking params and PDF re-exports change bytes without changing meaning, so the hash says everything changed and you re-embed the whole corpus nightly.
Hash normalised content, not raw bytes: Unicode NFC, collapse whitespace, strip volatile parts (render timestamps, session ids, 'last viewed' banners, page numbers), and for PDFs hash extracted text rather than the file, since re-exports change bytes without changing words. SHA-256 is the safe default (collisions are not a practical concern); xxHash64 or BLAKE3 are faster if you hash terabytes and only need change detection.
Use it at three levels: document hash to skip unchanged documents; chunk hash so a one-paragraph edit re-embeds 1 chunk instead of 40 (cache embeddings by hash(chunk_text + embed_model + prefix)); and corpus dedupe so 50 copies of the same policy do not fill top-k. Exact hashes miss near-duplicates (a changed footer, a date); for those use MinHash/LSH over shingles with a Jaccard threshold around 0.8, as in large-scale training-data dedupe. Include the pipeline version in the cache key, so a parser or model upgrade correctly invalidates everything.
SHA-256 over normalised extracted text; document hash in the registry; chunk embedding cache keyed by sha256(chunk_text) + embed_model + prefix_version + chunker_version; MinHash LSH (datasketch, 128 permutations, Jaccard 0.8–0.85) for near-duplicate collapse on web or marketplace corpora.
E-commerce catalog, 3.2M product pages re-crawled nightly. Raw HTML differs every night (prices, stock, CSRF tokens), so a byte hash said 100% changed. Hashing only the normalised description, specs and title showed 1.8% truly changed per night.
Chunk-level embedding cache keyed by sha256(chunk || 'text-embedding-3-small' || 'v2'): nightly embedding fell from 3.2M × 6 chunks to ~350k chunks. MinHash (5-word shingles, Jaccard ≥0.85) also collapsed 410k marketplace duplicate listings into canonical ids.
Define what you hash: "SHA-256 of normalised extracted text at document and chunk level, with the pipeline version in the key. It makes retries idempotent, skips unchanged work and gives exact dedupe; near-duplicates need MinHash." The usual follow-up: "metadata-only change, like an ACL edit?" Don't re-embed; update the payload and registry. ACL changes must never wait for a content change.
Numbers worth knowing
- Truly changed share on nightly re-crawl
- 1–5%catalogs and docs sites after normalisation
- SHA-256 throughput
- ~0.5–2 GB/s per coremodern x86/ARM with SHA extensions
- Near-dup threshold
- Jaccard ≥0.8 on shinglesMinHash LSH, common in corpus dedupe
How to decide
| When | Do this |
|---|---|
| Re-crawling sources that re-render on every fetch | Normalise before hashing, or every document looks changed. |
| Edits are small relative to documents | Chunk-level hash cache; re-embed only changed chunks. |
| Corpus contains copies with tiny differences | MinHash near-dup collapse to a canonical id. |
Cost maths
Savings = (1 − changed share) × nightly chunks × tokens × price. 19.2M chunks × 300 tokens = 5.8B tokens/night; at $0.02/M = $115/night ≈ $3,450/month re-embedding everything. With 1.8% changed: ≈ $62/month. With enrichment at ~$6.70 per M tokens, the same saving is about $1.1M/month vs $20k, which is why hashing comes before enrichment.
Operating it
- Nightly embedding volume close to corpus size: hash is on raw bytes or volatile content
- Top-k often contains identical passages from different URLs: add exact and near-dup collapse
- Share of documents whose hash changed per run: alert above 3× the trailing 7-day median
- Embedding cache hit rate: alert below 90% on routine re-crawls
- Duplicate-cluster count per corpus refresh: alert on a 2× jump
When it goes wrong
Embedding spend on a docs-site RAG jumped 30× overnight with no content changes.
Daily cost alert on the embeddings line item; re-embedded chunk count equalled the full corpus.
The docs site added a 'Last built at <timestamp>' footer; the hash included the footer, so every page looked new.
Normalisation rules per source with a test that hashes the same page fetched twice; cost dropped back the next night.
Where it shows up
Sources
- Deduplicating Training Data Makes Language Models Better — Lee et al., 2021paper
Check yourself
Three questions. Every option is explained once you answer.