Corpus
The full collection of documents a retrieval system can search: the source of truth that RAG answers are built from.
The corpus is all the content your search system knows about: wiki pages, PDFs, product pages, tickets, contracts. When a Retrieval-Augmented Generation (RAG) system answers a question, it can only use what is in the corpus.
If a document is missing, outdated or duplicated in the corpus, answers will be missing, outdated or confused too.
The library a librarian works from: a great librarian cannot hand you a book the library never bought, or stop you finding one it forgot to throw out.
Indexing everything once and treating the corpus as static. Stale and duplicate documents quietly become the top source of wrong answers.
Engineer the corpus as a managed dataset with a lifecycle: a Document registry tracking source, version, owner, ACL and Content hashing; an Ingestion pipeline that parses and chunks; change propagation via Change data capture (CDC); deletion via tombstones; and nightly Reconciliation job that diffs source ids against index ids.
Corpus quality caps system quality more often than model choice does. Usual problems: near-duplicate versions of the same page (MinHash at Jaccard ≥ 0.85 finds them), stale documents never removed, scanned PDFs needing OCR (optical character recognition), tables flattened into garbage, and documents without the metadata needed for filtering. Corpus statistics also shape retrieval: BM25 weights terms by corpus-wide frequency, and a jargon-heavy corpus is the signal for Fine-tuning an embedding model. Lee et al. (2021) showed deduplication materially changes model behaviour in training corpora; the same holds for retrieval corpora, where duplicates crowd out the diverse evidence a reranker needs.
A document registry table (doc_id, source, version, owner, ACL, content_hash, updated_at, deleted_at); CDC or webhook ingestion per source; MinHash dedup at Jaccard ≥ 0.85 on 5-gram shingles; only the latest version marked retrievable; nightly reconciliation; deletion SLA ≤ 1 hour.
An engineering knowledge base: 48,000 pages across Confluence, GitHub READMEs and runbooks. Dedup at Jaccard ≥ 0.85 removes 9,100 near-duplicates; another 7,400 pages untouched for 3+ years with no owner are archived. The remaining 31,500 pages become 620,000 chunks. Before the cleanup, 22% of wrong answers in error analysis cited a superseded copy of a page.
Frame the index as a cache of the corpus that must be provably in sync, deletes included. Commit to ownership (every source has an owning team), versioning (only the current version is retrievable), a deletion SLA, and reconciliation. The follow-up is often 'a customer exercises right-to-erasure, how do you prove the data is gone?': tombstone, purge from every index and cache, and show the reconciliation report.
Numbers worth knowing
- Near-duplicate threshold
- Jaccard ≥ 0.85MinHash on 5-gram word shingles
- Deletion SLA
- ≤ 1 hoursource delete to removal from every index and cache
- Full re-embed cost
- ≈ $40620k chunks × 500 tokens = 310M tokens at an assumed $0.13 / M (illustrative hosted list price)
How to decide
| When | Do this |
|---|---|
| Several versions of a document exist | Index only the current one; keep history in the registry, not the index. |
| A source has no owner | Do not index it until one is assigned. |
| Scanned PDFs or complex tables make up more than ~10% of the corpus | Add OCR and layout-aware parsing before chasing embedding models. |
Cost maths
Corpus cost = re-embed + storage + ingest compute. Assume $0.13 / M embedding tokens (illustrative, Oct 2026). Full re-embed of 620k × 500-token chunks = 310M tokens ≈ $40; dense storage 620k × 1,024 × 4 bytes ≈ 2.5 GB. The expensive part is people: an owner per source and a weekly review of the stale-document report.
Operating it
- Source-vs-index count drift above 0.5%
- Duplicate rate above 10% in a new source
- More than 5% of cited documents older than their owner's review window
- Source vs index document counts per source: alert on drift above 0.5%
- Ingest lag (source update to searchable): alert above 15 minutes p95
- Share of answers citing documents past their review date: alert above 5%
When it goes wrong
The engineering assistant gave two different answers to 'how do I rotate the database password?' depending on phrasing.
Error analysis of 150 thumbs-down answers: 22% cited a superseded copy of the runbook.
The runbook had been copied into three spaces over two years; all three were indexed and only one was maintained.
MinHash dedup at ingest, canonical-source rule in the registry, and an owner-review report for anything untouched in 12 months.
Related terms
Where it shows up
Sources
- Deduplicating Training Data Makes Language Models Better — Lee et al., 2021paper
- BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models — Thakur et al., 2021paper
Check yourself
Three questions. Every option is explained once you answer.