Document registry
The system of record for ingestion: one row per source document with hash, versions, status, ACL and chunk ids, so the index can be audited and rebuilt.
A vector index is a search structure, not a record book. The document registry is the record book: a normal database table with one row per source document saying where it came from, its current version, when it was last processed, which chunks it produced and whether it has been deleted.
With it you can answer 'is this document in the index, and is it the latest version?' and rebuild everything if the index is lost.
The ledger of a warehouse: shelves show what is there right now, but the ledger records what came in, what left, and when, so you can audit or restock.
Using the vector database's payload as the system of record. You lose transactions, cheap scans and history, and deletes become filter queries that miss chunks.
Columns that pay for themselves: doc_id (stable source id), tenant_id, source_uri, content_hash, source_version (ETag, revision, LSN), parser_version, chunker_version, embed_model + embed_prefix_version, chunk_ids[], acl, status (active | deleted | failed | quarantined), deleted_at, last_ingested_at, error. Store it in Postgres or DynamoDB, not in the vector DB's payload, because you need transactions, joins, cheap scans and point-in-time backups.
It powers: skip-if-unchanged (hash compare), precise deletes (look up chunk_ids instead of filtering the vector index), Backfill planning (WHERE embed_model != 'v3'), Reconciliation job (source vs registry vs index), audit and data-subject requests ('which chunks came from this person's file, and when were they removed?'), and rebuilds from scratch. Write the registry row after the index upsert succeeds, or use an outbox pattern, so a crash never records work that did not happen.
A Postgres table keyed by (tenant_id, doc_id) with the columns above, indexes on status and embed_model, row written after the index upsert (or via a transactional outbox), soft deletes with deleted_at, and raw source snapshots in object storage keyed by content hash so the index can be rebuilt without re-crawling.
Clinical guidelines corpus, 42k documents across 6 hospitals. A registry row: {doc_id: 'guideline-sepsis-v7', tenant_id: 'north-trust', content_hash: '9f2c…', source_version: 'rev-112', parser: 'docling-2.x', chunker: 'md-recursive-300@4', embed: 'bge-m3@2024-06', chunk_ids: [… 14 ids], acl: ['clinicians'], status: 'active'}.
When v8 supersedes v7, the pipeline upserts 15 new chunks, deletes the 14 ids listed for v7 directly, and marks v7 deleted with a timestamp the audit team can query.
Say why it exists: "The vector DB is a cache of the registry. The registry, in Postgres, holds hash, versions, ACL, status and chunk ids per document; deletes, backfills and reconciliation all run from it, and I can rebuild the index from it plus the raw store." Expect: "what if they disagree?" The registry wins; reconciliation repairs the index and alerts.
| Column | Example | Used for |
|---|---|---|
| content_hash | 9f2c… | Skip unchanged |
| embed_model | bge-m3@2024-06 | Backfill planning |
| chunk_ids[] | 14 ids | Exact deletes |
| acl / tenant_id | clinicians | Filter + audit |
| status, deleted_at | deleted 2026-09-14 | Erasure proof |
Numbers worth knowing
- Row size
- ~1–2 KBwith ~20 chunk ids and ACL list
- Registry for 10M documents
- ~10–20 GB in Postgresfits one managed instance
- Full rebuild from registry + raw store
- hours, not weeksno re-crawl; bounded by embedding throughput
How to decide
| When | Do this |
|---|---|
| More than one source or more than ~10k documents | Run a registry; payload-only bookkeeping stops being auditable. |
| Regulated data (health, finance, EU personal data) | Registry with deletion timestamps and chunk ids is your erasure evidence. |
| Registry and index disagree | Registry wins; reconcile the index and alert. |
Cost maths
Small: 10M rows × 1.5 KB = 15 GB; a db.r6g.large-class Postgres ≈ $200–300/month (illustrative, Oct 2026). It saves re-embedding: a rebuild from cached raw text and stored vectors avoids paying 10M × 9 chunks × 350 tokens ≈ 31B tokens again ($630 at $0.02/M, or far more with enrichment).
Operating it
- Deletes or backfills require full scans of the vector index: add a registry
- Audit or erasure requests take days to answer: registry with deletion timestamps
- Registry writes exceed ~5k/s: partition by tenant or move to DynamoDB
- Index chunks with no active registry row: alert on any non-zero count
- Registry rows in
failedorquarantinedolder than 24 h: alert above 100 - Distinct embed_model values among active rows: alert if >1 outside a backfill
When it goes wrong
After a vector DB restore from backup, 6% of answers cited documents that had been deleted months earlier.
Reconciliation found 210k index chunks with no active registry row.
The backup predated a large purge; there was no step comparing the restored index to the registry.
Post-restore reconciliation became mandatory: delete every chunk whose doc_id is not active in the registry before re-enabling traffic.
Sources
- Elasticsearch aliases — Elastic, 2025docs
- Debezium documentation — Debezium, 2025docs
Check yourself
Three questions. Every option is explained once you answer.