Reconciliation job
A scheduled job that compares the source of truth with derived stores (index, cache, ACLs) and repairs drift that events and retries missed.
Reconciliation is a regular audit that compares what should exist with what does exist, and fixes the difference. Your document database says there are 1.2 million documents; the search index should match. Events, retries and dual-writes get you close, but bugs, crashes and manual fixes leave gaps. A nightly reconciliation job finds missing, extra and outdated records and repairs them, so drift is caught in hours instead of when a user complains.
Balancing your chequebook against the bank statement at month end: most transactions match, and the job is to find the few that don't.
A reconciler that trusts an empty source. If the source API returns zero results because of an auth error or pagination bug, a naive reconciler 'repairs' by deleting the entire index.
Reconciliation is the level-triggered counterpart to event-driven sync: Kubernetes controllers work this way, repeatedly comparing desired with observed state rather than trusting that every event arrived. For a RAG index, compare per document (doc_id, content_hash, acl_hash, embed_model_version) from the Document registry against what the index holds. Classify each diff: missing (re-enqueue ingestion), orphaned (in index, not in source: delete, critical when the source deletion was an erasure request), stale (hash mismatch: re-ingest), ACL drift (permissions changed: update metadata immediately, highest priority). Don't diff 50M rows naively: bucket by hash(doc_id) % 1024, compare bucket checksums (Merkle-style), and only list rows in mismatched buckets. Rate-limit repairs so a bad reconciler can't flood the pipeline, and cap automatic deletes (e.g. abort if >1% would be deleted) because a reconciler reading an empty or truncated source will otherwise delete your whole index.
Nightly job (Temporal workflow or Kubernetes CronJob) comparing registry vs index on (doc_id, content_hash, acl_hash, model_version); 1,024 hash buckets; repairs enqueued to the normal ingestion queue at a capped rate; delete circuit breaker at 1% of corpus; ACL-only sync every 15 minutes for permissioned sources; report emitted as metrics.
An internal wiki has 2.4M pages in Confluence and 11M chunks in Qdrant. Nightly, the reconciler computes 1,024 bucket checksums on each side; 37 buckets differ. Listing those finds 212 stale pages (edited during a consumer outage), 18 orphans (pages deleted in a bulk cleanup whose events were dropped) and 6 pages whose space permissions changed. ACL fixes apply first (within minutes), then deletes, then re-ingestion. Total run: 14 minutes.
Event-driven sync is the fast path; reconciliation is the correctness guarantee. I'd run it nightly with bucketed checksums, repair ACL drift and deletes first, and put a circuit breaker on mass deletes. Follow-up: 'how fast must ACL drift be fixed?' Faster than nightly; permission changes get their own frequent sync, and retrieval re-checks the source ACL for sensitive spaces.
Numbers worth knowing
- Delete circuit breaker
- abort if >1% would be deletedguards against empty or truncated source reads
- Bucketed diff cost
- 1,024 checksums + rows in mismatched bucketsvs listing all 50M rows on both sides
- Healthy drift rate
- <0.01% of records per nightwith outbox + CDC in place; higher means a consumer bug
How to decide
| When | Do this |
|---|---|
| Derived store fed by events | Nightly reconciliation regardless of how good the events are. |
| Source has permissions | Separate ACL reconcile every 5–15 min; fix before content drift. |
| Reconciler wants to delete >1% | Abort and page; a human confirms. |
| Drift consistently >0.1% | Fix the event path; reconciliation is the net, not the pipeline. |
Cost maths
Scanning 11M chunk records for hashes is I/O, not tokens: minutes on a scroll API. Repairs cost tokens only for stale/missing docs: 212 pages × 5 chunks × 400 tokens = 424k tokens ≈ $0.01 at $0.02/M. The expensive alternative is an un-reconciled index serving a deleted HR policy for months.
Operating it
- Full reconciliation no longer finishes overnight: move to bucketed checksums or incremental by modified-time
- Drift rate >0.1% for a week
- Sources added with permissions (SharePoint, Drive): add a fast ACL reconcile loop
- Drift found per run by class (missing/stale/orphan/ACL): alert >0.1% total, >0 for ACL on restricted spaces
- Reconciler planned deletes as % of corpus: page at >1%
- Run duration and last-success time: alert if no success in 36 h
When it goes wrong
Overnight, a legal document search returned zero results for every query.
Synthetic search probes failed at 03:10; index doc count had fallen from 6.2M to 0.
The document-management API's OAuth token expired; listing returned an empty page with 200 OK, and the reconciler deleted every 'orphaned' document.
Restored from snapshot; added a delete circuit breaker (>1% aborts), treated empty source listings as errors, and validated source counts against the previous run.
Related terms
Where it shows up
Sources
- Controllers — Kubernetes, 2025docs
- Designing Data-Intensive Applications — Martin Kleppmann, 2017book
Check yourself
Three questions. Every option is explained once you answer.