How would you design RAG for continuously changing enterprise data?
- 1 min — freshness SLO first
- 3 min — CDC → queue → diff → upsert → tombstone
- 2 min — ordering, idempotency, backpressure
- 1 min — ACL changes and full-rebuild path
Event-driven incremental ingestion. Source systems emit change events (CDC, webhooks, or polling with checksums) → a queue → a worker that diffs at chunk level and re-embeds only changed chunks → upsert with a document-version key → tombstone deletions. Keep a full-rebuild path for schema or model migrations, and make it idempotent so a replay is safe.
Content-hash each chunk. On a document edit, most chunks are unchanged — re-embedding only the diff cut our embedding bill by roughly 90% on a corpus that churned daily.
- Freshness SLO. Commit to a number — "a source edit is searchable within 5 minutes at p95" — and alert on ingestion lag.
- Ordering. Out-of-order events can resurrect deleted content. Version-stamp every write and drop stale ones.
- Backpressure. A bulk edit (someone reorganises a wiki) can flood the queue. Rate-limit and prioritise by document importance.
- Permissions change too. An ACL update is a change event like any other, and it's the one with legal consequences if you miss it.
- Org scope: connectors are the long tail — buy them where a vendor covers your sources (SharePoint, Confluence, Google Drive), build the diff/upsert core once, and give each source system's owning team an on-call contract for its change feed.
Assumptions: 5M documents, 20 chunks/doc at ~500 tokens, 2% of documents edited per day, ~10% of chunks change per edit; embedding at $0.02 / M tokens (text-embedding-3-small list price, illustrative, Oct 2026).
- Edits: 100k docs/day ≈ 1.2 events/s average; plan for 20× bursts ≈ 25/s.
- Full re-embed of edited docs: 100k × 20 × 500 = 1B tokens/day ≈ $20/day.
- Hash-diffed: ~10% → 100M tokens/day ≈ $2/day, and 10× fewer index writes.
- Burst throughput: 25 docs/s × 2 changed chunks × 500 tokens = 25k tokens/s = 1.5M tokens/min — check it against the provider's rate limit.
The dollar cost is small; the binding constraints are rate limits and index write amplification.
What the answer sounds like at each level
Re-index documents on a schedule.
Event-driven incremental ingestion with chunk-level diffs, versioned upserts, tombstones and an idempotent rebuild path.
Commits to a freshness SLO, handles ordering, backpressure and ACL changes as first-class events, and sizes the pipeline for bulk-edit storms.
Follow-ups the interviewer will push
1Now a bulk wiki reorganisation emits 2 million change events in an hour.
Backpressure: the queue absorbs it, workers are rate-limited to the embedding provider's tokens-per-minute, and priority lanes keep ACL changes and deletes ahead of content edits. Most moves won't change content hashes, so they become cheap metadata updates.
2Now an ACL change takes 30 minutes to propagate. Is that acceptable?
No for revocations. Apply ACLs at query time from the source of truth (or a fast ACL cache with <1 min TTL) rather than only baking them into the index, so a removed user loses access immediately.
Red flags interviewers listen for
- Proposes nightly full re-index for a daily-churning corpus.
- Ignores out-of-order events and resurrection of deleted content.
- Treats ACL changes as an afterthought.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Pipeline design | Incremental, versioned, idempotent with a rebuild path. |
| Failure modes | Covers ordering, storms and lost events. |
| Security | Treats ACL revocation as the highest-priority change. |
Sources
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Lewis et al., 2020paper
- Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs — Malkov & Yashunin, 2016paper