Change data capture (CDC)
Change data capture: streaming inserts, updates and deletes from a database's transaction log, so the search index tracks the source within seconds.
When search content lives in a database (products, tickets, records), the index must follow every change. Polling the whole table every hour is slow and misses deletes.
Change data capture reads the database's own change log, the record it keeps of every write, and turns each insert, update or delete into an event. The ingestion pipeline consumes those events and updates the index within seconds.
Subscribing to a shop's delivery notes instead of walking the aisles every hour to see what changed.
Treating CDC events as exactly-once and in global order. Duplicates and replays are normal; without version checks an old update can overwrite a newer one or resurrect a deleted row.
Log-based CDC reads the write-ahead log (Postgres logical decoding via pgoutput, MySQL binlog, MongoDB change streams, DynamoDB Streams). Debezium is the common open-source connector, publishing one event per row change to Kafka with op (c/u/d), before and after images, and source position (LSN). Deletes arrive as an event plus a Kafka tombstone (null value) for compaction. The consumer re-renders the searchable text, checks the Content hashing, re-embeds changed chunks and upserts, or deletes chunk ids on op: d.
Guarantees: at-least-once delivery, ordered per key (partition by primary key). So consumers must be idempotent and use the source position or a version column to drop stale events ('apply only if lsn > stored lsn'). Operational traps: a Postgres replication slot whose consumer stops retains WAL until the disk fills (cap it with max_slot_wal_keep_size, PG13+); initial snapshots of large tables need a planned Backfill; schema changes must be versioned. Prefer CDC to Dual-write from application code, which drops writes whenever one side fails.
Debezium (Postgres pgoutput) to Kafka or a managed equivalent, topic partitioned by primary key, max_slot_wal_keep_size set, Kafka retention ≥ 7 days, consumer applying events only when source LSN/version is newer than the stored one, metadata-only updates skipping re-embed, nightly reconciliation against a table snapshot.
Product catalog in Postgres, 4M SKUs, ~200k updates/day. Debezium streams products changes to Kafka topic catalog.products (24 partitions, keyed by sku). Event {op: 'u', sku: 8812, lsn: 0/3A2F…, after: {title, specs, price}}. The consumer renders title + specs (price lives in metadata, not text), sees the content hash unchanged on 85% of updates (price-only), updates metadata only, and re-embeds the rest. Source-to-searchable p95: 12 s vs 24 h with the old nightly export. A discontinued product (op: 'd') leaves search within seconds.
Say why not dual-write: "Application code writing to both DB and index loses one side on partial failure. I read the WAL with Debezium, key by primary key, apply idempotently with the LSN as a version, handle deletes, and reconcile nightly." Expect: "the consumer is down for a day?" Kafka retention covers it if retention exceeds the outage; the replication slot does not grow because Debezium already committed to Kafka; and cap slot WAL in case Debezium itself stops.
Numbers worth knowing
- Source-to-searchable latency
- seconds (p95 5–30 s)CDC + streaming consumer, excluding batch embedding waits
- Kafka tombstone retention (delete.retention.ms)
- 24 h defaultcompacted topics
- Updates needing re-embed
- 10–30%catalogs where price/stock changes dominate (keep them in metadata)
How to decide
| When | Do this |
|---|---|
| Source of truth is an OLTP database | Log-based CDC; never polling or dual-write. |
| Freshness need is daily and the table is small (<1M rows) | Nightly snapshot diff by content hash is simpler. |
| Fields change often but don't affect meaning (price, stock) | Put them in Chunk metadata and update without re-embedding. |
Cost maths
Kafka + Debezium on a managed service ≈ $300–1,000/month for a small cluster (illustrative, Oct 2026). Embedding: 200k updates/day × 15% re-embed × 2 chunks × 250 tokens = 15M tokens/day × $0.02/M ≈ $9/month. The infrastructure, not tokens, is the cost; justify it by freshness and correct deletes.
Operating it
- Consumer lag above the freshness SLO: add partitions and consumers
- Replication slot retained WAL above 20% of disk: page immediately
- Nightly batch freshness causing user-visible staleness (sold-out items, closed tickets)
- Replication slot retained WAL (bytes): alert above 10 GB
- Consumer lag (events and seconds): alert above freshness SLO
- Events rejected as stale by version check: alert on spikes (replays, ordering bugs)
When it goes wrong
The primary Postgres database ran out of disk at 3 a.m. and went read-only, taking the storefront down.
Disk alert on the primary; pg_replication_slots showed one inactive slot holding 900 GB of WAL.
The Debezium connector had crashed on a schema change two days earlier; its slot kept all WAL since, and no slot-lag alert existed.
Set max_slot_wal_keep_size, alert on slot lag above 10 GB, made schema changes go through a registry-compatible migration, re-snapshotted the table into the index.
Sources
- Debezium documentation — Debezium, 2025docs
- Apache Kafka documentation (log compaction) — Apache Software Foundation, 2025docs
Check yourself
Three questions. Every option is explained once you answer.