Ingestion pipeline
The system that pulls source documents, parses, chunks, embeds and writes them to the index, and keeps it in sync as sources change and delete.
Before an AI system can search your documents, something has to collect them, extract the text, cut it into chunks, turn those into embeddings and store them. That chain is the ingestion pipeline.
The hard part is not the first load but keeping the index correct afterwards: updated files must be re-processed, deleted files must disappear from search, and a crash halfway through must not leave half a document behind.
A library's acquisitions desk: it receives new books, catalogues them, shelves them, swaps in new editions and pulls recalled titles, every day, without closing the library.
Building only the initial bulk load. Most production RAG incidents are about the second day: stale chunks after edits, duplicates after retries, and deleted documents still answering questions.
Stages: discover (crawler, webhook, or Change data capture (CDC) feed) → fetch → parse (Layout-aware parsing, OCR (optical character recognition)) → normalise and Content hashing → chunk → enrich (Chunk metadata, optional Contextual enrichment (breadcrumbs)) → embed → upsert to the vector and keyword indexes → record state in the Document registry. Each stage writes its output keyed by (doc_id, content_hash, pipeline_version) so retries are idempotent and unchanged documents skip expensive steps.
Design rules that matter in production: a per-document unit of work, so one bad PDF lands in a Dead-letter queue (DLQ) instead of blocking the batch; deterministic chunk ids (doc_id#section#n) so upserts replace rather than duplicate; deletes as first-class events (tombstones) rather than an afterthought; a version stamp (parser, chunker, embedding model, prefix) on every chunk so a Backfill knows what is stale; and nightly Reconciliation job comparing source inventory, registry and index counts. Orchestrate with a durable queue or workflow engine (Kafka or SQS plus workers, Temporal, Airflow or Dagster for batch). Embedding is usually the throughput bottleneck; batch 64–256 inputs per call and rate-limit against the provider's tokens-per-minute limit.
Event-driven per-document jobs on SQS or Kafka with stateless workers; Docling for parsing; SHA-256 content hash on normalised text; deterministic chunk ids; embedding batches of 64–256; upsert then delete stale chunk ids; registry write last; DLQ after 3 retries with exponential backoff; nightly reconciliation job. Temporal or Dagster if you need multi-step retries and visibility.
Internal wiki (Confluence, 180k pages, ~6k edits/day). Webhooks enqueue page ids; workers fetch, convert storage-format HTML to Markdown, hash, and skip 70% of events whose content hash is unchanged (label or permission edits only; ACL changes still update Chunk metadata). Changed pages are chunked (avg 9 chunks), embedded in batches of 128 with bge-m3 on one L4 GPU, and upserted with ids page-48812#3. Deletes and archive events write tombstones. Edit-to-searchable p95: 95 seconds. Nightly reconciliation flags any page whose registry hash differs from the index.
Draw it as a state machine, not a script: "Per-document jobs keyed by doc id and content hash, deterministic chunk ids, deletes as events, version stamps on every chunk, a DLQ, and nightly reconciliation of source vs registry vs index counts." The follow-up is usually "a source deletes 10k documents at once, what happens?" Tombstones flow through the same queue, the registry marks them deleted, and reconciliation verifies the index count fell by 10k.
Numbers worth knowing
- Unchanged-content events skipped by hash
- 50–80%wikis and CMSs where metadata edits dominate
- Embedding batch size
- 64–256 inputs per callAPI or self-hosted; bounded by tokens-per-minute limits
- Self-hosted embedding throughput
- ~300–1,000 chunks/sbge-m3 (568M) on one L4, 300-token chunks, FP16 (estimate)
- Freshness target
- p95 < 5 min event-driven; 24 h nightly batchedit-to-searchable
How to decide
| When | Do this |
|---|---|
| Source offers webhooks or a change log | Event-driven per-document jobs. |
| Source only supports listing | Scheduled crawl with ETag/last-modified plus content hash to skip unchanged docs. |
| Source is a database table | Change data capture (CDC) from the transaction log, not polling. |
Cost maths
Monthly = changed_docs × chunks × tokens × embed price + parse cost + LLM enrichment. 6k edits/day × 30% truly changed × 9 chunks × 350 tokens ≈ 170M tokens/month × $0.02/M (text-embedding-3-small) ≈ $3.40. Without hashing, every event re-embeds: ≈ $11. Parsing and enrichment usually dominate; embedding is rarely the bill.
Operating it
- Queue lag p95 above the freshness SLO: add workers or batch embeddings
- DLQ above 0.5% of jobs: fix the parser for the failing source type
- Reconciliation drift above 0.1% of documents: fix the event path, not the reconciler
- Edit-to-searchable latency p95: alert above SLO (e.g. 5 min)
- DLQ depth and rate: alert above 0.5% of jobs per hour
- Reconciliation: source count vs registry vs index; alert on any difference above 0.1%
When it goes wrong
A customer-support assistant cited an article the help-centre team had unpublished three weeks earlier.
Agent escalation; manual search found 1,140 unpublished articles still in the vector index.
The pipeline handled create and update webhooks but ignored article.unpublished, and there was no reconciliation job.
Unpublish and delete events now write tombstones; nightly reconciliation compares source ids to index ids and deletes orphans; alert on any orphan count above zero.
Where it shows up
Sources
- Docling Technical Report — Auer et al. (IBM Research), 2024paper
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Lewis et al., 2020paper
Check yourself
Three questions. Every option is explained once you answer.