How would you handle document ingestion failures?
- 1 min — durable state machine per document.
- 2 min — failure classes and the response to each.
- 2 min — silent-success quality gates.
- 1 min — customer-visible status and ownership.
Treat ingestion as a durable pipeline with per-document state, not a script. Each document moves through states (received → parsed → chunked → embedded → indexed) persisted in the registry. Retries with exponential backoff for transient failures, a dead-letter queue for permanent ones, idempotency so replay is safe, and partial-success handling so one bad page doesn't drop a 400-page document.
Classify failures, because they need different responses: transient (rate limit, timeout) → retry; data quality (corrupt file, unsupported format, scanned image) → DLQ with a user-visible reason; systemic (parser crash on a whole document class) → alert a human, because retrying won't help and will burn budget.
The real failure mode is silent success — a document that ingests "fine" and produces empty or garbage chunks. Add quality gates at ingestion: text density per page, chunk count sanity, embedding norm checks. And surface per-document status to the customer; "47 of 50 documents indexed, 3 failed because they're scanned images" is a vastly better product than a system that quietly knows less than the user thinks.
Org scope: buy the durable-queue primitives (SQS, Pub/Sub, Kafka, or a workflow engine like Temporal) and build the per-document state machine. The platform owns the pipeline and DLQ triage; source-system owners are paged when their export format breaks.
What the answer sounds like at each level
Retry failed documents and log errors so someone can fix them.
Persists per-document state, classifies failures into transient, data-quality and systemic, with backoff, DLQ, idempotent replay and partial success.
Adds quality gates against silent success, exposes per-document status to customers, and assigns DLQ triage and source-format breakage to named owners.
Follow-ups the interviewer will push
1The embedding API rate-limits you during a 2M-document backfill. What happens?
Workers back off exponentially with jitter, the queue absorbs the backlog, and a token-bucket limiter caps concurrency below the provider's limit. Live updates go on a separate priority queue so fresh documents aren't stuck behind the backfill.
2How do you make replay idempotent?
Chunk ids derived from (document id, chunk position, content hash), upserts instead of inserts, and a registry state check so a re-run skips stages already completed for the same hash.
Red flags interviewers listen for
- Retries everything, including permanent data-quality failures.
- No idempotency, so replays create duplicate chunks.
- Treats 'no error' as success without content-quality checks.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Durability | Per-document state, idempotent replay, partial success. |
| Failure taxonomy | Different handling for transient, data-quality and systemic failures. |
| Silent failures | Quality gates that catch empty or garbage chunks. |
Sources
- Using dead-letter queues in Amazon SQS — AWS, 2024docs