Design a pipeline that extracts structured data from millions of PDFs a month — invoices, insurance claims, financial filings.
- 5 min — scope: volume, document types, latency, compliance
- 10 min — routing, parsing and OCR, extraction
- 10 min — validation, confidence, review loop
- 10 min — reliability: state machine, retries, DLQ, drift
- 5 min — estimate and ownership
Route by document type and quality. Born-digital PDFs use the text layer with layout-aware parsing (PyMuPDF, pdfplumber, Docling); scanned pages go through OCR plus a layout model. Extraction is schema-constrained: an LLM with structured outputs (a JSON schema per document type), or a vision-language model on page images for tables and handwriting. A validation layer checks schema, business rules (line items sum to total ±0.01, plausible dates, tax-id checksums) and cross-document consistency. Confidence routes each document to straight-through processing or human review. Every document moves through an idempotent state machine with a dead-letter queue.
Claims intake, 5M PDFs/month. A first-page classifier labels type (medical bill, repair invoice, police report). For a medical bill: provider, dates of service, procedure codes, charges. Validation checks code formats and that charges sum. Fields under 0.8 confidence go to a review UI that highlights the bounding box; reviewer corrections are stored as labels and feed monthly prompt and fine-tune updates — a data flywheel.
- Measure at field level. Accuracy per field per document type, straight-through rate, review minutes per document, cost per document. Document-level accuracy hides that 'due date' is wrong 6% of the time.
- Expensive failures pass the schema: swapped invoice and due dates, a total read from the wrong column. Business rules plus a weekly sampled audit catch them.
- Failure modes: vendor template drift (alert on per-sender confidence drops); injection text inside documents — the extraction model has no tools and output is schema-bound; PII in logs (redact).
- Org scope: a document platform team owns the pipeline; each business line owns its schemas and rules. Buy OCR (Textract, Document AI, Azure Document Intelligence) to start; own schemas, validation and the review loop. Self-host an open-weights VLM once volume justifies it.
Assumptions: 5M PDFs/month × 12 pages = 60M pages; 30% scanned; ~1.5k tokens/page → 18k input + 1k JSON output per document; illustrative $1/M input, $5/M output with a 50% batch discount; layout + OCR ~10 pages/s per L4; 3% of documents reviewed at 2 min each; reviewer cost $25/h (illustrative).
- Throughput: 60M pages/month ≈ 23 pages/s; scanned 18M pages ≈ 7 pages/s → 1 L4 on average, 4 L4 for month-end peaks ≈ $2.5k/month.
- LLM: 90B input × $0.50/M = $45k + 5B output × $2.50/M = $12.5k ≈ $57.5k/month ≈ $0.012/document.
- Review: 150k docs × 2 min = 5,000 h × $25 ≈ $125k/month — over 2× the model bill.
- Lever: +1 point of straight-through rate = 50k fewer reviews ≈ 1,670 h ≈ $42k/month saved.
- Storage: ~2.5 TB of PDFs/month at ~500 KB each; page images are transient.
What the answer sounds like at each level
OCRs the PDFs and asks an LLM to return JSON.
Routes digital vs scanned, uses schema-constrained extraction, validates with business rules and routes low confidence to review.
Optimises the review loop as the main cost, measures per-field accuracy, uses batch economics, and splits schema ownership to business lines.
Follow-ups the interviewer will push
1Quarter-end brings a 10× spike.
Queue-driven autoscaling workers with backpressure; the batch API absorbs non-urgent volume within 24 h; urgent claims use a separate realtime path with its own quota.
2The data is HIPAA-covered.
BAA-covered endpoints only, in-region processing, encryption at rest and in transit, minimum-necessary fields in logs, and no training on documents without contractual permission.
3You want to upgrade the extraction model.
Shadow it on 5k labelled documents, diff field by field, and gate on no field regressing more than 0.5 points; roll out per document type.
Red flags interviewers listen for
- Reports document-level accuracy only.
- No validation beyond JSON schema.
- Synchronous per-request processing with no queue, retries or DLQ.
- Ignores the cost of human review.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Pipeline | Type and quality routing; schema-constrained extraction; durable state machine. |
| Correctness | Business-rule validation and field-level accuracy with sampled audits. |
| Economics | Batch pricing and review cost quantified; straight-through rate as the lever. |
| Ownership | Schemas owned by business lines; buy-then-build path for OCR. |
Sources
- Nougat: Neural Optical Understanding for Academic Documents — Blecher et al., 2023paper
- Batch processing — Anthropic, 2025docs