How would you handle documents containing images and text together?
- 1 min — three strategies by cost
- 2 min — which document types need which
- 1 min — keep image references
- 1 min — caching and re-ingest cost
Three strategies, increasing in cost. (1) Caption and index as text: run a vision model over each image, store the description as a chunk linked to the image. Works with your existing text pipeline. (2) Multimodal embeddings: embed images and text into a shared space. (3) Page-as-image: embed rendered page images directly with a visual retrieval model and feed the page image to a vision LLM — strong on complex layouts, expensive.
For technical documentation, captioning wins: diagrams usually have surrounding text that explains them, and the caption plus that text is enough. For slide decks and scanned forms, where the layout is the information, page-as-image is worth the cost.
Keep the image reference on the chunk so the answer can show the diagram, not just describe it. Also: caption quality is a model-dependent cost you'll pay again on every re-ingest, so cache captions keyed by image hash. Org scope: caption generation is a shared ingestion step with a hash-keyed cache; buy the vision model, and gate page-as-image retrieval to the sources that need it so its GPU cost doesn't land on every team.
What the answer sounds like at each level
Use OCR or captions so images become searchable text.
Compares captioning, multimodal embeddings and page-as-image retrieval, and picks by document type.
Prices each strategy, caches captions by hash, keeps image references for display, and routes sources to strategies.
Follow-ups the interviewer will push
1Now the corpus is 2M slide decks where layout carries the meaning.
Page-as-image retrieval (ColPali-style late interaction over page images) for those decks, captioning for everything else. Budget for multi-vector storage — roughly 1,000 patch vectors per page instead of one.
2Now the captioning model is deprecated.
Captions are cached by image hash with caption_model_version; re-caption lazily on next ingest or in a background backfill, and re-embed only chunks whose caption text changed.
Red flags interviewers listen for
- Ignores images entirely.
- Uses one strategy for all document types.
- Re-captions every image on every re-ingest.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Options | Names all three strategies with their cost profile. |
| Routing | Matches strategy to document type. |
| Operations | Caches by hash and versions the captioner. |
Sources
- ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT — Khattab & Zaharia, 2020paper