Multimodal & vision models
Models or pipelines that handle more than text, such as images, scanned PDFs, audio or video, in the same input or index.
A multimodal model can take in more than words: a photo, a chart in a PDF, a screenshot or an audio clip, and reason about it alongside text.
For Retrieval-Augmented Generation (RAG) this matters because many real documents are not clean text. A maintenance manual's key fact might live in a wiring diagram or a table that is really an image.
A colleague who reads the report and also looks at the attached photos and charts, instead of only reading the captions.
Running plain text extraction on complex PDFs and assuming the tables and charts reached the index intact.
Vision-language models cut images into patches, encode them, and project them into the same token stream the LLM attends over. Images are expensive in Token terms: Claude bills ⌈width/28⌉ × ⌈height/28⌉ visual tokens, so a 1092 × 1092 page is 1,521 tokens, capped at 1,568 on standard-resolution models and 4,784 on high-resolution ones (Oct 2026).
Three retrieval patterns. (1) Text-first: OCR (optical character recognition), captioning and table extraction with Layout-aware parsing, then a normal text index; cheapest, loses layout. (2) Shared embedding space: CLIP-style models place images and text in one Vector space; good for photo search, weak on dense documents. (3) Page-image retrieval: ColPali embeds each page image as ~1,030 patch vectors of 128 dimensions with late interaction and outperformed OCR pipelines on its ViDoRe benchmark; the retrieved page images go to a VLM at answer time. Choose by whether layout carries meaning (charts, forms, engineering drawings) and by token budget.
Layout-aware parsing (e.g. Docling, Unstructured, or a cloud document-AI service) for every PDF; flag pages where table or figure area exceeds ~40% and index those as page images too; resize images to the model's native limit before sending; store page images alongside chunks so the answer step can attach them; a 100-question table-and-chart slice in the golden set.
A field-service assistant indexes 50,000 pages of equipment manuals. Text pages go through OCR into a text index (~700 tokens per page in context). The 6,000 pages that are mostly tables and wiring diagrams are indexed as page images with ColPali; at answer time the top 2 page images (~3,000 visual tokens) plus 1,500 tokens of text go to the VLM. A technician asking 'torque for the M8 flange bolts on pump P-40?' gets '25 N·m [manual P-40, p.112, table 7-3]' from the table image the OCR had scrambled.
Commit: text-first by default, page images for the pages where layout is the content, and measure on a slice of table and chart questions. Quote token costs per page so the interviewer sees you budgeted it. Follow-up: 'scanned handwriting?': OCR with confidence scores, human review below a threshold, never silently index low-confidence text.
| Approach | Index cost per page | Tokens at answer time | Keeps layout |
|---|---|---|---|
| OCR text + text embeddings | 1–2 vectors | ≈700 | No |
| VLM caption + OCR at ingest | 1 VLM call + vectors | ≈900 | Partly |
| Page image (ColPali) | ≈1,030 × 128-d vectors | ≈1,500 (image) | Yes |
Numbers worth knowing
- Image token cost (Claude)
- ⌈w/28⌉ × ⌈h/28⌉1092 × 1092 px = 1,521 tokens; capped at 1,568 standard, 4,784 high-resolution (Oct 2026)
- ColPali page representation
- ≈1,030 vectors × 128 dimsper page image, late interaction
- OCR text per page
- ≈500–800 tokensdense manual or filing page
How to decide
| When | Do this |
|---|---|
| Documents are mostly prose | Text-first; don't pay image tokens. |
| Answers live in tables, charts, forms or drawings | Index those pages as images (ColPali-style) and send the page image to a VLM. |
| Searching photos by description (catalog, claims photos) | Use a shared image-text embedding model. |
Cost maths
Ingest: captioning 6,000 visual pages × (1,521 in + 300 out) at $3 / $15 per M (illustrative) = $27 + $27 ≈ $54 once. Answer time: 2 page images + 1.5k text ≈ 4.5k input tokens = $0.0135 per query vs ~$0.006 text-only. At 100k queries/month where 15% hit visual pages: +$1,100/month for the visual path, against the cost of a wrong torque value.
Operating it
- Golden-set accuracy on table/chart questions 10+ points below prose questions
- OCR confidence below 0.8 on more than 5% of pages
- Image tokens exceed 40% of input spend
- Golden-set accuracy on the table/chart slice: alert at −5 points
- OCR low-confidence page rate per ingest batch: alert above 5%
- Image tokens per query p95: alert on +50% week over week
When it goes wrong
Technicians received wrong torque values for a pump family; one flange was over-torqued during maintenance.
A technician report, then a 100-question table slice showing 64% accuracy against 91% on prose.
Plain OCR merged adjacent table columns, so the value for M10 bolts was indexed next to the M8 label.
Layout-aware table extraction, page-image indexing for table-heavy pages, page image attached at answer time, and a 'verify against the cited page' link in the UI.
Related terms
Sources
- Learning Transferable Visual Models From Natural Language Supervision (CLIP) — Radford et al., 2021paper
- ColPali: Efficient Document Retrieval with Vision Language Models — Faysse et al., 2024paper
- Vision — Anthropic, 2026docs
Check yourself
Three questions. Every option is explained once you answer.