Multimodal RAG: Retrieving from Images, PDFs, and Mixed-Media Enterprise Archives

Most RAG systems quietly assume the world is plain text. Enterprise archives are not. They're scanned contracts, financial charts, architecture diagrams, slide decks, engineering drawings, forms, invoices, and medical documents — a huge share of an organization's most valuable knowledge, locked in formats a text-only pipeline simply cannot see. A text-only RAG system reading those archives is reading with one eye closed, and it will confidently tell users "I couldn't find that" when the answer was right there in a figure.

Multimodal RAG closes that gap by making the visual content of enterprise documents retrievable — the charts, diagrams, tables, and scans, not just the paragraphs around them. For document-heavy industries, that's the difference between a system that searches a fraction of the corpus and one that searches all of it. Here's how it works and where it pays off.

The knowledge text-only RAG can't see

Walk through a real enterprise archive and the problem is obvious:

• A scanned PDF with no text layer is, to a text pipeline, a blank page.
• A financial chart carries the trend that answers the question — invisible to text extraction.
• An engineering drawing encodes the dimension, the part, the tolerance — in lines and labels, not prose.
• A slide deck puts the actual decision in a diagram with three words of text around it.
• Forms and invoices hold structured values in a visual layout a naive parser shreds.

Text-only RAG either skips this content or mangles it. Multimodal RAG treats the visual content as first-class — extractable, describable, and retrievable.

Two approaches (and the pragmatic blend)

There are two ways to make visual content retrievable, and the best systems use both.

1. Extract to text (OCR + vision description). Run OCR to recover text from scans, and use a vision model to describe images, charts, and diagrams in words ("a bar chart showing Q3 revenue up 12% over Q2"). Those descriptions are then chunked, embedded, and retrieved like any other text. This is robust, works with your existing text-retrieval stack, and makes the meaning of a figure searchable.

2. Native multimodal embeddings. Embed images directly into a shared vector space alongside text using a multimodal embedding model (CLIP-style models, Amazon Titan Multimodal Embeddings), so a text query can retrieve a relevant image by semantic similarity — no description step required.

In practice, a strong enterprise pipeline blends them: OCR for scanned text, vision-model descriptions to capture what charts and diagrams mean (and to keep answers explainable and citable), and native image embeddings where direct image retrieval adds value. The blend you choose depends on the corpus — invoice-heavy archives lean on layout extraction; diagram-heavy ones lean on vision description and image embeddings.

Multimodal ingestion: where it gets done

Multimodal RAG lives or dies in the ingestion pipeline (the layer where most RAG projects struggle anyway — see RAG data ingestion). A production multimodal ingestion flow:

1. Classify and route each document/page by type (native text, scan, chart, drawing, form) so each gets the right treatment.
2. OCR scanned content to recover text, with layout awareness so tables and columns survive.
3. Describe visual elements — pass charts, diagrams, and images through a vision model to generate searchable descriptions, preserving the link back to the source image for citation.
4. Extract structured data from forms and invoices into clean, retrievable fields with their labels intact.
5. Chunk with the figure attached — keep a chart or diagram together with its caption and surrounding context, not severed from it (a chunking concern).
6. Embed and index text and (where used) image vectors, with the same metadata and permissions as everything else.

This is squarely document intelligence territory — extraction plus search plus audit — which is exactly where Sphere's work in document-heavy workflows lives. The medical-documentation and medical-device order-automation engagements are real examples of AI handling structured and unstructured documents inside existing operational workflows (EMR, ordering), not just answering questions in a standalone chat.

Retrieval over mixed-media corpora

Once visual content is represented, retrieval has to handle a heterogeneous index. The practical strategies:

• Unified text retrieval over OCR text + vision descriptions: simplest, and keeps everything in one searchable, citable text space.
• Cross-modal retrieval where a text query can match image embeddings directly, surfacing the actual figure.
• Fusion: retrieve across both representations and rank together, so the best answer wins whether it lives in a paragraph or a chart.

And the same enterprise rules still apply: permission-aware retrieval (a restricted diagram is as sensitive as a restricted paragraph), citations back to the exact source (the page and the figure), and confidence signals.

Where it shines: engineering drawing Q&A

A concrete payoff. An engineer asks: "What's the specified torque for the fasteners on the X-450 mounting bracket?" The answer isn't in any paragraph — it's a callout on a CAD-derived engineering drawing. Text-only RAG returns nothing useful. Multimodal RAG, having ingested the drawing with OCR (for the callouts and title block), vision description (for the layout and components), and structured extraction (for the spec table), retrieves the right drawing, surfaces the torque spec, and cites the exact drawing so the engineer can verify against the source. That's a query that was previously a manual hunt through a document management system, answered in seconds — and it generalizes to any technical, forms-heavy, or visually-encoded archive.

Is your knowledge locked in scans, charts, and drawings? Get a RAG Readiness Assessment — we'll evaluate your mixed-media archive and design the OCR, vision, and retrieval pipeline to unlock it.

Related: the enterprise RAG pillar guide, RAG data ingestion, and RAG chunking strategies.