Multimodal AI: Vision, Documents & Audio with LLMs

Multimodal AI

Multimodal AI refers to models that process more than just text — most commonly images, but increasingly documents, audio, and video. A multimodal LLM can look at a screenshot and describe what's on screen, read a scanned invoice, analyze a chart, or answer questions about a photo, all through the same interface you already use for text.

This capability collapses a whole category of previously separate tooling. Tasks that once required a dedicated OCR engine, a document-parsing pipeline, or a specialized computer-vision model can now be a single API call with a plain-language instruction. The tradeoff is cost and precision: images consume tokens, and a general model is not always as accurate as a purpose-built one. This page covers how to work with images and documents effectively.

TL;DR

Multimodal models accept images alongside text in the same request — describe, extract, compare, reason over visual content with natural-language instructions.
Images cost tokens roughly proportional to resolution. High-resolution vision improves accuracy on dense content but multiplies cost — downsample when fidelity isn't needed.
Documents (PDFs, screenshots, scans) are a killer use case: OCR, data extraction, and Q&A over layout-heavy content without a separate pipeline.
Send images as base64 or URL; place the image before the text instruction in the content array.
Multimodal replaces bespoke OCR/vision pipelines for many tasks — but a general model isn't always as precise as a specialized one for high-volume, narrow jobs.
Vision is an attack surface too: images can carry prompt injection via embedded text.

Quick Example

Send an image with a question. The model reads the image and answers in the same call:

One request replaces an OCR step, a layout parser, and a field-extraction model.

Core Concepts

How Models "See"

A vision-capable model splits an image into patches, encodes each into the same embedding space as text tokens, and reasons over the combined sequence. Practically, this means the image is context — it occupies tokens, sits in the context window alongside your text, and the model attends to both together. There's no separate "vision API"; it's one multimodal request.

Image Token Cost

Images are billed as tokens, roughly proportional to their pixel area. A small thumbnail costs little; a full-resolution screenshot costs a lot. Modern high-resolution vision (supporting larger images on the long edge) improves accuracy on dense content — small text, detailed charts, packed UIs — but a full-resolution image can cost several times more tokens than a downsampled one.

The practical rule: send the smallest resolution that preserves the detail you need. For reading dense text or fine chart data, use higher resolution; for "what's in this photo?", downsample aggressively. Measure token cost on representative images before assuming.

Input Formats

Place the image block before the text instruction in the content array — the model reads in order, and "here's an image, now do X" works better than the reverse.

What Multimodal Is Good At

Document Understanding

The highest-value use case. Multimodal models read PDFs, scans, screenshots, and photos of documents — extracting fields, answering questions, and preserving layout awareness (which number is the total vs the subtotal) that plain OCR loses. Invoices, receipts, forms, contracts, and reports all become a single extraction call. PDFs can often be sent directly as a document block, letting the model reason over both text and visual layout.

Screenshots and UI Understanding

Feed a screenshot and ask what's on screen, why an error appears, or what a user should click. This underpins computer-use agents, automated QA, and support tools that reason over what the user sees.

Charts, Diagrams, and Visual Data

Extract data from a chart image, explain a diagram, or transcribe a whiteboard. High-resolution vision matters most here — the difference between reading a chart's axis labels and guessing.

General Image Q&A

Describe photos, identify objects, compare images, generate alt text for accessibility. Broad but usually lower-stakes than document work.

Best Practices

Right-Size Every Image

Downsample before sending unless you need the fidelity. A UI screenshot for "is there an error?" can be far smaller than a scanned contract you're extracting clauses from. This is the single biggest lever on multimodal cost.

Ask for Structured Output

Pair vision with structured outputs so extractions come back as validated JSON, not prose you have to re-parse. Document extraction is far more useful when the fields land in a schema.

Guide the Model to the Relevant Region

For dense images, tell the model where to look ("read the total in the bottom-right table"). This improves accuracy and reduces the chance it fixates on the wrong region.

Verify High-Stakes Extractions

A model can misread a digit on a low-quality scan just as a human can. For financial or legal extraction, validate against expected ranges or a second pass, and surface confidence to the user rather than silently trusting the result.

Treat Image Text as Untrusted

Text embedded in an image (a document, a screenshot) can carry prompt-injection payloads. If the model reads and acts on image content, apply the same "tool output is untrusted data" discipline you'd use for fetched web pages.

Common Mistakes

Sending Full Resolution by Default

Expecting Perfect OCR on Poor Scans

A general vision model reads well but isn't infallible on faded, skewed, or low-DPI scans. For high-volume, precision-critical OCR, a specialized engine may still win. Validate critical fields.

Putting the Instruction Before the Image

FAQ

Do multimodal models replace OCR and document-parsing tools?

For many tasks, yes — they read documents, preserve layout awareness, and extract fields in one call, replacing a multi-stage OCR-plus-parser pipeline. But for extremely high-volume, narrow, precision-critical extraction, a specialized OCR engine can still be cheaper and more accurate. Prototype with a multimodal model; switch to a specialist only if cost or accuracy demands it.

How much do images cost?

Roughly proportional to pixel area, billed as tokens. A downsampled image costs little; a full-resolution one can cost several times more. High-resolution vision improves accuracy on dense content but multiplies cost. Right-size images to the detail the task actually needs, and measure on representative inputs.

Can I send a PDF directly?

Often yes — many models accept PDF document blocks and reason over both the text and the visual layout (tables, figures, positioning). This is more powerful than extracting raw text first, because layout carries meaning. Check page and size limits for your provider.

Do these models handle audio and video?

Support varies by model and is expanding. Some accept audio for transcription and understanding; video is less common and often handled as sampled frames. Image and document understanding are the most mature multimodal capabilities today — treat audio/video support as model-specific.

Is vision a security concern?

Yes. Text embedded in an image can carry prompt-injection instructions the model may follow. If your app reads image content and takes actions on it, treat that content as untrusted data and apply AI guardrails — the same discipline as fetched web pages.