Image Inputs for Production Multimodal LLM Apps in October 2026: Fix Orientation, Strip Metadata, Resize by Task, and Budget Image Tokens Before the Call

Most teams add image input to an LLM app the same way: take whatever the user uploaded, base64 it, and drop it into the message. It works in the demo. In production it causes a steady trickle of problems: phone photos that the model reads sideways, 12-megapixel screenshots that eat a large share of the request budget, uploads that quietly carry GPS coordinates into your logs, and "the model can't read this" complaints that turn out to be your own resize step blurring small text. This guide walks through a practical preprocessing pipeline for image inputs to multimodal models, what each step protects against, and how to measure whether it is working.

Image: Tony Webster via Wikimedia Commons (CC BY 2.0)

Why image preprocessing matters more than it looks

A vision-capable model does not see your file the way an image viewer does. The provider decodes it, often rescales it to fit internal limits, and converts it into a sequence of image tokens that count against your context window and your bill. Three consequences follow:

• Cost and latency scale with pixels. Most providers price images by resolution, either per tile or by a pixel-based formula. A full-resolution phone photo can cost many times more than a version sized to what the task actually needs, and larger inputs also add prefill time before the first token.
• The provider's resize is not your resize. If you send an image above the provider's limit, it gets downscaled server-side with a method and target size you do not control. Text that was readable at your resolution may not be readable at theirs.
• Metadata and orientation travel with the file. Camera files carry EXIF data, including an orientation flag and often location. Viewers honor the orientation flag; many decoding paths do not.

The fix is to do the decode, orient, resize, and strip steps yourself, deterministically, before the call. Then you know exactly what the model received.

Step 1: Validate the upload before you touch it

Treat user images as untrusted input, the same way you treat uploaded documents.

• Check the real format, not the extension. Open the bytes with an image library and read the detected format. Reject anything outside an allowlist (for example JPEG, PNG, WebP, and GIF if you need it). Convert formats your provider does not accept, such as HEIC from iPhones, to JPEG or PNG on your side.
• Cap file size and pixel count. Set a maximum byte size at the upload layer and a maximum width times height before full decoding. Highly compressed files can expand to enormous bitmaps in memory. Pillow, for example, has a MAX_IMAGE_PIXELS safeguard that warns or raises on very large images; keep it enabled and set it to a value that fits your workers.
• Handle animated and multi-frame files explicitly. Decide whether you take the first frame, sample a few frames, or reject. Do not let a 300-frame GIF go through by accident.

Step 2: Apply EXIF orientation, then strip metadata

Phones usually save the pixels in the sensor's native orientation and set an EXIF orientation tag that tells viewers to rotate on display. If your pipeline decodes pixels and ignores that tag, a portrait photo of a receipt reaches the model rotated 90 degrees. Models can sometimes cope, but accuracy on text, tables, and fine detail drops, and the failures look random because they depend on how each user held their phone.

The fix is a single call in most libraries. In Python with Pillow:

from PIL import Image, ImageOps

img = Image.open(path)
img = ImageOps.exif_transpose(img)   # rotate pixels to match the orientation tag
img = img.convert("RGB")             # drop alpha/palette modes for JPEG output

Once the pixels are upright, remove the metadata. Re-encoding into a new image without passing the original EXIF block drops it, including camera serials and GPS coordinates. This matters because image payloads often end up in request logs, traces, evaluation datasets, and support tickets. Removing location data at ingestion is far easier than chasing it through every downstream store later.

Image: Kprateek88 via Wikimedia Commons (CC BY-SA 4.0)

Step 3: Resize to what the task needs, not to the maximum

The right resolution depends on what the model has to read.

• Scene-level questions ("What room is this?", "Is there a person in the photo?") work fine at modest sizes. Downscale aggressively.
• Documents, receipts, and screenshots need small text to stay legible. Keep more resolution, or crop to the region that matters instead of shrinking the whole page.
• Charts and diagrams fall in between: labels must be readable, but large flat areas carry little information.

A practical approach is to define two or three named profiles, such as scene, document, and screenshot, each with a maximum long edge and an output format. Route each request to a profile based on the feature calling it, not on the image itself. Then read your provider's documentation for its maximum dimensions and how it counts image tokens, and pick profile sizes that stay at or under the provider's own downscale threshold. Staying under that threshold means the provider does not resize again, so what you tested is what the model sees.

PROFILES = {
    "scene":      {"max_edge": 768,  "format": "JPEG", "quality": 85},
    "document":   {"max_edge": 1600, "format": "PNG"},
    "screenshot": {"max_edge": 1400, "format": "PNG"},
}

def prepare(img, profile):
    p = PROFILES[profile]
    img.thumbnail((p["max_edge"], p["max_edge"]), Image.LANCZOS)  # keeps aspect ratio, never upsizes
    return img, p

The numbers above are starting points to tune, not provider limits. Use a high-quality resampling filter such as Lanczos when downscaling text-heavy images; cheaper filters blur thin strokes. Prefer PNG for screenshots and documents, where JPEG artifacts around text hurt recognition, and JPEG or WebP for photos, where they save real bandwidth.

Crop before you shrink

If the useful region is known, crop first. A form field, a product label, or a single chart in a dashboard screenshot can be sent at full detail while costing less than the whole frame at low detail. When your UI lets users select a region, send that region. When it does not, consider a cheap first pass that asks the model (or a simple detector) to locate the area of interest, then a second call on the crop.

Step 4: Use the provider's detail controls deliberately

Several APIs expose a setting that trades fidelity for cost, such as a low or high detail flag on image parts. Low detail usually means the provider sends a single small rendering of the image; high detail means it processes the image in more pieces. Do not leave this on a default you never chose. Tie it to your profiles: scene requests use low detail, document and screenshot requests use high detail. Check your provider's current documentation, since the exact behavior and naming vary between vendors and change over time.

Step 5: Budget image tokens before the call

If your app counts text tokens before sending a request, images need to be in that count too. Otherwise a conversation with a few attached photos can push past the context window or a tenant's budget without warning.

1. Estimate per image from your own measurements. Send a small calibration set of images at each profile size and record the input token usage the API reports. Build a lookup from (profile, width, height) to observed tokens. This is more reliable than reimplementing a provider's formula from memory.
2. Cap images per request. Decide a maximum number of images per turn and per conversation. When a user exceeds it, ask them to choose, or keep the most recent ones.
3. Drop or summarize old images in long conversations. Images from ten turns ago rarely need to be re-sent at full detail. Replace them with a short text description generated when they first arrived, and re-attach the original only if the user refers back to it.
4. Reconcile against actual usage. Log estimated versus reported tokens per request. If the gap grows, the provider changed something or your profiles drifted.

Step 6: Cache prepared images and keep requests stable

Preprocessing is deterministic, so cache it. Key the cache on a hash of the original bytes plus the profile name, and store the prepared output. This saves CPU when the same image is sent again, for example on retries or when a conversation is replayed. It also keeps the encoded image byte-for-byte identical across turns, which matters if your provider applies prompt caching to repeated prefixes: re-encoding the same photo slightly differently each turn defeats that cache.

For large images, prefer uploading once and passing a file reference or URL if your provider supports it, rather than base64-encoding the same payload into every request. Base64 adds roughly a third to the payload size and makes request logs heavy.

Step 7: Test with the images your users actually send

Image pipelines fail on edge cases that never show up in curated test sets. Build a small regression set from real (consented, scrubbed) traffic, and include:

• Portrait phone photos with each EXIF orientation value.
• HEIC uploads, transparent PNGs, and CMYK JPEGs.
• Dense screenshots with small UI text, and photographed documents with glare or skew.
• Very wide or very tall images, such as full-page scroll captures, where uniform downscaling makes text unreadable. Consider splitting these into vertical sections instead.

For each case, assert on the prepared output (dimensions, format, no EXIF, correct orientation) and on the model's answer to a fixed question. Run it whenever you change profile sizes, libraries, or model versions.

A compact checklist

• Validate the real format and cap bytes and pixel count before full decoding.
• Apply the EXIF orientation, then re-encode without metadata.
• Resize to a named profile chosen by feature, at or below the provider's own downscale threshold.
• Crop to the region of interest when you can.
• Set the provider's detail option explicitly per profile.
• Count image tokens with a calibrated lookup, cap images per turn, and summarize old ones.
• Cache prepared images by content hash so retries and repeat turns stay identical.
• Keep a regression set of awkward real-world images and run it on every change.

None of these steps is complicated on its own. Together they turn image input from the least predictable part of a multimodal app into one of the most controllable: you know what the model saw, what it cost, and why it got the answer it did.