Field noteNov 16, 20265 min read1 source

Meta's Vision Encoders in Multimodal Models

In one minute

Meta researchers compared four ways to equip multimodal models with vision, training them under a controlled setup. Their results suggest that dedicated visual capacity matters more than separate encoder pretraining or a separate encoder boundary—a finding with implications for compute budgets and inference latency.

What changed: A new study finds that dedicated vision parameters matter more than separate pretraining or a pre-fusion encoder, with encoder-free models needing about three times the compute to match visual-question-answering performance.

What happened

In “Unbundling Multimodal Vision Encoding,” Ellis Brown, Chris Hoang, Shengyi Qian and co-authors examine three design choices bundled into a conventional vision encoder: a visual prior learned through separate pretraining, parameters dedicated to vision, and a position before visual and text tokens interact. The study compares four recipes: a pretrained encoder, an encoder trained from scratch alongside the multimodal model, an encoder-free model that sends image patches into the backbone, and “in-fusion,” which puts vision-specific modules inside the backbone.

The researchers trained the recipes from random initialization, using the same text and image-text training pairs, backbone architecture and training recipe, with images at 224×224 during pretraining. They compare visual question-answering performance using an average across four categories from 16 Cambrian benchmarks, after instruction tuning on Cambrian-7M with 448×448 images. They also report text benchmarks and account for training across stages in lifetime compute. Their budgets range from 6.5×10²⁰ to 2.9×10²² FLOPs; the fixed instruction-tuning step used to evaluate checkpoints is not counted.

The study finds that separate encoder pretraining is not beneficial at matched lifetime compute: training the encoder alongside the backbone performs on par with encoders pretrained on the same data. Longer pretraining can improve ImageNet recognition accuracy without improving VQA, even when its compute is not charged to the budget. By contrast, removing dedicated vision capacity hurts visual understanding. At 1.7B, encoder-free models trail the scratch encoder by 4.1–6.4 Avg. VQA points. Giving the backbone the encoder’s 0.4B parameters as extra width, depth or both improves text results but moves VQA by at most 0.4 points.

In-fusion preserves dedicated vision capacity while placing it inside the backbone. In the reported design, vision tokens use vision-only modules in the first eight layers. At 1.7B, this configuration scores 45.0 Avg. VQA, versus 44.4 for the scratch encoder; it also performs comparably across model sizes in the researchers’ scaling experiments. In tests using a pretrained 1.7B language model, in-fusion, early-layer mixture-of-transformers and a scratch encoder perform comparably, and all lead encoder-free. For serving, the researchers measure one H200 handling 16 concurrent requests, each with one 448-pixel image and 256 text tokens. In-fusion has lower time to first token than the separate encoder at every tested vision-only budget.

What the archive adds

New Runtime’s earlier coverage noted Meta AI’s release of meta-encoder-v1 weights on 2026-11-12. This study is a new research result about multimodal architecture, not a restatement of that distinct model release.

Trend signal

This does not yet fit a pattern New Runtime tracks.

Why now

The researchers frame the comparison around a practical accounting problem: published models often differ in data, architecture, training stages and compute, making it hard to isolate what the vision encoder contributes. Their lifetime-compute comparison counts the encoder’s pretraining rather than treating it as free, and they separately test whether the conclusions hold when adding vision to a pretrained language model.

Why it matters

For a team building a multimodal model, the result challenges two assumptions that can shape a costly architecture choice: that a vision encoder must be separately pretrained, and that visual processing must happen in a separate component before fusion. In these experiments, the useful resource is dedicated capacity for vision, not necessarily the conventional encoder boundary or its pretraining history.

That distinction gives builders a more specific design question. Rather than simply removing the encoder or expanding the general-purpose backbone, they can test whether vision-specific modules placed early in the shared model preserve visual performance while improving serving latency. The reported figures are promising, not a guarantee for every workload: teams still need evaluations on their own image types, tasks, data and deployment constraints. But the study offers a controlled comparison to guide those experiments and a reminder to count all training stages when comparing architectures under a fixed budget.

Limits and uncertainty

The authors train their own recipes on common data to isolate design choices; the study does not establish that its conclusions hold for every dataset, model family or production workload. Its encoders see the researchers’ training images, and the authors note that released encoders such as SigLIP 2 train on far more data, so the results do not rule out benefits from new data those models bring. The reported serving test uses one H200 and a specific concurrency and input configuration. Independent replication and evaluations on other hardware and workloads would help establish how broadly the performance and latency trade-offs transfer.

Skill to build now

When evaluating a multimodal architecture, compare designs at a matched total training-compute budget, including any encoder pretraining. Then test whether early vision-specific modules inside the backbone can preserve performance against a separate encoder. Measure both task quality and time to first token on the image sizes, concurrency and hardware your deployment actually uses.

Reader takeaway

The study’s central distinction is between dedicated vision capacity and a standalone vision encoder. In its controlled experiments, models need the former more than they need separate pretraining or pre-fusion placement. For builders, that is a case for testing where visual parameters live—and accounting for the full training budget—rather than treating the usual encoder recipe as inevitable.

Evidence / sources

  1. 1unbundling.github.io
All Field Notes

Next reads

Read next

Open the graph
  1. 01topicMultimodalExplore the multimodal topic hub.
  2. 02archiveField NotesOpen the latest editorial analysis.
  3. 03sourcesSourcesSee the original sources this archive cites.
Who read this page?Machine requests, hidden until opened

Loading the privacy-safe route aggregate...

Open the JSON contract