--- author: - Shengbang Tong - David Fan - John Nguyen - | \ Ellis Brown - Gaoyue Zhou - Shengyi Qian - Boyang Zheng - Théophane Vallaeys - Junlin Han - Rob Fergus - Naila Murray - Marjan Ghazvininejad - Mike Lewis - Nicolas Ballas - Amir Bar - Michael Rabbat - Jakob Verbeek - | \ Luke Zettlemoyer - Koustuv Sinha - Yann LeCun - Saining Xie bibliography: - paper.bib date: 'March 3, 2026' title: | Beyond Language Modeling:\ An Exploration of Multimodal Pretraining --- ```{=latex} \newcommand{\suggestion}[2]{ \begin{tcolorbox}[ colback=blue!5, colframe=blue!70!black, arc=5pt, boxsep=5pt, left=2pt, right=2pt, top=2pt, bottom=2pt, boxrule=0.8pt, drop shadow=gray!50!white, enhanced jigsaw, before skip=12pt plus 2pt minus 2pt, % Space above the box after skip=12pt plus 2pt minus 2pt, % Space below the box ] \phantomsection\label{suggestion:#1}% \noindent\textbf{\textit{Suggestion #1:}} #2 \end{tcolorbox} } ``` ```{=latex} \newcommand{\suggref}[2]{\hyperref[suggestion:#1]{\textcolor{blue!50!black}{\textit{#2}}}} ``` ```{=latex} \newcommand{\displaytodo}[1]{#1} ``` ```{=latex} \newcommand{\displaytodo}[1]{} ``` ```{=latex} \newcommand{\pt}[1]{\displaytodo{\textcolor{blue}{[\textbf{PT:} #1]}}} ``` ```{=latex} \newcommand{\ks}[1]{\displaytodo{\textcolor{brown}{[\textbf{KS:} #1]}}} ``` ```{=latex} \newcommand{\jn}[1]{\displaytodo{\textcolor{red}{[\textbf{John:} #1]}}} ``` ```{=latex} \newcommand{\eb}[1]{\displaytodo{\textcolor{Emerald}{[\textbf{EB:} #1]}}} ``` ```{=latex} \newcommand{\df}[1]{\displaytodo{\textcolor{green}{[\textbf{DF:} #1]}}} ``` ```{=latex} \newcommand{\gz}[1]{\displaytodo{\textcolor{purple}{[\textbf{GZ:} #1]}}} ``` ```{=latex} \newcommand{\jh}[1]{\displaytodo{\textcolor{purple}{[\textbf{Junlin:} #1]}}} ``` ```{=latex} \newcommand{\fair}{{\color{fairblue}\heartsuit}} ``` ```{=latex} \newcommand{\nyu}{{\color{nyupurple}\vardiamondsuit}} ``` ```{=latex} \newcommand{\fairmark}{\textcolor{fairblue}{1}} ``` ```{=latex} \newcommand{\nyumark}{\textcolor{nyupurple}{2}} ``` ```{=latex} \newcommand{\theHalgorithm}{\arabic{algorithm}} ``` ```{=latex} \newcommand{\ours}{\textcolor{methodgreen}{TransNext}\xspace} ``` ```{=latex} \newcommand{\mix}[1]{{\fontfamily{lmss}\selectfont #1}} ``` ```{=latex} \newcommand{\texttok}{\textcolor{DiTblue5}{\texttt{T}}\xspace} ``` ```{=latex} \newcommand{\imtok}{\textcolor{DiTred4}{\texttt{I}}\xspace} ``` ```{=latex} \newcommand{\vidtok}{\textcolor{DiTred4}{\texttt{V}}\xspace} ``` ```{=latex} \newcommand{\actiontok}{\texttt{A}\xspace} ``` ```{=latex} \newcommand{\Language}{\textcolor{DiTblue4}{\textbf{Language}}\xspace} ``` ```{=latex} \newcommand{\Vision}{\textcolor{DiTred4}{\textbf{Vision}}\xspace} ``` ```{=latex} \maketitle ``` ```{=latex} \vspace{3em} ``` `\noindent`{=latex} ```{=latex} \begin{tcolorbox}[ colback=blue!3, colframe=blue!60!black, arc=4pt, boxrule=0.5pt, boxsep=3pt, left=6pt, right=6pt, top=4pt, bottom=3pt, enhanced jigsaw, drop shadow=gray!40!white, ]{\noindent\sffamily\bfseries Suggestions at a Glance}\\[-2pt] {\color{blue!30!black}\hrule height 0.3pt}\vspace{4pt} \begin{enumerate}[leftmargin=1.5em, itemsep=6pt, parsep=0pt, topsep=0pt, label={\small\textbf{S\arabic*.}}] \vspace{6pt} \item \textbf{Unify visual representations}: RAE bridges visual understanding and generation. \hfill \suggref{1}{\S\ref*{sec:ablation_visual_representation}} \item \textbf{Embrace diverse data}: multimodal co-training yields cross-modal synergy. \hfill \suggref{2}{\S\ref*{sec:data_composition}} \item \textbf{Unlock world modeling}: capabilities naturally emerge from unified multimodal pretraining. \hfill \suggref{3}{\S\ref*{sec:world_modeling}} \item \textbf{Use MoE}: MoE scales multimodal training efficiently and harmonizes capacity across modalities. \hfill \suggref{4}{\S\ref*{sec:arch_design},\ref*{sec:scaling}} \end{enumerate} \end{tcolorbox} ``` ```{=latex} \clearpage ``` `\hypersetup{linkcolor=black}`{=latex} `\tableofcontents`{=latex} `\clearpage`{=latex} Introduction ============ The foundation model era has been defined largely by the success of language pretraining [@OpenAI2022ChatGPT; @Gemini]. By scaling autoregressive models on trillions of text tokens, we have created systems with remarkable reasoning capabilities. Yet, fundamentally, text is a human abstraction---a lossy compression of reality. To borrow the allegory of Plato's Cave [@plato_republic], language models have mastered the description of shadows on the wall without ever seeing the objects casting them. They capture symbols well but miss the high-fidelity physics, geometry, and causality of the physical world. Beyond this philosophical limitation lies a hard, pragmatic ceiling: high-quality text data is finite and approaching exhaustion [@sutskever2025]. In contrast, the visual world possesses an endless stream of signal \`\`outside the cave", capturing the raw dynamics of reality that language misses. As a result, the path forward requires moving beyond the shadows to model the source directly. We turn to unified multimodal pretraining, treating the visual signal not as an auxiliary input, but as a first-class citizen alongside language. The scientific landscape of unified multimodal pretraining remains largely opaque. While recent efforts [@emu3; @zhou2024transfusion; @cui2025emu3] have begun to move beyond language-only pretraining, the design space is rife with confounding variables. Rather than jointly learning from vision and language from scratch, most current methodologies [@mogao; @deng2025bagel] rely on initialization from pretrained language models [@grattafiori2024llama3; @qwen3]. This paradigm prioritizes preserving existing language capabilities while adapting the model to become multimodal. Moreover, the knowledge already embedded in these pretrained backbones confounds any conclusions drawn about the multimodal training itself, making it difficult to disentangle what is learned from unified training versus what is inherited from language pretraining. Consequently, the fundamental dynamics and scaling relationships between vision and language remain poorly understood. In this work, we seek to provide empirical clarity to this landscape. We focus squarely on pretraining, as a model's core capabilities are largely acquired during this phase [@zhou2024lima]. We train a single model from scratch using the Transfusion framework, with next-token prediction for language and diffusion for vision, on diverse data including text, video, image-text pairs, and action-conditioned video. We conduct controlled experiments to isolate key variables and evaluate across a comprehensive spectrum of tasks, ranging from language evaluation and visual understanding/generation, to planning capabilities in world modeling. Specifically, we investigate the following axes: - **Visual Representation:** We evaluate a broad spectrum of visual representations, ranging from Variational Autoencoders (VAE) and semantic representations to raw pixels. We identify Representation Autoencoders (RAE) as the optimal representation. (`\Cref{sec:ablation_visual_representation}`{=latex}) - **Data:** We study a wide range of data mixtures, from pure text and video to paired image-text and action-conditioned video. We find minimal interference between modalities and even observe positive synergy in certain cases. (`\Cref{sec:data_composition}`{=latex}) - **World Modeling:** We extend our evaluation to the Navigation World Model (NWM) setting, formatting actions directly as text tokens. We demonstrate that physical prediction capabilities emerge primarily from general multimodal pretraining (e.g. video) rather than domain-specific data. (`\Cref{sec:world_modeling}`{=latex}) - **Architecture:** We study design choices for Mixture-of-Experts (MoE) in this unified multimodal setting and observe natural formation of modality separation and unification. (`\Cref{sec:arch_design}`{=latex}) - **Scaling Properties:** We conduct IsoFLOP experiments to derive scaling laws for vision and language during unified pretraining. We uncover a scaling asymmetry where vision is significantly more data-hungry than language, and find that MoE architectures effectively bridge this gap. (`\Cref{sec:scaling}`{=latex}) ```{=latex} \centering ``` ![ **Overview of our study.** **Top:** High-level model architecture. We train a single autoregressive model with next-token prediction for text and next-visual state prediction with flow matching. **Bottom:** We study five axes: visual representations, data, world modeling, architecture, and scaling. ](figures/setup/pipeline_v2.png){#fig:pipeline width="\\textwidth"} Experiment Setup {#sec:method} ================ Our goal is to study the science and design choices around unified multimodal pretraining. Across experiments, we keep the compute budget and training hyperparameters controlled within each ablation family. Unless otherwise specified, we evaluate at the end of pretraining (without any instruction tuning). We provide an overview of our training and inference protocols here, and more details in `\Cref{sec:extended_methodology}`{=latex}. Training and Inference {#sec:method_training_and_inference} ---------------------- #### Language modeling. Given a text sequence $(x_1, \dots, x_n)$, we minimize standard autoregressive cross-entropy: $$\mathcal{L}_{\text{LM}} = -\sum_{i=1}^{n}\log p_\theta(x_i \mid x_{