Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generation

Source

Raw Markdown: paper_explorative-modeling-2026.md
Paper PDF: paper_explorative-modeling-2026.pdf
Preprint: arXiv 2607.27372v1
Official project page: https://explorative-modeling.github.io/
Official author blog: https://alexiglad.github.io/blog/2026/explorative_modeling/
Official code: https://github.com/alexiglad/XM
Official Hugging Face paper page: https://huggingface.co/papers/2607.27372
Official X thread: Alexi Gladstone, 2026-07-31
IMLE prior-art thread: Ke Li, 2026-08-03
XM author response: Alexi Gladstone, 2026-08-03
Repository snapshots: papers/explorative-modeling-2026/x_thread_alexiglad_2083230922196107288.json, x_thread_kl_div_2084059577247797554.json, x_thread_alexiglad_2084322087691497979.json, their normalized Markdown snapshots and archived media, the linked pre-announcement capture x_post_alexiglad_2081834973456433485.json, official_project_snapshot.md, official_blog_snapshot.md, github_repo_snapshot.md, and github_readme_snapshot.md

Status And Credibility

The paper was submitted to arXiv on 2026-07-29. The rendered PDF labels it Preprint. Under review. No accepted venue or independent replication was found at ingest time.

The authors are Alexi Gladstone and Heng Ji at UIUC and Yilun Du at Harvard. This is a current paper from a credible generative-modeling team, accompanied by an official project page, author blog, full launch thread, and an Apache-2.0 code repository. The evidence is therefore worth tracking as an important source, but all benchmark and scaling claims remain author-reported preprint results.

The public code snapshot is partial. It contains image/video training code and MDLM-related modules, but the README says the RAE-specific implementation, masked-diffusion language-model experiments, and control-task code are still forthcoming. It had no model weights, checkpoints, or GitHub release at ingest time. The headline RAE, XMDLM, Explorative Policy, and Explorative World Model results should not yet be described as fully reproducible from the public repository.

Core Claim

Explorative Modeling (XM) changes the training objective rather than the inference procedure. For each observed target, the model generates several latent-conditioned candidates and trains only on the closest candidate. The intended effect is to let different latent regions commit to different valid outputs instead of pushing every prediction toward an average between modes.

The paper uses this mechanism in two ways:

Hybrid XM: add candidate exploration to an existing diffusion/flow, Jumpy, or masked-diffusion language model while keeping its inference procedure unchanged.
End-to-end XM: train a direct generator whose sampling path is the same at training and inference, moving candidate-search cost into training rather than paying for many generation steps at inference.

The phrase generative expressivity is the paper’s name for how many distinct modes a training objective permits its loss minimizers to retain. It is a useful proposed diagnostic, not yet a settled third scaling-law variable established across frontier model families.

Method

Forward XM

Forward XM fixes one real target $x$ , samples $K$ candidate generations, and backpropagates only through the candidate with the smallest reconstruction loss:

L_{F} (θ) = E_{x} [k \in {1, \dots, K} min J (G_{θ} (z_{k}, c), x)] .

This is structurally a best-of- $K$ , multiple-choice, or winner-take-all objective. The paper explicitly does not claim to invent best-of- $K$ ; its novelty claim is the generalized interpretation of candidate width as a training-time factorization and scaling variable for generative models.

Forward XM gives every observed target a nearest candidate and therefore pushes toward support or mode coverage. Its cost grows with candidate count. The paper estimates that in its FLOP-efficient implementation, XM- $K$ costs roughly $(K + 2) /3$ ordinary training steps because additional candidates require forward passes but only the winner receives a backward pass. Candidate generation can be folded into the batch dimension.

Reverse XM

Reverse XM fixes one generated candidate $\overset{y}{^}$ and selects its nearest target among $K$ data samples:

L_{R} (θ) = E_{\overset{y}{^}} [k \in {1, \dots, K} min J (\overset{y}{^}, x_{k})] .

This direction is cheaper when many targets can be searched without extra model forward passes, but it is mode-seeking. The paper’s theory and the author’s X follow-up both state that bare Reverse XM can collapse or memorize; it needs an entropy or coverage constraint. The reported reverse-language experiment uses vector search plus a train-coverage heuristic, is not reported as a full benchmark result, and is not open sourced.

flowchart LR
  C["Condition / history"]
  Z["K latent candidates"]
  G["Shared generator"]
  Y["K candidate outputs"]
  J["Distance to observed target"]
  W["Winning candidate only"]
  U["Parameter update"]

  C --> G
  Z --> G --> Y --> J --> W --> U

Theory Boundary: Smooth Mixture Versus Implemented Hard Minimum

The paper gives a clean likelihood interpretation for a smooth Forward-XM objective that scores the average density of $K$ candidate kernels. Under squared error and stated regularity assumptions, that smooth objective is an expected negative log-likelihood for a sampled $K$ -component Gaussian mixture and approaches a forward-KL objective as $K$ grows.

The implemented objective is the hard minimum. The appendix explicitly notes that the large- $K$ hard-min limit is a coverage objective: it can reward any generator whose support comes close to every data point without fixing how probability mass is distributed over that support. This distinction is central for probabilistic time-series and world-model use:

Forward XM can be strong evidence for proposal coverage without being evidence for calibrated mode probabilities.

The paper also notes a boundary in its proposed generative-expressivity definition. Exact nonparametric autoregression and continuous-time diffusion can already represent all data modes at their optima, so the at-optimum mode-count definition does not distinguish consistent factored objectives. The practically relevant deficit moves to per-prediction residual multi-modality, optimization, finite model/data scale, and finite sampling procedures. The “third pretraining axis” claim should therefore be read as an empirical finite-regime allocation hypothesis, not a theorem that existing exact objectives have a finite global mode cap.

Main Evidence

Setting	Reported result	What it establishes	Boundary
RAE ImageNet-256	XRAE reaches the baseline target with $6.2 \times$ fewer processed samples and $4.1 \times$ fewer FLOPs.	Exploration can be a compute-efficient training allocation in this recipe.	“Data efficiency” means samples processed to a target metric at fixed batch size, not $6.2 \times$ fewer unique images. RAE-specific code is not yet released.
RAE final quality	XM-2 reports unguided gFID $1.43$ and FDr $^{6}$ $3.91$ , versus $1.55$ and $4.42$ for the reproduced RAE baseline.	A small $K$ improves the paper’s strongest image recipe without guidance.	Author-reported ImageNet-256 result; no independent reproduction.
Parameter sweep	A Large model with XM-5 outscales an XLarge baseline with $47%$ more parameters.	Candidate width can substitute for some parameter scale in this experiment.	One architecture family and metric; not a general parameter-equivalence law.
Image/video $K$ sweeps	Image and video FID/FVD improve monotonically over the tested $K$ values; some video settings improve by more than $20%$ .	Direct within-recipe evidence that candidate width can improve sample quality.	Tested ranges are finite; monotonicity beyond the maximum $K$ is an extrapolation.
Masked-diffusion language model	XM-8 improves the reported generative-perplexity/entropy frontier at 8 and 256 sampling steps.	Candidate exploration can transfer to a discrete masked generator.	The model is small: 6 blocks, 384 dimensions, context 256, 210k steps. It is not evidence for frontier autoregressive LMs.
Scale sweeps	XM-5 gains rise from $13%$ to $23%$ over Small-to-XLarge models and from $7%$ to $36%$ as an XLarge model consumes more training steps/data.	Benefits increase over the tested parameter and processed-data ranges.	This is not yet a fitted three-variable compute-optimal scaling law or a frontier-scale run.
Fixed-dataset video generalization	Best validation FVD improves from $37.5$ without exploration to $30.0$ with exploration.	Positive evidence that higher $K$ can reduce overfitting in one SSv2 setup.	One video dataset, architecture, and metric; not a general theorem that exploration prevents memorization.
Robomimic behavior cloning	XM-10 matches or exceeds the reproduced Diffusion Policy success rates with one NFE versus 100.	Strong inference-step reduction in the tested state-observation control setup.	CNN models, changed robomimic version, no released control code, and no wall-clock/training-cost frontier.
Maze2D goal-conditioned world modeling	XM-10 averages score $130.0$ at 2.3 NFE versus reproduced Diffuser $127.2$ at 192 NFE; individual NFE reductions span $16$ - $256 \times$ .	A direct low-NFE goal-conditioned trajectory-generation result.	Single-task Maze2D protocol, altered training steps/goal radius, no broad action-conditioned world-model evidence.

Root X-Post Claim Audit

The authenticated root post is preserved separately from replies and paper interpretation.

Root-post claim	Paper support	Calibration
“A third pretraining axis beyond parameters and data”	The paper presents matched within-family $K$ sweeps, FLOP comparisons, and gains that rise over tested model size and training duration.	Promising empirical hypothesis; not yet a universal or frontier-scale compute-optimal law.
“Scaling exploration monotonically improves existing models across images/video/language”	Monotonic improvements are reported over tested $K$ values for image/video and a better XMDLM perplexity-entropy frontier.	Scope is the tested architectures and ranges; autoregressive LMs remain a reported hard case.
“In the simplest case, it’s just a for loop”	Forward XM is a short best-of- $K$ loop and can be batched.	The mechanism is simple, but cost, memory mode, candidate coupling, collapse controls, and calibration are not.
$6.2 \times$ data, $4.1 \times$ FLOPs, $47%$ parameters, gFID $1.43$	All numbers appear in the paper under the specific RAE/DiT protocols above.	These numbers are not one interchangeable efficiency score; each comes from a different target comparison.
“Trade training compute for generalization”	SSv2 validation FVD improves as $K$ increases in the fixed-dataset overfitting experiment.	One controlled positive case, not broad evidence across modalities or distribution shifts.
“Up to $256 \times$ less inference compute on control tasks”	NFE counts support the headline on Robomimic and Maze2D.	NFE is not end-to-end wall-clock, memory, energy, training compute, or system latency.

A later same-author reply speculates that exploration could yield $100 \times$ improvements for training a Veo-class model. That is social speculation, not a paper result, and is not propagated as evidence.

IMLE Priority And Relation Audit

Ke Li, an author of the original IMLE and conditional-IMLE papers, published a 45-part X thread arguing that XM substantially rebrands the IMLE line. The authenticated capture preserves 46 author posts—the numbered thread plus one addendum—and 45 claim-bearing screenshots. The thread is first-party prior-art commentary, not peer review; its technical claims were checked against the local primary papers.

The strongest verified point is objective-level identity. Conditional IMLE, published in IJCV 2020, optimizes

E [i \sum j min L (T_{θ} (x_{i}, z_{i, j}), y_{i})] .

End-to-end Forward XM optimizes the same per-condition hard minimum after renaming $m$ to $K$ , the input to $c$ , and the loss to $J$ . The basic direct conditional estimator and winner-only gradient routing are therefore prior art, not a 2026 algorithmic invention.

The XM paper does acknowledge both best-of- $K$ and original IMLE. It describes IMLE as a specific end-to-end Forward-XM instance with a shared global sample pool and places novelty in the exploration/generative-expressivity interpretation. This is a defensible umbrella taxonomy, but it underemphasizes that peer-reviewed conditional IMLE already used separate candidates for every conditioning input and is essentially identical to standalone conditional Forward XM.

The reverse inclusion claim must remain scoped:

End-to-end conditional Forward XM is conditional IMLE at the hard-objective level.
Original unconditional IMLE differs mainly by drawing one global model-sample pool shared across datapoints.
Hybrid XM is not literally the 2018/2020 algorithm: its candidates may use different corruptions of the same datapoint and train one intermediate prediction inside a factored sampler.
Reverse XM changes the assignment direction. SCADE supplies a related mode-seeking/reverse-cross-entropy precedent inside a NeRF/depth-prior system, but not the same dataset-level language-model procedure used in XM.

The durable novelty verdict is therefore narrower than the paper’s broad presentation: XM’s strongest new evidence is the candidate-width scaling program, hybrid use inside modern factored generators, Forward/Reverse organization, and cross-modal experiments—not nearest-of- $K$ target matching itself.

XM Author Response And Reconciled Verdict

Alexi Gladstone published an unnumbered root plus a contiguous 12-part response on 2026-08-03. The authenticated capture contains 14 same-author posts including an addendum linking Ke Li’s thread, while 30 posts from the public conversation remain in the raw JSON. The response concedes that XM does not invent best-of- $K$ , describes end-to-end Forward XM as an independent convergence on the IMLE/MCL/SOG mechanism, promises more IMLE-family and SCADE citations in v2, and argues that XM’s contribution is the broader generative-expressivity framework and empirical scaling program.

The response improves the framing of the dispute, but it does not erase objective-level priority:

Response claim	Source audit	Durable reading
Neither XM nor IMLE invented the generic best-of- $K$ /winner-take-all mechanism.	Supported by competitive mixture-of-experts and Multiple Choice Learning antecedents; the XM v1 paper already says it does not claim best-of- $K$ .	IMLE still remains the historically relevant data-to-generated-sample and likelihood-framed antecedent, while conditional IMLE remains the exact direct conditional objective precedent.
XM can recover original IMLE’s shared-pool form.	True for the population objective: expectation distributes over the sum of datapoint losses, so sharing one i.i.d. sample pool or drawing an independent identically distributed pool per datapoint gives the same expected scalar objective.	The finite Monte Carlo algorithms are not identical: a shared pool reuses candidates and couples assignments/gradient noise; fresh per-target pools do not. Conditional IMLE is the exact finite per-condition analogue used by direct Forward XM.
XM as a framework generalizes IMLE.	Defensible as an umbrella statement because hybrid and Reverse XM are outside the original IMLE algorithm.	Under direct conditional generation and hard winner-only matching, the inclusion reverses equally well: Forward XM is conditional IMLE up to notation. Framework inclusion MUST NOT be used to claim estimator novelty.
More than 90% of XM results “go against IMLE theory.”	No denominator or claim ledger supports the percentage. Most main experiments are outside original IMLE’s empirical scope, especially hybrid diffusion/flow/Jumpy/MDLM experiments, but being unstudied is not the same as contradicting a theorem.	Read this as rhetorical emphasis, not a measured result. Reverse XM rejects IMLE’s mode-covering preference, while most hybrid results neither test nor falsify IMLE’s MLE-equivalence theorem.
Hybrid XM and exploration scaling are new.	No exact IMLE-family antecedent was found for wrapping arbitrary-timestep diffusion/flow/MDLM predictions with the paper’s candidate search, and the model/data/ $K$ sweeps are the clearest distinct evidence. A universal “no prior work” claim cannot be established from a bounded literature audit.	This is XM’s strongest defensible novelty layer: integration, scaling experiments, and a unified research program rather than the nearest-candidate estimator.
Reverse XM shows MLE is not the mechanism that matters.	Bare Reverse XM has no likelihood objective for the trained generator and is mode-seeking; SCADE already used the same sample-to-nearest-target structural direction in a CVPR 2023 NeRF system. XM adds an explicit symmetric taxonomy and reports a coverage-constrained language-model experiment.	Reverse XM shows that likelihood is not necessary for every useful mode-seeking objective. It does not falsify the MLE interpretation of Forward XM or IMLE; likelihood and candidate-set expressivity can both explain different properties of the same forward objective.
Guidance helps by pushing away from a mode-collapsed or conditional-mean model.	Classifier-free guidance algebraically extrapolates the conditional score or vector field away from its unconditional counterpart. However, the unconditional diffusion score is the score of a marginal distribution—not generally a collapsed generator or a global conditional mean—and diffusion denoisers can induce multi-modal samples despite conditional-expectation parameterizations.	The geometric “push away” description is supported; the stronger causal story that guidance works by escaping mode collapse or secretly increasing candidate exploration is plausible but unproven and SHOULD NOT be stated as an established mechanism.
The new “Mode Forcing” term is opposite to CHIMLE’s use.	Supported. CHIMLE names spurious-mode creation or real-mode removal by diffusion KL terms a failure called mode forcing; the forthcoming XM-side paper uses the name for intentionally making a loss minimizer land on modes.	The terminology collision is real but not evidence of copied theory. A 2026-07-27 X post verifies that a three-paper program and unifying theory were announced before the dispute, but the Mode Forcing paper is not yet public, so its content and claimed prediction chronology remain independently unauditable.
v2 will cite SCADE and IMLE follow-ups.	This is a public commitment in the response. As of 2026-08-03, arXiv still exposes only v1.	Treat the promised repair as pending, not completed. Independent-development history cannot be externally verified and does not change prior-art attribution.

The mathematical pool distinction is worth making explicit. Original unconditional IMLE uses one shared pool $S_{1 : m}$ :

E_{S_{1 : m}} [i \sum j min d (S_{j}, x_{i})] = i \sum E_{S_{1 : m}} [j min d (S_{j}, x_{i})] .

Replacing each occurrence on the right by an independent pool $S_{1 : m}^{(i)}$ with the same law leaves every expectation unchanged. This validates Alexi’s population-objective inclusion claim, while preserving Ke Li’s finite-algorithm point that conditional IMLE already implements separate per-condition candidates exactly.

The response also sharpens a theory boundary rather than resolving it. The XM paper itself acknowledges that its at-optimum global mode-count definition saturates for exact autoregression and continuous-time diffusion, and leaves the noise-searching continuous hybrid objective and composition across sampler steps to future work. The “third axis” is therefore best treated as a promising finite-model, finite-data, finite-optimization allocation hypothesis. The reported scale sweeps support it over tested ranges; they do not yet identify a universal causal mechanism or a frontier-scale compute-optimal law.

Relation To Nearby Sources

VJEPA And FlowWM

VJEPA makes the future latent distribution explicit but evaluates a unimodal diagonal-Gaussian head. FlowWM generates several future DINOv3 trajectories through conditional flow matching and tests geometric mode coverage, but still lacks calibrated mode mass and action conditioning.

XM adds a third option: retain a simple or direct generator, sample $K$ latent-conditioned candidates during training, and train only the closest candidate. This is a useful candidate-coverage mechanism, but it does not replace FlowWM-style density modeling when calibrated probabilities are required. A strong combined experiment would compare:

deterministic point prediction;
Forward XM over $K$ raw or latent trajectories;
conditional flow/diffusion over trajectories;
flow/diffusion plus XM;
the same models under mode-coverage, calibrated-mass, invalid-region, tail-risk, and decision-utility metrics.

Probabilistic JEPA Predictor Idea

For Probabilistic JEPA Predictor Using Flow Matching, XM is a simpler and highly relevant baseline. A JEPA-style predictor could draw $K$ future latent trajectories and minimize distance only for the trajectory closest to the observed future. This directly tests whether candidate width prevents a deterministic future latent from averaging separated regimes.

The key boundary is that the hard minimum does not determine sample frequencies. If XM produces recovery, degradation, and failure trajectories but assigns them the wrong probabilities, it succeeds as a proposal generator and fails as a calibrated predictive belief. The experiment should therefore report both coverage and mass calibration rather than only best-of- $K$ error.

Energy-Based Transformers

Energy-Based Transformers exposes candidate scoring and gradient-based search but reports many-mode failures. XM supplies a plausible training-time pairing mechanism for such models, and the Explorative Modeling paper explicitly proposes combining the two. This remains future work: there is no experiment showing that XM fixes EBT’s many-mode energy landscape or that gradient-based latent search preserves the inference prior.

Foundation TSFM Relevance

Agenda slot	Verdict	Evidence	Missing pieces
Multi-modal future distributions	partially closes	Outside numeric time series, Forward XM supplies a simple candidate-width objective that improves mode/sample coverage across image, video, and masked-language generation; the paper also tests goal-conditioned trajectories.	Needs calibrated mode probabilities, unequal/rare branches, invalid-between-mode tests, and direct multivariate time-series evidence.
Generation and editing	adjacent	Candidate exploration improves several reconstructive generators and supports direct low-NFE trajectory generation.	Needs dense numeric fidelity, constraints, irregular timing, events, and controllable edits in time-series space.
Control and counterfactuals	adjacent	Explorative Policy and Maze2D Explorative World Model give state-observation behavior-cloning and goal-conditioned planning evidence.	Needs typed actions/control inputs, stochastic action consequences, broader multi-seed closed-loop tasks, real-system transfer, and simulator-exploitation audits.
Dynamic compute and scaling	adjacent	Moves candidate-search width into training, reports FLOP-matched gains, and observes a rising compute-optimal $K$ over tested ranges.	Needs a fitted joint frontier over parameters, unique data, training steps, $K$ , memory traffic, and wall-clock cost at TSFM scale.
Benchmark hygiene	warning	The paper reproduces key Diffusion Policy and Diffuser baselines under its changed setups and reports NFE explicitly.	NFE must remain separate from training compute, wall-clock, memory, energy, code availability, and search/hyperparameter cost.

Limitations And Gotchas

This is a fresh under-review preprint with no independent replication.
Best-of- $K$ is prior art; the new claim is the generalized interpretation, Forward/Reverse framing, and scaling evidence.
The strongest “third axis” evidence is finite-range within-family experimentation, not a frontier-scale scaling-law fit.
Processed-sample efficiency is not unique-data efficiency. Repeated ImageNet samples over more training steps should not be described as a larger corpus.
Hard-min Forward XM is a coverage objective in the large- $K$ limit and does not guarantee calibrated mode mass.
Reverse XM is mode-seeking and can collapse or memorize without entropy or coverage control.
Continuous hybrid XMs search over different noise corruptions of the same datapoint. The paper says the clean smooth-mixture theory does not directly analyze how these gains compose across factored sampler steps.
The reported “end-to-end” definition means that the model’s sampling procedure during training matches inference. It does not mean the whole perception, state estimation, planning, control, and deployment stack is jointly trained.
Training cost shifts upward even when inference NFE falls. End-to-end wall-clock, memory, energy, accelerator utilization, and total training-plus-serving break-even are not reported.
The current code release does not cover all headline experiments and has no released weights or checkpoints.
The video “world-model” setup conditions on frames 0, 1, and the goal frame 9 while generating intermediate frames; it is goal-conditioned visual generation, not a general action-conditioned latent-state simulator.
Autoregressive language modeling remains a weak point because it lacks an obvious latent to search and each next-token conditional may already be relatively narrow.

Open Questions

Does Forward XM preserve calibrated probabilities, or only ensure that at least one candidate reaches each common mode?
Can a soft-mixture objective retain the empirical gains of hard winner-take-all training while restoring a likelihood/calibration interpretation?
How should $K$ scale jointly with parameters, unique data, training duration, horizon, mode count, and accelerator memory?
Does XM improve direct multivariate time-series generation under rare regimes, irregular sampling, and cross-channel constraints?
Can XM prevent latent-state averaging in JEPA-style predictors without causing the decoder to ignore the candidate latent?
Does a flow/diffusion predictor plus XM outperform either mechanism alone under matched total training and serving cost?
Can Reverse XM use a principled entropy or coverage objective rather than dataset-removal heuristics and vector-search engineering?
What is the total-cost break-even point where extra Forward-XM training pays back lower inference NFE in a deployed control or generation service?
Do Explorative Policy and Explorative World Model remain competitive under image observations, stochastic dynamics, broader tasks, independent seeds, and real closed-loop transfer?

Alex Open Research Wiki

Explorer

Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generation

Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generation

Source

Status And Credibility

Core Claim

Method

Forward XM

Reverse XM

Theory Boundary: Smooth Mixture Versus Implemented Hard Minimum

Main Evidence

Root X-Post Claim Audit

IMLE Priority And Relation Audit

XM Author Response And Reconciled Verdict

Relation To Nearby Sources

VJEPA And FlowWM

Probabilistic JEPA Predictor Idea

Energy-Based Transformers

Foundation TSFM Relevance

Limitations And Gotchas

Open Questions

Graph View

Table of Contents

Backlinks

Alex Open Research Wiki

Explorer

Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generation

Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generation

Source

Status And Credibility

Core Claim

Method

Forward XM

Reverse XM

Theory Boundary: Smooth Mixture Versus Implemented Hard Minimum

Main Evidence

Root X-Post Claim Audit

IMLE Priority And Relation Audit

XM Author Response And Reconciled Verdict

Relation To Nearby Sources

VJEPA And FlowWM

Probabilistic JEPA Predictor Idea

Energy-Based Transformers

Foundation TSFM Relevance

Limitations And Gotchas

Open Questions

Related Pages

Graph View

Table of Contents

Backlinks