Inference Dynamics
Summary
Inference dynamics tracks how a model’s internal state changes during a forward pass or repeated inference computation. It is different from Training Dynamics: the question is not how optimization shapes weights over training, but how representations, predictions, calibration, memory, or latent state evolve while a trained model answers.
Current Evidence
Is One Layer Enough? is the current tabular anchor. It studies six tabular foundation models and finds staged inference over depth: latent mapping, feature engineering and labeling, prediction ensembling, and prediction calibration. The paper’s tuned-decoder results show that intermediate states can already contain predictive information before they are aligned with the original final decoder.
The source also provides a small looped-depth result: a single nanoTabPFN block reused six times can match a six-layer nanoTabPFN baseline in the reported setting, while a one-pass one-layer model underperforms. This supports repeated state refinement as a plausible tabular inference mechanism, but only in a small static-tabular setting.
PoLar turns depth interventions into an input-conditioned execution policy over a frozen pretrained LLM. It predicts contiguous layer segments and assigns skip, keep, or repeat to each segment. The paper reports pass@1 gains and, for one Qwen latency study, fewer executed layers and lower wall-clock latency. This is stronger than an after-the-fact probe because it changes the forward path, but the released controller still needs a separate frozen 0.6B text encoder and supervised MCTS-derived paths, and its pass@k metric for is oracle-set success rather than a deployable selection rule.
Flow Reasoning Models adds a perturbation-based inference-dynamics probe. A completed discrete-flow solution is re-noised and re-solved; return-to-candidate cross-entropy ranks correct Sudoku/Zebra states much better than the sampler generates them. The important caution is that the inner refinement loop can still converge to a confident wrong fixed point, so convergence, perturbation stability, and correctness are separate variables.
The Flexibility Trap adds generation order and local entropy to the inference-dynamics frame. Confidence-driven arbitrary-order sampling can fill easy positions first and return to logical forks only after future context has collapsed their uncertainty. Its JustGRPO result also shows that training-time rollout order and inference-time execution order can be deliberately different.
LaViDa adds a denoising-step and cacheability axis: Prefix-DLM caches the fixed image/prompt prefix, while timestep shifting trades refinement steps against quality. This is serving evidence for a static multimodal prefix, not an online-updated latent history state.
The LaViDa extensions add three distinct inference axes. LaViDa-O inserts explicit planning and reflection before final image generation; Sparse-LaViDa makes attention depend on the denoising step so later refinement can reuse cached prompt and revealed-token representations; and LaViDa-R1 branches tree search from promising partial diffusion states. These results support revisable intermediate multimodal state, but do not establish calibrated uncertainty or causal state estimation.
ICLR: In-Context Learning of Representations adds a context-length axis. During one long random-walk event stream, a frozen LLM’s late-layer token activations reorganize toward the hidden graph while next-token probability shifts toward valid neighbors. Dirichlet energy falls before the rapid accuracy ascent, and a conflicting pretrained semantic ring can remain in leading principal components while the context-defined ordering appears in later components. The evidence is genuine inference-time state change, but later induction-circuit scrutiny warns that the visible geometry may be produced by previous-token mixing while task performance relies on successor retrieval rather than a reusable world model.
Reading Frame
For this wiki, inference-dynamics claims should name:
- the state being tracked: residual stream, recurrent state, memory token, depth key/value state, decoder logits, prediction entropy, or task-specific probe state;
- the probe or intervention: tuned decoder, logit lens, probing classifier, layer skip, layer swap, layer repeat, early exit, halting statistic, recurrent loop count, or perturb-and-resolve stability test;
- the data contract: static tabular context, time-series history, event stream, action-conditioned trajectory, text prompt, or another modality;
- the budget contract: unique parameters, effective depth, expected FLOPs, wall-clock latency, memory bandwidth, and whether early exits are actually served.
- the update/commitment order: which positions are resolved first, which remain revisable, how local entropy changes, and whether the order used during training differs from deployment inference.
- the inference axis: token position/context length, layer depth, recurrent step, or denoising/refinement step; these axes SHOULD NOT be collapsed into one generic notion of progress.
Time-Series Boundary
Static tabular inference dynamics are adjacent to time-series modeling, not direct evidence for it. A multivariate time-series model must preserve temporal order, channel identity, rare regimes, exogenous variables, events, and possibly actions or control inputs. A useful time-series inference-dynamics study would probe whether those variables survive across depth, recurrent updates, early exits, and decoder alignment.
PoLar adds a program-routing boundary. Porting skip/keep/repeat control to a time-series foundation model would require a router that sees numeric features, channel identity, temporal order, missingness, event context, and any actions or control inputs. The language-only results do not establish that these variables remain recoverable after skipped segments or repeated state updates, so a transfer study must pair compute measurements with layerwise state probes and downstream calibration.
FRM sharpens the time-series boundary: a stable internal update is not automatically a correct state estimate. Perturbation stability could become a useful diagnostic, but it must be calibrated on stochastic and multi-modal futures so that rare valid transitions are not rejected merely because the model’s learned basin is narrow.
The Flexibility Trap adds a leakage boundary. A future-block generator may use bidirectional attention within the candidate block, but a time-series evaluation must not expose held-out target observations. The relevant analogue is whether the model preserves multiple valid future trajectories or action consequences under different internal commitment schedules.
Sparse-LaViDa and LaViDa-R1 sharpen the systems boundary: a useful state must be both revisable and cheap to continue from. A time-series analogue would cache or branch from a latent history state without revealing held-out observations, then measure whether the resulting candidate futures remain diverse, calibrated, and action-consistent under matched wall-clock and memory budgets.
Open Questions
- Can per-layer decoder alignment become a reliable early-exit interface, or does it leak assumptions from each model’s synthetic prior?
- Can a lightweight router choose skip/keep/repeat programs for a time-series foundation model under a hard expected-FLOPs budget without erasing rare events, interventions, or exogenous-variable effects?
- When looped depth improves a static tabular model, does the gain survive matched latency, expected FLOPs, memory bandwidth, and larger-context scaling?
- Which time-series variables survive across depth or recurrent inference updates: temporal order, channel identity, rare regimes, events, exogenous variables, and control input history?
- Can perturb-and-resolve stability distinguish robust state estimates from stable shortcuts, averaged futures, and overconfident wrong attractors?
- Which commitment schedule best preserves rare but valid time-series futures: strict temporal order, confidence order, entropy-first forks, or parallel revisable blocks under a matched serving budget?
- Which fixed-prefix caching gains survive when an observation history must update online while preserving rare events, exogenous variables, and action/control-input history?
- Can a model learn when to invoke planning, reflection, or direct generation from expected quality gain, uncertainty, latency, memory, and energy rather than using one fixed inference policy?