Distribution Priors In Self-Supervised Learning
Summary
SSL objectives often encode assumptions about the desired distribution of embeddings, clusters, prototypes, or latent states. These assumptions can prevent collapse, but they can also bias which factors are preserved when the pretraining data is imbalanced.
What The Wiki Currently Believes
- The Hidden Uniform Cluster Prior in Self-Supervised Learning shows that SimCLR, VICReg, SwAV, and MSN can impose a hidden uniform cluster or feature prior through volume maximization, whitening, entropy maximization, or prototype balancing.
- The same source shows that this prior can be useful on class-balanced ImageNet-like data and harmful on long-tailed data; power-law or class-matched priors help when they match the data distribution and hurt when they do not.
- A Cookbook of Self-Supervised Learning gives the practical context: batch statistics, projectors, predictors, augmentations, collapse prevention, and evaluation protocol are method-defining choices, not implementation trivia.
- When Does LeJEPA Learn a World Model? adds a positive case for a distribution prior: in Gaussian latent worlds with OU-style positive pairs, a Gaussian or whitening constraint helps make the learned state linearly identifiable. The same source is also a warning, because policy-shaped non-Gaussian trajectories degrade recovery.
- LeNEPA applies SIGReg as a temporal distribution prior for no-augmentation next-latent time-series SSL, showing a useful fixed-recipe case while leaving broader non-Gaussian and long-tail state preservation open.
- VISReg adds a regularizer-design pressure test: keep the Gaussian/sketching prior, but split scale and shape so the loss has stronger gradients under collapse and can scale through distributed random slices.
- LeVLJEPA adds the cross-modal version: independent SIGReg priors on image and text embeddings stabilize non-contrastive image/text prediction, but the objective still needs predictor/stop-gradient asymmetry.
- VJEPA adds a probabilistic JEPA case where the target latent family is regularized toward a prior, while BJEPA shows that modular priors can help or fail depending on misspecification and concentration.
- Learn From Your Own Latents And Not From Tokens adds another positive-but-scoped prior case: own-latent targets recover hidden hierarchy on a balanced and separated synthetic grammar, which makes the real-data prior mismatch question sharper rather than resolved.
- No Filter adds an empirical VLM warning: an English-only data prior improves familiar ImageNet/COCO metrics while damaging culturally and socioeconomically diverse representation quality.
- A Bitter Lesson for Data Filtering adds a compute-scale warning: a heuristic quality prior over text can help low-compute training but lose to unfiltered data when model size and training duration are large enough.
Evidence
The Hidden Uniform Cluster Prior paper gives both a theoretical and empirical argument: K-means-like assumptions appear inside several modern SSL losses, and changing the mini-batch class distribution changes semantic transfer for methods with explicit volume-maximization regularizers. The Cookbook explains why this belongs in the broader SSL checklist: collapse-prevention mechanisms, batch construction, and projector heads are part of the representation objective itself. LeJEPA Identifiability and LeNEPA add that distribution priors are not only risks: if the prior matches the latent process or the fixed-recipe stress test, it can make the recovered state more usable. VISReg adds a mechanism-level refinement: even if a Gaussian/sketching prior is desirable, scale control and shape control may need separate gradients. LeVLJEPA adds the cross-modal caution: marginal SIGReg can preserve rank, but direct symmetric regression can still collapse or underperform unless prediction asymmetry is present. The operational question is whether the prior and its decomposition match the domain, not whether priors can be avoided.
The two no-filter sources add the data-selection side of the same problem. No Filter shows that the English-only prior can look good under popular benchmarks and still erase underrepresented visual/cultural structure. Bitter Lesson shows that a heuristic quality prior can become compute-suboptimal when larger models can exploit weak signal in the raw pool. Both support treating filters and curricula as explicit distribution priors that need tail-slice audits.
Implications For Time-Series And World Models
Temporal data is usually long-tailed. Rare incidents, regime changes, interventions, treatments, control inputs, asset types, sensors, patients, users, and environments may be sparse but central. A uniform latent-cluster prior can make the model overrepresent common balanced-looking factors or suppress naturally imbalanced factors that matter downstream.
For time-series JEPA, contrastive, or non-contrastive SSL, agents should ask which prior the loss imposes over latent regimes. If the prior is accidental, benchmark results may depend on sampler design, batch composition, horizon selection, and whether the evaluation set hides or exposes tail regimes. If the prior is deliberate, as in Gaussian or whitening constraints, agents should ask whether exploration and windowing make the latent state approximately Gaussian and isotropic, or whether policy bias, bounded variables, periodic variables, and rare interventions make the prior distort the state.
Gotchas
- Uniformity is not the opposite of collapse; it is a specific anti-collapse prior.
- Balanced mini-batches are not automatically more faithful to the data distribution.
- Long-tailed priors are not universally better; they help only when they match the relevant semantic or regime structure.
- Downstream aggregate scores can hide rare-regime damage. Evaluation should report tail regimes, anomaly windows, intervention windows, and domain-specific slices when available.
- Data filters are distribution priors too. A filter that removes non-English, low-quality-looking, high-loss, or rare windows can improve a proxy benchmark while shrinking the support needed for a foundation model.
Relation To Foundation TSFM Agenda
This page is a warning for the Foundation Time-Series Model Research Agenda, especially the data-diversity and anti-collapse slots. It does not propose a time-series foundation model, but it names a failure mode that would make one brittle: an anti-collapse prior can preserve variance while still suppressing rare regimes, intervention effects, or long-tail system states.
Open Questions
- Can weak metadata, clustering, textual descriptions, or causal structure estimate useful SSL priors without labels?
- Which collapse-prevention mechanisms preserve long-tailed temporal regimes best?
- Should time-series foundation models expose sampler and batch-composition details as part of the model card?
- Can Gaussian or whitening priors be adapted to bounded, periodic, discrete, graph-structured, or long-tailed latent states without losing identifiability?
- Does temporal SIGReg behave as a useful stabilizer or a distortion source when LeNEPA-style targets include rare regimes, periodic variables, high-channel structure, and action histories?
- Does VISReg-style scale/shape decoupling make Gaussian sketching safer for long-tailed time series, or does a stronger variance gradient force unnatural state geometry faster?
- Does LeVLJEPA’s per-modality SIGReg recipe extend to time-series/text target families, or does heterogeneous target structure require separate priors for labels, captions, numeric values, and event intervals?
- When do own-latent targets recover real hierarchical structure, and when do balanced or separated synthetic assumptions hide long-tail, ambiguous, or context-dependent failures?