LeNEPA
Status: published MILETS 2026 result plus follow-up research directions for target families, dense-state preservation, and action-conditioned extensions.
Collaboration
If this direction resonates with you, I would be happy to talk with like-minded people, collaborate on research, and work on use-cases together.
Ideas are not the bottleneck. Hands are. Time-series modeling should be moving at least as fast as vision, audio, and robotics.
- Email: alexander.chemeris@gmail.com
- X: @chemeris
- Telegram: @alexanderchemeris
Summary
LeNEPA now has a source-backed MILETS 2026 result: LeNEPA: No-Augmentation Next-Latent Prediction for Time-Series Representation Learning implements a no-augmentation next-latent-token objective with a causal backbone, temporal SIGReg stabilization, and frozen-probe evaluation on PTB-XL, Aionoscope Diag, and a CauKer-to-UCR check. VISReg is the closest new regularizer-neighbor because it keeps the SIGReg/LeJEPA distribution-prior line but decouples scale and shape to fix SIGReg’s weak gradient under collapse.
The idea page should now track follow-up design questions rather than the existence of the method. LeNEPA still lives close to Next-Embedding Prediction, LeWorldModel, EIDOS, NextLat, and VISReg. NextLat remains especially relevant because it supervises the model’s own next hidden state; VISReg is especially relevant because it turns SIGReg’s monolithic Gaussian sketch into a scale/shape regularizer ablation. A follow-up LeNEPA program should compare external patch embeddings, contextual target embeddings, own-hidden targets, temporal SIGReg, and temporal VISReg-style variants under matched compute and preservation probes.
Published MILETS 2026 Result
The published LeNEPA paper partially answers the original idea by showing that temporal SIGReg can replace stop-gradient/EMA stabilization in a no-augmentation next-latent recipe under a fixed-recipe stress test. The main result is not a universal foundation encoder claim; it is evidence that the same SSL recipe can be reused across PTB-XL and Aionoscope Diag with less domain-specific augmentation/view engineering than an ECG-tuned JEPA masking recipe.
The remaining research program is sharper after the paper: test whether the recipe preserves dense numeric state, rare events, multivariate channel relationships, irregular timing, and eventually typed action/control-input histories.
Placement
| Neighbor | Shared idea | Difference from LeNEPA |
|---|---|---|
| NEPA | Predict future embeddings instead of raw pixels or tokens. | Original NEPA is vision-only, relies on stop-gradient, and does not use LeNEPA’s temporal SIGReg target distribution control. |
| LeJEPA | Use an isotropic Gaussian target distribution / SIGReg to avoid collapse and dimensional degeneration. | LeJEPA is the broader JEPA regularization claim; LeNEPA would specialize that prior to next-embedding or next-state autoregression. |
| LeWorldModel | Combine next-embedding prediction, Gaussian regularization, and latent world-model use. | LeWorldModel is pixel-control evidence with explicit actions; LeNEPA should be tested for time-series/event streams and typed control inputs. |
| EIDOS | Time-series next-embedding prediction with observation-space grounding. | EIDOS uses stop-gradient plus grounding; LeNEPA asks whether distribution regularization can reduce heuristics while preserving dense numeric state. |
| NextLat | Predict a future latent/hidden state and evaluate whether next-observation accuracy hides weak internal maps. | NextLat keeps next-token training and predicts the Transformer’s own next hidden state from the current hidden state plus next token; LeNEPA should compare own-hidden targets against NEPA-style embedding targets. |
| VISReg | SIGReg-family anti-collapse regularization with a stronger collapse-stage gradient via separate scale and SWD shape losses. | VISReg is vision SSL evidence, not time-series evidence; LeNEPA should test a temporal VISReg variant as a regularizer ablation against temporal SIGReg. |
Interface Sketch
flowchart LR X[time-series window / event stream] --> Tok[tokenizer or embedder] Tok --> H[current latent state] U[event, exogenous variable, action, control input, or intervention] --> Pred[predictor] H --> Pred Pred --> Zhat[predicted next embedding/state] Target[next embedding or hidden state target] -. stop-gradient or online target .-> Align[latent alignment] Zhat --> Align Zhat --> Reg[SIGReg / Gaussian distribution regularizer] Zhat --> Ground[optional observation grounding]
The core decision is target construction. A safe first implementation SHOULD compare at least three targets under matched compute:
- Patch- or point-wise embedding target: closest to NEPA and EIDOS; likely preserves local numeric state better.
- Contextual embedding target: may encode useful state, but can mix away patch-level detail.
- Own-hidden target: closest to NextLat; may create compact belief state, but needs probes to ensure dense numeric detail and rare events survive.
Hypotheses
- LeNEPA can make NEPA-style next-embedding prediction less dependent on stop-gradient and teacher/student heuristics by using a target distribution prior such as SIGReg.
- VISReg suggests that temporal SIGReg may need a scale/shape split when embeddings collapse or when slice count and projection dimension scale up; this should be tested rather than assumed.
- NextLat suggests that own-hidden-state prediction is a strong baseline for LeNEPA. A LeNEPA experiment SHOULD include a NextLat-style own-hidden target, not only external embeddings.
- Target-layer choice will dominate outcomes. The existing NEPA topic already warns that contextual or internal-layer targets can degrade quality; LeNEPA should treat target-layer ablation as a first-class result, not an appendix.
- Time-series LeNEPA needs observation grounding or another dense-value preservation check. A Gaussian latent that predicts well can still erase rare spikes, cross-channel deviations, event timing, or action history.
- For action-conditioned settings, the transition input SHOULD be a typed action, control input, intervention, event, treatment, or exogenous variable, not an ambiguous “action” token.
Follow-Up Experiment Shape
A practical follow-up experiment should now start from the published LeNEPA baseline and expand the target and data contracts:
- Add target-family ablations: external patch embeddings, contextual embeddings, own-hidden targets, and hybrid targets.
- Add regularizer-family ablations: temporal SIGReg, temporal VISReg-style scale/shape/center regularization, and a no-regularizer or stop-gradient control.
- Move from univariate or low-channel regular samples toward multivariate, irregular, event-stream, and useful-signal-poor domains.
- Evaluate not only forecast or classification loss, but also latent probes for regime, event timing, channel relationships, rare-state retention, and dense numeric recoverability.
- Add a small action- or intervention-conditioned environment only after the passive target-layer and regularizer comparisons are stable.
- Report wall-clock and memory cost, because distribution regularization and extra target heads must beat simple forecasting or deeper-backbone baselines under matched serving constraints.
Relation To Foundation TSFM Agenda
This is an idea page, so the verdicts below describe the intended contribution if the proposed system works. Evidence status is recorded separately in the Evidence and Missing pieces columns.
| Agenda slot | Verdict | Evidence | Missing pieces |
|---|---|---|---|
| Latent-state prediction | partially closes | The MILETS 2026 source demonstrates next-latent prediction for time-series representation learning under a fixed-recipe stress test. | Extend beyond passive regular-sampled settings and run matched target-family/own-hidden-state ablations. |
| Anti-collapse regularization | partially closes | The MILETS 2026 source uses temporal SIGReg as the LeNEPA stabilizer instead of stop-gradient/EMA; VISReg now adds a concrete SIGReg-family scale/shape ablation target. | Show that regularization prevents collapse without erasing rare regimes, dense numeric values, or action-relevant state. |
| Representation quality | partially closes | Frozen probes and intermediate-layer readouts show useful representations on PTB-XL, Aionoscope Diag, and a UCR check. | Need probes for regime, cross-channel state, exogenous variables, events, interventions, and dense recoverability. |
| Control and counterfactuals | adjacent | LeWorldModel and NextLat motivate latent transition interfaces, but LeNEPA still needs explicit typed action/control/intervention inputs. | Add candidate-action rollout or intervention benchmarks after passive state learning works. |
| Benchmark level | warning | NextLat shows next-token legality can hide poor internal maps; NEPA warns target-layer choice can dominate results. | Define benchmark diagnostics before claiming a learned world model. |
Open Questions
- Is LeNEPA best defined as SIGReg-regularized NEPA, NextLat-style own-hidden prediction with a Gaussian prior, or a target-family comparison that includes both?
- Which target path is safest for time series: point-wise embeddings, independent patch embeddings, contextual embeddings, internal Transformer layers, or own hidden states?
- Beyond the published PTB-XL/Diag fixed-recipe setting, when does temporal SIGReg replace stop-gradient without losing dense numeric detail, rare events, or action-relevant state?
- Does a temporal VISReg variant improve LeNEPA’s collapse recovery or scaling versus temporal SIGReg, or does the added variance/shape machinery over-regularize long-tailed numeric states?
- Does own-hidden-state prediction improve belief-state quality on multivariate time series, or does it mostly optimize a self-consistency shortcut?
- How should LeNEPA incorporate typed actions, control inputs, interventions, events, treatments, and exogenous variables?
- Which metrics distinguish a compact useful belief state from a compressed latent that merely improves average forecast loss?