Time-Series Benchmark Hygiene

Summary

Time-series foundation model rankings are brittle unless task, protocol, context length, horizon, covariates, leakage controls, and adaptation mode match. Future source pages should link here instead of repeating the same benchmark caveats.

The latent-state wiki frame adds one more hygiene rule: average forecast error is not enough to establish system understanding. Benchmarks should say whether they test observation forecasting, representation quality, context use, rare-event sensitivity, state maintenance, or action-conditioned rollout.

Foundation Time-Series Model Research Agenda is the central rubric for this page: benchmark claims should identify which agenda slot they test and whether they close, partially close, or merely touch the path toward a foundation time-series model.

Required Separations

• Action-selection reports should separate offline or simulator-derived warm-up pseudo-observations, acquisition-function candidate scores, expensive simulator reranks, actually executed actions, and action-space restrictions. Causal-PIK is the local accounting case: each run starts from nine uncounted simulator warm-ups; each counted attempt scores 500 Sobol candidates with GP-UCB, probabilistically simulates five, and executes one; some comparison methods search drastically reduced action spaces. AUCCESS over executed attempts alone does not expose all of those decision-time costs.
• Sparse masked-decoding reports should separate adaptation cost, output length, block size, prompt/revealed-token caching, truncation, hardware, batch size, quality, and end-to-end latency. Sparse-LaViDa is the upstream multimodal case: gains are large on long outputs and negligible on short QA or grounding, while the paper itself disagrees between H100 and A100 for the post-training hardware.
• Reward-trained generative reasoning should separate final-answer correctness, trace quality, judge/reward score, independent task utility, forcing rate, and reward-server cost. LaViDa-R1 is the multimodal warning: injected answers and learned judges can create high reward without faithful reasoning, and full answer forcing collapses training.
• Zero-shot base-model results should be separated from few-shot adaptation, linear probing, full fine-tuning, and dataset-specific training.
• Fine-tuned and ensemble entries should be separated from base released checkpoints. Toto 2.0, for example, reports base models, a fine-tuned 2.5B variant, and a family-and-friends ensemble.
• Frozen feature extraction for classification should be separated from label-free zero-shot prediction. Many classification “zero-shot” results still train a Random Forest, SVM, logistic-regression head, or similar downstream classifier on target labels.
• Label-scarce classification and SSL reports should separate supervised-only hypersweeps, validation-set size, model-selection budget, and unlabeled-objective gains. S4L is the historical vision analogue: before crediting auxiliary unlabeled-data objectives, compare against strong supervised-only baselines and validation-size sensitivity.
• Single-model results should be separated from representation fusion, such as MantisV2 plus TiViT features.
• Scaling reports should separate the fully tuned frontier from fixed-recipe or best-of-small-$n$ runs, state the search budget per scale, reserve larger scales for validation/test, and distinguish near-data fit from far extrapolation. Small-Scale Experiments is upstream language-model evidence that a real law can be absent at 4 or 16 configurations per scale and emerge only after much deeper search; its 256-configuration result is setup-specific, not a TSFM constant.
• Point forecasting, quantile forecasting, probabilistic forecasting, imputation, anomaly detection, classification, and reasoning benchmarks should not be ranked as if they measure one ability.
• Long-horizon forecasting reports should separate pointwise error, horizon-wise marginal realism, conditional calibration, trajectory realism, coverage, and sharpness. Expectations vs. Realities is the direct numeric time-series case: whenever conditional uncertainty is nonzero, the MSE-optimal deterministic conditional mean is necessarily under-dispersed in the marginal target distribution. Its 5% MSE tolerance-band protocol exposes near-tied models with a median 17.3% marginal-$W_1$ gain across nine benchmarks, but marginal agreement can still pair the wrong future with a context or ignore temporal order. Reports should therefore use MSE–realism Pareto fronts as an inexpensive first diagnostic, then add proper probabilistic scores, per-context calibration, tails, multivariate constraints, and trajectory checks.
• Forecast-based anomaly-detection reports should separate the forecasting backbone, context/covariate construction, quantile or interval calibration, and downstream detector heuristics. Impact-Driven Event Embeddings is the local case: learned covariates improve zero-shot Chronos-2, but the final F1 also depends strongly on materiality and duration rules, and the best public precision is 0.3513. Report raw interval-breach, materiality-filtered, duration-filtered, and fixed-review-budget operating points separately.
• Mixed-type dependence reports should pin timestamp alignment, numeric precision, repeated-value rate, event cardinality, autocorrelation-aware effective sample size, lag-search multiplicity, estimator bias/variance, and whether relevance is marginal or conditional. HUMI is the local case: the unique-versus-repeated partition changes with precision and sample size, the $[0,1]$ normalization is heuristic, negative values are clipped, and the official event-clustering code differs from the paper’s distance and threshold description.
• State, intervention, counterfactual, and decision questions should be reported by capability level rather than collapsed into one reasoning score. FactoryBench is the local case: L3 uses approximate MMD-matched physical reruns rather than exact shared-state counterfactuals, while L4 uses a three-judge LLM ensemble rather than deterministic scoring. Reports should pin physical episode, fault/intervention pair, template family, judge model/version, rubric, agreement, and cost.
• Horizon-conditioned event-prediction reports should separate per-horizon discrimination, aggregate h-AUROC, calibration, false-alarm burden, label budget, and stopping rule. HEPA is the local case: the paper and released C-MAPSS reproduction use different fine-tuning stopping rules.
• Cross-embodiment industrial transfer reports should separate raw absolute error from mean-centered dynamic-shape error and pin the exact public corpus inventory. FactoryNet is the local case: the paper reports 51M datapoints across six embodiments, while the visible Hugging Face snapshot does not transparently reconcile with the full corpus.
• Wearable setup-transfer reports should separate unseen subject, body location, within-location mounting position/orientation, device hardware, sampling protocol, and label-space shift. AnyMo mixes several of these across its 14-dataset zero-shot evaluation, while AnyMo Bench isolates a same-position co-located-device change. AnyMo Bench reports 154,695 headline windows but publishes 123,874 train plus 30,965 test rows, so reproductions should use the pinned exported-row counts and preserve the discrepancy.
• Hierarchical ontology-classification reports should pin the ontology version, class closure/masking rule, chunk length, split unit, and artifact snapshot. BTS is the local case: the paper reports 240 classes, the older data card reports 215, and Brick by Brick uses 94 modified point subclasses; the paper and competition also use different partitions and evaluation surfaces. Cross-building zero-shot forecasting should additionally separate raw irregular streams from resampled grids and native multivariate models from channel-wise fallbacks.
• Multi-future generators should separate sample validity, mode coverage, probability calibration, oracle best-of-$N$ coverage, and deployable selector quality. FlowWM is the local case: Bouncing Shapes reports nearest-mode precision and recall errors, while FuturePerception chooses the best realized sample for AP${}_{\mathrm{L}}$(N); neither score establishes calibrated mode mass or selection before the future is observed.
• Manifold-supported generators should separately report target-support or trajectory-manifold hit rate, regional or mode probability mass, invalid-between-mode mass, one/few-pass quality, and end-to-end latency. Beckmann Transport Models is the local warning: an autonomous flow can reach the right support while allocating badly wrong basin mass.
• Manifold-geometry probes should report the exact theorem regime, noise direction, perturbation count, empirical-rank ceiling, Jacobian normalization, local-versus-global scope, and known-geometry recovery before presenting one dimension or topology number. DiffusionGeometryProbe is the local warning: its normal-bundle covariance has rank at most the number of perturbations minus one, local PCA is applied to isotropic perturbations of one point, and the strongest image-dimension conclusion uses a high-noise value despite citing small-noise limits. Estimator disagreement is diagnostic evidence, not automatic corroboration.
• Normalization comparisons should pin the training scaler, per-window transform, inverse transform, channel aggregation, zero- or near-zero-scale handling, and whether metrics are computed in normalized or raw units. Noise or Signal? is the local warning: its DLinear/RevIN Electricity MSE failure is numerically real, but repeated result tables drift, only three of eleven stated datasets receive detailed forecasts, no seed variance is reported, and $k_{\mathrm{emp}}=\mathrm{std}/\mathrm{MAD}$ can explode because MAD is near zero rather than because the numerator proves an outlier. Reports should add corrupt-window, genuine-event, per-channel, and tail-error slices instead of treating average MSE or average rank as a complete robustness verdict.
• Candidate-width objectives should separate the number of explored training candidates, processed-sample count, unique data, total forward/backward FLOPs, activation memory, wall-clock utilization, inference NFE, final sample quality, and mode-mass calibration. They should also name historical baselines correctly: IMLE introduced data-to-nearest-sample training and conditional IMLE already used the direct per-condition hard objective. Explorative Modeling is the modern scaling case: Forward XM can improve proposal coverage and lower inference NFE while increasing training work, and its hard minimum does not determine sample frequencies.
• Recoverability-boundary audits should separate simulator-verified branches from learned-surrogate proposals, keep sibling branches from one replay snapshot in the same split, and report intervention family, capability limits, delay, horizon, constraint contract, oracle-call cost, late-alarm rate, and monitor calibration. Counterfactual PoNR Auditing is the local case: distilled PoNR monitors are useful only if episode/snapshot/capacity leakage is controlled and the label scope is pinned.
• Observation forecasting, latent-state prediction, representation learning, and action-conditioned world modeling should not be collapsed into one claim.
• Generalization reports should separate hypothesis-space capacity, selected-solution complexity, and observed held-out performance. Deep Learning is Not So Mysterious or Different is the upstream warning: fitting randomized labels or reaching zero training loss shows flexibility, not that the selected structured-data solution is complex or will generalize poorly. Time-series randomization controls should preserve or deliberately break autocorrelation, channel structure, event timing, and exogenous-variable alignment so the tested complexity measure has a clear meaning.
• Memorization and privacy reports should separate unintended memorization, extraction, membership inference, attribute inference, and formal differential-privacy guarantees. How much can language models memorize? is the upstream language-model case: its standard loss-based average membership attack approaches chance as data exceeds measured capacity, while rare-token sequences remain disproportionately memorized. Reports should pin the reference model, compression estimator, data-to-capacity ratio, attack family, threshold selection, average-versus-worst-case scope, duplicate controls, rare-sample slices, and tokenization or time-series information unit.
• Latent-state accessibility diagnostics should separate coarse component or class presence from dense state variables. Aionoscope is the local case: categorical component AUROC can look strong while timing, phase, amplitude, frequency, or regime variables remain weakly recoverable under dense probes.
• Fixed-recipe stress tests should not be read as fully tuned leaderboards. LeNEPA intentionally measures recipe reuse after changing pretraining signal family, not the best possible Diag-tuned JEPA recipe.
• One-step prediction, decision-usable multi-step rollout, closed-loop transfer, and evidence-driven model revision should be separated. Agentic World Modeling names these as L1 Predictor, L2 Simulator, and L3 Evolver boundaries.
• Time-series-agent reports should separate perception, reasoning, planning/action, memory/knowledge, temporal world modeling, and reliability layers. Awesome Agentic Time Series is useful as a taxonomy for this separation, but its listed papers still need primary-source checks before their scores or claims are compared.
• Interactive agent benchmarks should separate query planning, evidence integration, hypothesis construction, query efficiency, and non-informative action rate. Agentic Automata Learning is the local symbolic benchmark warning for this split: final task success alone can hide whether an agent gathered insufficient evidence or failed to infer a stable latent model from sufficient evidence.
• Learned-simulator scores should be separated from real-environment or live-stand transfer. World Models is the historical hygiene case: a controller can maximize reward inside an imperfect learned dynamics model while failing after transfer.
• Action-conditioned world-model reports should separate prediction error, planning success, solver budget, per-step latency, distribution-shift regime, and factor-of-variation settings. stable-worldmodel is the current local case: its Push-T analyses show that prediction error can overlap heavily between successful and failed plans under distribution shift.
• Test-time adaptation inside control loops should separate the frozen pretrained model, adapted parameters, update objective, buffer policy, number of gradient steps, reset or persistence policy, added latency, and whether the reported gain comes from prediction-error reduction or better action ranking. AdaJEPA is the local case for this split.
• Power-grid control, surrogate, and security-analysis reports should separate challenge survival/cost score, single-agent versus multi-agent protocol, FACTS setpoint control versus topology reconfiguration, topology-only versus topology-plus-redispatch/storage action spaces, topology-action preprocessing, disconnected-line bus-assignment handling, topology reversion policy, candidate-action feasibility filters, stochastic seed policy, evaluation horizon or chronics length, contingency stochasticity, grid size, runtime-shield lookahead horizon, fallback action, action-space restriction, false-veto/false-accept rates, preventive N-1 robustness before a contingency, corrective behavior after a contingency, learned-screening latency, residual risk at fixed simulator budget, simulator-call count, wall-clock latency, simulator cost to reach a risk target, simulator access, load-flow solver emulation, DC-approximation baselines, topology-size generalization, physical-line assumptions, sparse/dense implementation speed, explicit-topology synthetic tests versus observational records with hidden or surrogate topology, simulator speedup versus deployment-grade accuracy, downstream policy/control utility, and online planning budget. Grid2Op is the local case: RL2Grid, MARL2Grid-TR, L2RPN challenge papers, soft-label action ranking, Gibbs-prior risk surrogates, runtime shields, LLM-guided replay-buffer shaping, and policy-distillation controllers are related but not interchangeable evidence axes.
• Graph time-series and graph-control reports should separate modeling quality from sparse-kernel/backend quality. IO-Aware GNN Layers is the local case: direct GNN baselines should report whether they use fused graph attention, cached cuSPARSE, degree-aware reductions, graph reordering, or default DGL/PyG composition, plus peak memory and wall-clock latency.
• Embodied world-model evaluations should separate open-loop action-conditioned prediction, closed-loop task utility or policy evaluation, and physical consistency, controllability, and executability diagnostics. World Model for Robot Learning Survey is the local robotics survey map for that split.
• Training-in-imagination reports should separate dynamics-transition error, reward-model error, reward annotation cost, reward noise, and reward bias. On Training in Imagination is the current local source for why zero-mean reward noise and systematic reward bias have different policy-gradient consequences.
• Clinical imagined-rollout reports should additionally separate OPE estimators, behavior-policy propensity quality, effective sample size and policy overlap, learned reward quality, outcome calibration, and prospective validation. medDreamer is the local case: positive WIS/WPDIS/CWPDIS and estimated-mortality signals do not identify causal treatment effects, and the main sepsis mortality interval and reduction arithmetic are internally inconsistent.
• Upstream dynamic-compute results, such as EBT language/video/image scaling, should be separated from numeric time-series forecasting, generation, and control evidence unless the benchmark directly tests those interfaces.
• Test-time-memory results should separate memory-only modules, full architectures, chunk size, update cost, baseline strength, and adaptation mode. Titans Revisited is the current local check for this failure mode.
• Segment-level recurrent-memory results should separate synthetic recall/rewrite tasks, long-context QA, language-modeling transfer, segment size, sequential update cost, and direct numeric or action-conditioned evidence.
• Sleep-time consolidation results should separate memory capacity, consolidation compute, wake-time prediction latency, training sequentiality, and whether the evidence is synthetic/language or numeric/action-conditioned.
• Looped-efficient-attention reports should separate zero-shot quality, long-context retrieval, prefill throughput, decode throughput, KV-cache footprint, recurrent-state footprint, batch-size OOM frontier, and kernel availability. LT2 is the local case: linear/sparse looped mixers improve long-context language serving, but that is not direct evidence for numeric time-series state retention or action-conditioned rollout.
• Sparse-attention reports should separate selector training, selected-token budget, block size, retrieval/selection recall, dense-baseline quality, prefill throughput, decode throughput, hardware generation, public kernel availability, and full serving-stack behavior. MiniMax Sparse Attention is the local case: vendor-reported 1M-context speedups and benchmark parity are important, but they should not be treated as numeric time-series state-retention or action-conditioned rollout evidence.
• Architecture papers with follow-up benchmark narratives should separate paper results, official but unreleased claims, and public-code reproducibility. Dragon Hatchling’s Sudoku result is official Pathway narrative, not currently open-reproduced from the public BDH repository.
• Recursive puzzle papers should separate controlled recursive-model baselines from frontier LLM reference scores. GRAM reports large-reasoning-model Sudoku/ARC numbers as benchmark-difficulty context, not as matched training-data, prompting, tool-use, or inference-budget comparisons.
• Generation benchmarks should separate pointwise fidelity metrics, retrieval or rank metrics, text-condition ablations, and downstream utility. T2S’s WAPE, MSE, and MRR@10 results show text-conditioned fidelity, not forecasting utility or world-model readiness.
• TimeCraft-style generation benchmarks should separate distributional fidelity, text controllability, target-aware downstream utility, causal/interventional validity, irregular-continuous reconstruction, and online pretraining gain. TimeCraft, TimeDP, BRIDGE, TarDiff, CaTSG, OATS, and Diff-MN are not comparable on one scalar leaderboard.
• For BRIDGE-style text-controlled generation, separate generated-description quality, no-text/no-prototype ablations, text-alignment scores, human ratings, and downstream forecasting augmentation. None of these alone proves operational context use or causal validity.
• For irregular-continuous generation, separate arbitrary-time reconstruction or interpolation quality from real irregular sampling, missingness-not-at-random, streaming updates, and downstream decision utility. Diff-MN is the local case because much of its irregularity is simulated by random observation dropping.
• Diffusion forecasting reports such as MG-TSD should separate probabilistic forecast metrics such as CRPS, NMAE, and NRMSE from synthetic-generation fidelity, text-control, downstream-utility, or world-model evidence.
• Bias-mitigation reports should separate bias or shortcut metrics, fidelity/quality metrics, prompt or condition alignment, and OOD forecasting metrics. InvDiff is the local case for this split.
• Financial-market generation benchmarks such as DiGA and MarS should separate control-target error, stylized-fact fidelity, market-impact or what-if validity, downstream trading-agent utility, latency, data-access/reproducibility, and simulator-assumption checks.
• Healthcare TSFM reports should separate inherently irregular clinical datasets from regular datasets with simulated missingness, out-of-distribution tests from pretraining-source-family tests, and passive forecasting from treatment or intervention modeling. MIRA is the local irregular-clinical case; SensorFM is the wearable-health case, where benchmark hygiene should additionally separate private versus public data, one-minute aggregate features versus raw waveforms, frozen linear probes versus agent-searched downstream heads, and Personal Health Agent grounding from diagnosis or intervention planning.
• Retrieval-augmented forecasting should report knowledge-base provenance, overlap, and retrieval candidates separately from base-model zero-shot scores. TimeRAF is the local case.
• Streaming-generation benchmarks should separate aggregate quality from temporal artifact diagnostics such as repetition, false silence or background activity, drift over time, and quantization degradation. Moshi is the current local audio case.
• Generative model reports should separate sample quality from memorization risk. Why Diffusion Models Don’t Memorize shows that a diffusion model can reach good samples before late training-set memorization, so generated time-series benchmarks should report checkpoint/update count, dataset size, and memorization or duplicate probes alongside fidelity metrics.
• Generative-model reproducibility reports should separate average sample quality from same-noise cross-run consistency. Diffusion Consistency RMT shows that deterministic diffusion samplers can align across disjoint data splits because they share stable first and second moments, while still overshrinking lower-variance directions. A time-series protocol should pin the independent split unit, effective sample size, architecture, sampler stochasticity, matched-noise seeds, checkpoint age, moment/regime balance, and frequency/channel/latent-state slices; high consistency alone can mean shared structure, shared bias, or mode erasure.
• Target-optimized synthetic or curriculum data should separate surface sample quality from downstream training effect. Synthetic Data for any Differentiable Target shows in language-model SFT that benign-looking examples can be optimized to induce hidden weight or metric changes, so time-series curriculum and synthetic-data benchmarks should audit representation drift, rare-state retention, normal-behavior retention, and poisoning-like side effects. Target-aware generators such as TarDiff also need guidance-set isolation, downstream-model diversity checks, and metric-overfitting audits. OATS-style online TSFM augmentation additionally needs reference-set provenance and evaluation-dataset isolation audits, because its guidance can depend on small reference samples from evaluation-like distributions.
• Query-conditioned data-attribution reports should separate the query distribution, upstream source corpus, sampled candidate pool, fraction inspected for scoring, selected unique examples, repeated sample exposures, optimizer updates, fine-tuning compute, attribution compute, target metrics, untargeted retention metrics, and query/test taxonomy overlap. Motion Attribution for Video Generation is the local case: “10% beats 100%” means fine-tuning a pretrained model on 1,000 selected clips versus all 10,000 clips in the sampled candidate pool on targeted dynamic degree. Attribution still processes all 10,000 candidates and costs about 150 A100-hours; the paper does not establish 10% candidate-data access, matched total compute, or a 10-times end-to-end saving, and not every visual-quality metric improves.
• Reference-guided reasoning or RL benchmarks should report correct-reference, wrong-reference, no-reference, and judge-size controls before treating dense partial-progress scores as reliable training signals. ExpRL is the current language-reasoning case: problem-matched references help capable judges verify partial progress, while wrong references and too-weak judges degrade the reward signal.
• Data filtering and curriculum reports should include no-filter, loose-filter, and dedup-only controls when the source corpus has natural diversity. A Bitter Lesson for Data Filtering shows that filtering wins can reverse with more compute, and No Filter shows that benchmark-friendly filters can hurt underrepresented distribution slices.
• Scaling-law reports should separate clean pretraining fits from perturbation-aware fits, including noise/SNR model, low-bit serving, post-training pressure, fit grid, and extrapolation distance. LLMs as Noisy Channels is the current upstream language-model case.
• Compute-optimal or fixed-FLOPs reports should state the cost model, parameter-count convention, data or effective-data unit, fit region, loss precision, fitting objective, and sensitivity to optimizer, schedule, tokenizer or patcher, data mix, and repeated data. Scaling Laws, Carefully is the current upstream method-hygiene source.
• Training-dynamics reports should separate aggregate loss curves from capability-emergence probes, including absolute thresholds, checkpoint density, probe construction, representation-extraction layer/head, and whether relative thresholds change cross-model conclusions. Implicit Curriculum Hypothesis is the current upstream language-model case.
• Representation-similarity reports should separate raw metrics, calibrated metrics, layer-search aggregates, and downstream utility. Aristotelian Representation Hypothesis is the current calibration case: raw global CKA-style convergence and max-over-layer trends can be width/depth artifacts, and temporal data need dependence-preserving nulls rather than naive row shuffles.
• Representation-geometry benchmarks should separate visible low-dimensional structure, calibrated nulls, local retrieval or induction shortcuts, causal edits, and downstream rollout utility. ICLR: In-Context Learning of Representations is the local graph-tracing case: context induces graph-like activations, but successor retrieval and previous-token mixing can explain task behavior and visible class-mean geometry without establishing a reusable transition model.
• Hybrid-architecture reports should separate state-conditioned targets from exact-copy, repeated-normal, and structural-closure targets before crediting recurrence, attention, or a specific mixer for aggregate gains. Comparing Transformers and Hybrid Models at the Token Level is the current upstream language-model case: filtered token losses expose a state-like versus copy-like split that aggregate loss hides.
• Long-context hybrid-memory reports should additionally separate router or retention policy, explicit-cache budget, maintained recurrent-state size, dual-state update cost, selected-span recall, cache-read normalization, and serving latency. Gated DeltaNet is the scalar-gated recurrent baseline behind the newer hybrid-memory comparisons; Oryx, Hybrid Associative Memories, and HOLA are current upstream language-model cases. HOLA also requires separating parameter overhead from cache-state bytes and distinguishing a 2k-trained RoPE full-attention checkpoint from a length-extrapolating baseline. Their results should not be treated as time-series state preservation until rare-regime, event-timing, exogenous-variable, and action-history probes are reported under matched budgets.
• Training-system claims should separate memory footprint, wall-clock throughput, total training compute, from-scratch versus pretrained-conversion settings, and objective-specific proxy metrics. DiffusionBlocks is the current local case for this separation.
• Upstream language diffusion metrics such as generative perplexity, entropy, BLEU, and ROUGE should stay separate from numeric time-series fidelity, calibration, generation utility, and control evidence. ELF is the current local case.
• Masked diffusion language-model benchmarks should separate likelihood or likelihood-bound claims from task-specific scoring surrogates. iLLaDA reports confidence-based multiple-choice scoring that performs better empirically than a likelihood-style baseline, so those scores should not be treated as direct likelihood evidence or as numeric time-series calibration evidence.
• KV-cache or retrieval-memory compression results should separate byte savings, quality recovery, hardware-native compute support, dequantization cost, latency, throughput, memory-pressure regime, and target-task quality. TurboQuant is the current local case for this separation.
• Sharpness, flatness, and edge-of-stability claims should state optimizer, loss, batch regime, and whether the reported quantity is full-batch Hessian sharpness or batch sharpness.
• Incident-response QA charts such as ARFBench should not be treated as forecasting leaderboards. They may mix frontier VLMs, LLMs, post-trained or hybrid time-series/VLM systems, domain experts, and oracle combinations.

Benchmark Families

Forecasting benchmark names in this wiki include BOOM, Context is Key, GIFT-Eval, TIME, Chronos-ZS, fev-bench, Monash, LSF/LTSF, Time-Series-Library, Informer-style ETT tasks, Darts, Chronos Benchmark II, and Time-HD. Multi-task observability datasets such as TelecomTS and incident-response QA benchmarks such as ARFBench should be compared on their own task axes rather than forced into a forecasting-only rank. These differ in horizon, frequency, metric, target/covariate interface, text/context interface, channel count, channel-dependency structure, and leakage policy.

Dataset Anchors

Dataset	Use It For	Main Caveat
Context is Key	Text-conditioned probabilistic forecasting where context is essential.	Univariate and text-only; benchmark-first with no training split; numeric-only forecasters see an intentionally incomplete interface.
TelecomTS	5G observability anomaly detection, root-cause analysis, forecasting, and time-series/text Q&A.	Lab/testbed data with 18 channels; synthetic anomalies and generated tickets need artifact checks; no operator action channel.
Time-HD	HDTSF at the thousand-channel scale.	Passive forecasting only; sparse public dataset card.
BOOM	Observability forecasting at the grouped-query, high-cardinality scale.	No operator actions/interventions; Datadog pre-production source.
FactoryNet	Multi-embodiment industrial control-loop telemetry and cross-embodiment dynamics transfer.	Paper scale, visible public inventory, subset licenses, and raw-versus-mean-centered metrics need separate reporting.
FactoryBench	Industrial telemetry state, intervention, counterfactual, and decision reasoning.	L3 uses approximate matched reruns; L4 uses LLM judges; the current Hugging Face viewer fails schema generation.
NeoRL-2	Non-vision action-conditioned transition benchmark with rewards and simulators under delays, exogenous factors, safety constraints, conservative policies, and limited data.	Simulated tasks; 2025 arXiv preprint; exact Hugging Face configs and license terms need pinning.
CityLearn	Building-energy control and demand-response trajectories from a simulator/schema with continuous storage and device controls.	Not a single immutable payload; package version, schema, reward/cost function, action set, and source-data provenance must be pinned.
BTS	Real-building irregular telemetry, Brick channel/topology context, cross-building ontology classification, and cross-building forecasting.	Paper/data-card/competition class counts and size snapshots drift; paper and challenge protocols differ; forecasting baselines do not share one native multivariate interface.
Grid2Op	Graph-structured power-grid control episodes with topology/control inputs, contingencies, safety constraints, rewards, and simulator-backed next observations.	Challenge ecosystem, not fixed payload; pin Grid2Op version, backend, chronics, action masks/reductions, reward/cost, train/test scenarios, simulator access, planning budget, latency, single-agent versus multi-agent setup, and whether learned surrogates rank actions, predict risk, or model full transitions.
Tennessee Eastman Process Simulation Data	Industrial process anomaly detection and fault diagnosis with measured and manipulated variables.	Synthetic simulator data; manipulated variables need preprocessing into control inputs; no rewards, remediation actions, or counterfactual intervention protocol.
Aionoscope	Controlled latent-state accessibility diagnostics with exact categorical and dense labels.	Synthetic single-channel public validation snapshot; mean-pooled linear probes and native-length adapters limit ranking claims.
AnyMo Bench	Fine-grained wearable IMU recognition under unseen-subject and same-position cross-device shift.	Four configurations reuse one window inventory; mixed automated/human label pipeline; 154,695 headline windows conflict with 154,839 exported rows per configuration; no action channel.
GIFT-Eval	Broad general-purpose TSFM evaluation and leaderboard comparisons.	Dataset-count/version summaries differ; not observability-specific.
TIME	Strict zero-shot and contamination-resistant claims.	Not primarily HDTSF or observability.
Time-Series-Library	Legacy LSF/LTSF continuity, especially ETT/Electricity/Weather comparisons.	Low-dimensional and often saturated.

Classification and representation-learning benchmarks include UCR and UEA. They test labeled shape or sequence discrimination, not direct future-value forecasting.

Static tabular benchmarks such as TALENT or small-data TabPFN-style suites are adjacent but should be kept separate from time-series evaluations because rows are not temporal histories by default.

TabPFN-3 makes this separation especially important. Its technical report includes static TabArena results, API/enterprise TabPFN-3-Plus and Thinking entries, and a specialized TabPFN-TS-3 time-series checkpoint; those are not one benchmark protocol.

LimiX and LimiX-2M add a second static-tabular protocol warning. Reports SHOULD separate bare-checkpoint inference from attention-guided context retrieval, feature/label-order pipelines, schema transforms, fine-tuning, and multi-pipeline ensembling, and SHOULD name when a benchmark such as BCCO was introduced by the same team.

Leakage And Overlap Risks

Broad pretraining corpora can include public datasets, training splits, or near-duplicates that later appear in benchmark reports. Chronos-2 explicitly discusses GIFT-Eval overlap and reports a synthetic-only ablation for stricter zero-shot evidence. TiRex flags overlap risks for some baselines and reports an appendix update intended to remove full GiftEval overlap. FlowState makes data-overlap handling part of its GIFT-style zero-shot claim and separately evaluates unseen sampling rates. Toto 2.0 says public forecasting datasets were excluded from pretraining, making leakage control part of its claim. Sundial is another protocol-sensitive case: it reports excluded evaluation datasets for TimeBench, uses different sampling counts for FEV and GIFT-Eval, and separately reports zero-shot and once-fine-tuned FEV settings.

Synthetic data does not eliminate benchmark risk by itself. Template families can resemble benchmark dynamics, and synthetic generators can encode artifacts that make downstream ranks look stronger than real transfer would be.

For target-optimized synthetic or curriculum data, surface quality is not enough. Synthetic Data for any Differentiable Target is language-model evidence rather than time-series evidence, but it shows that generated examples can be optimized for hidden downstream training effects while looking benign. Time-series curriculum and synthetic-data benchmarks should therefore audit downstream representation drift, rare-state retention, normal-behavior retention, and poisoning-like side effects, not only sample fidelity or aggregate score.

For filtered pretraining corpora, “filtered” is not automatically cleaner evidence. A Bitter Lesson for Data Filtering shows that the best data choice can depend on compute scale; No Filter shows that familiar benchmark gains can hide cultural and socioeconomic coverage loss. Time-series reports should therefore name the filter target, compare no-filter or loose-filter controls, and include tail-slice probes before claiming a curriculum or filter improves representations.

For checkpoint-time capability reports, aggregate loss is not enough. Implicit Curriculum Hypothesis shows in language models that absolute-threshold probe emergence can be stable while relative-threshold definitions change the conclusion. Time-series reports should therefore define the capability threshold, probe suite, checkpoint spacing, and representation-extraction protocol before comparing training runs.

Cross-Paper Comparison Checklist

Before comparing two reported ranks, check:

• Is the task forecasting, classification, imputation, anomaly detection, reasoning, or generation?
• Is the model zero-shot, few-shot, linear-probed, fine-tuned, or ensembled?
• For tabular foundation models, is the score from one forward pass, retrieval-selected context, schema-transformed pipelines, or a multi-pipeline ensemble?
• Are context length, horizon, patch size, frequency, and rollout method comparable?
• Are known future exogenous variables, covariates, or grouped series available to both models?
• Is textual context essential to the correct forecast, merely descriptive metadata, or unavailable?
• Are metrics aligned, such as MASE, CRPS, weighted quantile loss, MSE, MAE, WAPE, MRR@10, accuracy, macro-F1, or rank?
• For generation tasks, is the report measuring text-conditioned fidelity, retrieval/ranking, sample diversity, or downstream task utility?
• For observability datasets, is the target forecasting, anomaly presence, anomaly interval, root cause, or language QA?
• Does a pretraining corpus, adaptation split, retrieval knowledge base, or synthetic generator include benchmark train/test data, private telemetry, near-neighbor windows, or unreleased generation artifacts?
• Is the model univariate, channel-independent, or native multivariate?
• Is the benchmark channel count high enough to test high-dimensional multivariate behavior rather than ordinary low-channel forecasting?

Evidence

The source pages already show why this page is needed. Toto 2.0 mixes base, fine-tuned, and ensemble leaderboard entries; the Toto 2.0 TSALM talk adds ARFBench, where VLMs, hybrid Toto/VLM systems, humans, and oracle combinations are compared on incident-response QA; TiRex, FlowState, and Chronos-2 discuss overlap or stricter zero-shot settings; Moirai 2.0 reports a smaller model outperforming larger variants under one aggregate; U-Cast argues that low-channel forecasting benchmarks under-test native multivariate dependency modeling; Context is Key changes the input contract by making natural-language context essential; TelecomTS changes the target from forecasting alone to observability diagnosis and time-series/text QA; Titans Revisited shows that test-time memory introduces separate protocol axes for chunking, memory-update cost, baseline strength, and whether the report measures a full architecture or an isolated memory component; TabPFN-v2, TabPFN-3, TabICL, LimiX, and LimiX-2M operate mainly on static tabular tasks, with the LimiX family adding BCCO provenance and retrieval/ensemble configuration as explicit comparison axes; TabPFN-TS-3, TempoPFN, MantisV2, and UniShape each change the task or adaptation mode again.

Impact-Driven Event Embeddings adds a compact synthetic-benchmark example. Its 21% WAPE and 78% F1 poster headlines compare the best lag-bucketed context with no-event Chronos; the stricter comparison with binary-event Chronos is 2.8% lower WAPE and 13.5% higher F1. Its unseen-account split shares the event environment, and its intrinsic retrieval metric uses same-name cross-year positives. Reports should therefore distinguish event awareness from learned representation quality, account transfer from event transfer, and embedding quality from detector post-processing.

Aionoscope adds a local diagnostic example: one benchmark can deliberately separate categorical component accessibility from dense process-state accessibility, and the paper’s own framing prevents treating the public validation snapshot as a stable leaderboard. LeNEPA adds the matching method-side protocol example: its PTB-XL/Diag comparison is a fixed-recipe stress test and should not be rewritten as a claim that no possible Diag-tuned JEPA recipe could match it.

S4L adds an older cross-domain baseline warning. Its semi-supervised ImageNet gains are only interpretable because the paper reports strong supervised-only baseline sweeps, validation-size sensitivity, and model-selection details. Time-series classification and SSL benchmarks should carry the same burden before treating unlabeled-data objectives as true representation gains.

World Models adds the learned-simulator version of the same problem. Virtual-environment reward, real-environment transfer, and exploitability/uncertainty settings must be reported separately because a controller can exploit an imperfect learned dynamics model.

stable-worldmodel adds the modern JEPA/world-model implementation version of the problem. Common solvers, trajectory storage, and factor-of-variation sweeps make robustness claims easier to audit, but they also show why one aggregate score is too small: in-distribution planning, OOD planning, prediction error, latency, and solver settings are distinct protocol axes.

AdaJEPA adds the adaptation-mode variant of the same problem. A report can improve closed-loop success by changing the model during evaluation, so the benchmark record must say whether the comparison is frozen, within-episode adapted, persistently fine-tuned, or trained with extra target-domain data.

World Model for Robot Learning Survey adds the embodied-robotics taxonomy version. A future-video score can be useful for world-model work, but it should not be treated as control evidence unless the protocol also tests action responsiveness, closed-loop decision utility, or executability.

On Training in Imagination adds the reward-economics version. If a benchmark or paper says imagined training works, the wiki should ask whether the dynamics model and reward model were learned from different data streams, whether the reward labels were cheap/noisy or expensive/clean, and whether any reward error was zero-mean noise or systematic bias.

medDreamer adds the clinical OPE version. Its policy is trained in a learned latent model and evaluated through importance-sampling estimators, estimated clinician propensities, a learned reward, and a return-to-mortality mapping. Reports in this regime should expose effective sample size and policy overlap, keep estimated mortality separate from observed deployed-policy outcomes, and audit internal interval and arithmetic consistency before describing clinical benefit.

Awesome Agentic Time Series adds the agentic-time-series benchmark map. It makes clear that forecasting benchmarks, reasoning/QA benchmarks, engineering-agent tasks, future-prediction tasks, and decision-centric benchmarks stress different parts of a system. The wiki should therefore avoid rolling them into one “agentic time-series” leaderboard unless the environment, action space, feedback, tool budget, memory policy, and evaluation target match.

The fresh Grid2Op sources are a concrete example of this hygiene rule. RL2Grid is a single-agent benchmark preprint whose ICLR 2025 OpenReview submission is withdrawn; MARL2Grid-TR is an accepted ICLR 2026 multi-agent benchmark; AI challenge for safe and low carbon power grid operation is a challenge-analysis paper; soft-label topology actions is an imitation-learning action-ranker; Gibbs-prior topology control is a preprint with a one-step overload-risk surrogate; runtime safety shielding is simulator-backed online filtering; LLM-guided safe RL is training-time transition refinement; interpretable policy distillation is controller compression/auditability evidence; GNN transmission-grid topology is representation/OOD-topology hygiene; and targeted exploration is line-switching/cascading-failure evidence with survival-time, action-diversity, and fixed-budget training metrics. Older power-system world-model/shield papers such as WMAP should stay separate from Grid2Op topology-control benchmarks because FACTS setpoint control, topology reconfiguration, and simulator-backed action ranking are different protocols. They should not be collapsed into one Grid2Op SOTA rank without matching environment, action space, simulator budget, graph representation, closed-loop protocol, and safety protocol.

Relation To Foundation TSFM Agenda

This page is the benchmark-hygiene slot for the Foundation Time-Series Model Research Agenda. It should be used to decide what a reported score actually tests: observation forecasting, state prediction, context sensitivity, rare-regime preservation, native multivariate scaling, generation fidelity, reasoning, or action-conditioned rollout. The agenda verdict for benchmark claims SHOULD stay slot-level; one aggregate rank cannot close the whole foundation-model path.

Open Questions

• What minimum overparametrization audit should a TSFM report: train fit, held-out loss, effective rank or dimensionality, compression protocol, structured-versus-randomized controls, and rare-state retention?
• Should the wiki maintain a normalized benchmark table only for results that share protocol and metrics?
• How should private training corpora or private observability benchmarks be weighted relative to fully reproducible public benchmarks?
• Should HDTSF benchmarks report channel-count, correlation, hierarchy, memory, and training-time axes next to forecasting error?
• Should context-aided benchmarks report context ablations, corrupted-context controls, and region-of-interest metrics by default?
• Should event-aware anomaly benchmarks require leave-one-event-family-out, missing-calendar, shifted-date, false-event, and fixed-review-budget tests in addition to unseen-account transfer?
• What is the right benchmark for action-conditioned time-series world models where interventions, not only forecasts, matter?
• What is the minimum reproducible protocol bundle for imagined-rollout training: source trajectories, action schema, reward-label provenance, reward-noise/bias audit, solver config, factor-of-variation settings, distribution-shift split, failed-action receipts, and latency budget?
• What minimum memorization-probe bundle should generated time-series benchmarks report: checkpoint/update count, dataset size, nearest-neighbor or subsequence matches, duplicate checks, membership inference, and downstream leakage?
• What minimum paired-reproducibility bundle should a generative TSFM report: independent trajectory split, matched base-noise paths, deterministic-versus-stochastic sampler, same-noise cross-run error, spectral/regime slices, calibration, diversity, memorization, and downstream utility?
• What minimum hidden-effect audit should target-optimized synthetic or curriculum data report: representation drift, rare-state retention, normal-behavior retention, metric overfitting, and poisoning-like side effects?
• What minimum no-filter audit should a data filter report: no-filter and loose-filter baselines, dedup-only baseline, compute scale, model size, tail-slice coverage, regime/tenant/device representation probes analogous to geo-localization, and corrupt-window rejection?
• What minimum reliability protocol should an agentic time-series benchmark require across perception, reasoning, tool calls, memory updates, actions, feedback, grounding, replay, auditability, safety, and cost before a system is called deployable?
• What minimum protocol should within-episode test-time adaptation report: update target, buffer, step count, reset policy, latency, adaptation-failure cases, safety shield, and frozen/retrained baselines?
• What minimum scarce-label protocol should classification and SSL benchmarks report: supervised-only hypersweeps, validation-set size, model-selection budget, downstream classifier or fine-tuning mode, and leakage controls?
• What minimum latent-state accessibility protocol should report categorical versus dense targets, layer-selection budget, probe capacity, public versus hidden streams, native-length confounds, and real-task transfer checks?

Related Pages

• Time-Series Foundation Models
• Foundation Time-Series Model Research Agenda
• Latent-State Time-Series Modeling
• Context-Aided Forecasting
• Time-Series Scaling And Efficiency
• High-Dimensional Time Series Forecasting
• Time-Series Classification Foundation Models
• Synthetic Data For Time Series
• Tabular Foundation Models
• Representational Similarity Calibration
• Heterogeneous Temporal Dependence