A Few Gaussians Is All You Need: SSOG

Source

• Raw source synthesis: paper_ssog-2026.md
• Official blog: A Few Gaussians Is All You Need
• Original X status: https://x.com/i/status/2089002002185384395
• Canonical X thread: https://x.com/ml_4rtemi5/status/2089002002185384395
• Official code: 4rtemi5/ssog
• Audited commit: c6f00afb7affec06735d94aa00b7064b9df4c266
• Local blog snapshot: papers/ssog-2026/source_blog_article.md and papers/ssog-2026/source_blog.html
• Local X provenance: normalized thread, root API JSON, and conversation API JSON under papers/ssog-2026/
• Local code provenance: pinned source archive, repository metadata/README, and papers/ssog-2026/code_artifact_audit.md

Status And Credibility

This is a 2026-08-15 author technical blog with a 2026-08-16 X launch thread and public JAX/Flax code. It is not a paper, technical report, or peer-reviewed result. The artifacts are credible as a concrete mechanism proposal because the blog, thread, and code agree on the main architecture and the public implementation supports same-resolution forward execution. The headline evidence remains author-reported: the audited repository has no ImageNet pipeline, checkpoints, raw logs, complete seed-level records, optimized kernel benchmark, or license. The blog reports three-seed CIFAR runs and standard deviations for selected ImageNet SSOG variants; the ImageNet SDPA and other bare rows are single runs. Treat SSOG as a promising research prototype rather than established SOTA.

Core Claim

SSOG replaces query-key dot-product scores in a Vision Transformer with a small mixture of learned two-dimensional Gaussian fields. Each head learns a fixed geometric routing prior; an optional query-conditioned controller can make bounded residual adjustments to each atom’s center, width, and mixture weight. Because each Gaussian factorizes over image rows and columns, value aggregation can use two one-dimensional passes rather than a dense $N\times N$ attention matrix.

The durable hypothesis is:

In domains with a meaningful coordinate system, most token routing may be carried by a compact learned geometric field, while content only needs to steer that field rather than compare every query with every key.

Mechanism

The blog writes each head’s score as a log-sum-exp mixture:

$$s(p,q)={\mathrm{logsumexp}}_r\left[\log {\lambda }_r+\log \mathcal{N}(p_y-q_y;{\mu }_{r,y},{\sigma }_{r,y})+\log \mathcal{N}(p_x-q_x;{\mu }_{r,x},{\sigma }_{r,x})\right],$$

then normalizes over key positions $q$ and applies the result to values. The steered variant uses bounded query-conditioned residuals such as:

$$\mu (p)={\mu }_0+\alpha \tanh (\Delta \mu (p)).$$

This creates a deliberately narrow content interface: geometry is the base operator, and query content changes a few interpretable routing parameters rather than generating unrestricted pairwise scores.

For square grids with $N=HW$ and fixed atom count $R$, the factorized spatial term scales as $O(RN\sqrt{N}d)$ rather than dense attention’s $O(N^{2}d)$. The steered implementation still materializes per-query one-dimensional kernels, so memory traffic and realized hardware performance remain empirical questions.

flowchart LR
    Q[query token] --> S[bounded steering]
    C[query coordinates] --> G[head-level Gaussian atoms]
    S --> G
    V[value grid] --> F[vertical and horizontal passes]
    G --> F
    F --> M[mixture output]

Author-Reported Results

Setting	SDPA	SSOG	Result boundary
CIFAR-100 toy ViT, matched 90 epochs	53.2% at learning rate $2\times 10^{-3}$; 54.3% at $10^{-3}$	70.0–70.4% across fixed and $+\mu \delta$ variants	The three-seed chart supports a 17.2-point gap against the author’s main SDPA row and 16.1 points against its better-learning-rate row. Steering is null at chart precision; no strong local/convolutional baseline is shown.
ImageNet-1k, $d=256$, 6 layers	3.66M, about 1.5G FLOPs, 64.34% (single run)	3.00M, about 1.0G FLOPs, $65.28\%\pm 0.02$ for full steering (three seeds)	+0.94 points, but the comparator has no uncertainty; fixed SSOG is 63.21–63.39%.
ImageNet-1k, $d=384$, 12 layers	14.94M, about 6.3G FLOPs, 71.84%	11.96M, about 4.4G FLOPs, 72.02%	+0.18 points; about 19.9% fewer parameters and 30.2% fewer tabulated FLOPs, with no uncertainty reported.
SSOG resolution transfer	no matched SDPA curve	72.0% at $224^2$; 73.7% at $288^2$; 71.6% at $384^2$; 63.4% at $512^2$	Requires position-embedding resize, is not monotone, and is not implemented in the release.

The blog also reports faster early ImageNet convergence and visualizes learned atom geometry, steering gates, and attention-mass flow. Without raw logs, checkpoints, seed-level records, or causal interventions, those plots are useful diagnostics but not independent performance or interpretability evidence.

Public-Code Audit

The audited commit contains the core attention module, a minimal ViT, and a CIFAR-100 script. Same-resolution forward passes succeeded. Four issues materially constrain the claims:

1. No headline reproduction path. The repository has no ImageNet code/config, checkpoints, metric logs, figure scripts, tests, CI, release, or license. The CIFAR script defaults to 100 rather than the blog’s matched 90 epochs.
2. Resolution transfer is not implemented. The ViT directly adds a token-count-specific position embedding. A live 8×8 to 12×12 test fails with ScopeParamShapeError for both SSOG and dot-product models. External bilinear parameter resizing is required but absent.
3. Displayed equation and code differ. The implementation normalizes each atom’s vertical/horizontal kernels separately and mixes outputs afterward; the blog displays one global softmax over a log-sum-exp Gaussian mixture. Boundary truncation and temperature make these operators non-equivalent. A retained deterministic check found maximum absolute weight difference 0.29178012726036207 for one non-degenerate two-atom case.
4. Asymptotic FLOPs are not realized speed. The steered path stores per-query one-dimensional kernels. End-to-end latency, peak memory, throughput, batch-size scaling, precision, and optimized-kernel comparisons are missing.

Novelty And Prior Art

The ingredients are individually established:

• Synthesizer, ICML 2021: token mixing without query-key dot products;
• AFT, 2021 preprint and LambdaNetworks, ICLR 2021 Spotlight: structured/position-aware linear alternatives to pairwise attention;
• DRAW, ICML 2015: recurrent-state-conditioned differentiable Gaussian filterbanks for selective image read/write, including a close precedent for content-controlled separable Gaussian geometry outside ViT mixing;
• Spatial Transformer Networks, NeurIPS 2015 and Deformable ConvNets, ICCV 2017: feature-conditioned spatial transforms and learned sampling offsets;
• Peripheral Vision Transformer, NeurIPS 2022 and positional attention, NeurIPS 2022: learned position-driven visual attention, including Gaussian-like regions and competitive position-only mixing;
• Gaussian Attention Bias, BMVC 2023, GAFlow, ICCV 2023, and LocAtViT, ICLR 2026: explicit Gaussian attention biases, deformable Gaussian attention, and query-specific Gaussian locality;
• Deformable DETR, ICLR 2021 and DAT, CVPR 2022: content-dependent spatial sampling/offsets.

SSOG’s plausible contribution is the combination: a few head-level Gaussian priors, bounded per-query steering, no query-key score matrix, and separable per-atom value aggregation. A formal novelty claim needs matched experiments against the closest Gaussian, positional-only, deformable, local, and attention-free mixers.

Why It Matters For This Wiki

SSOG is not time-series evidence, but it is a clean upstream architecture hypothesis for Time-Series Scaling And Efficiency: replace unrestricted pairwise matching with a small structured routing field when the domain has a useful coordinate system.

A temporal or world-model version cannot simply reuse symmetric image Gaussians. It would need:

• causal/asymmetric and multi-scale lag atoms;
• irregular timestamp support;
• separate routing over time, channels, and known topology;
• event- or state-conditioned bounded steering;
• tests for delayed effects, rare regimes, exogenous variables, action history, control inputs, interventions, and future-state utility.

The transfer target is therefore not “Gaussian attention for forecasting.” It is a structured temporal mixing interface whose geometric prior is cheap, interpretable, and only as restrictive as preservation probes allow.

Foundation TSFM Relevance

Agenda slot	Verdict	Evidence	Missing pieces
Scaling and efficiency	adjacent	Offers a coordinate-structured route from $N^2$ pair scores toward $N\sqrt{N}$ square-grid mixing with few routing parameters.	No time-series benchmarks, optimized kernels, latency/memory frontier, or full reproduction.
Inductive bias and sample efficiency	adjacent	The three-seed CIFAR chart is consistent with a useful vision-domain geometric prior on small data.	Strong convolutional/local/positional baselines, complete seed-level records with clearly defined CIFAR error bars, and architecture-matched hyperparameter tuning are absent.
Streaming latent state	insufficient evidence	A compact routing field could reduce token-mixing cost.	SSOG still recomputes over a retained grid and provides no recurrent state or constant-memory update.
Action-conditioned world models	insufficient evidence	Query-conditioned steering is a narrow content interface.	No actions, control inputs, interventions, trajectories, rollouts, or decision-utility evaluation.
Interpretability	adjacent	Centers, widths, mixture weights, and steering gates can be plotted directly.	Parameter readability is not causal faithfulness; intervention and counterfactual routing tests are needed.

Limitations

• Blog/code/X source rather than a formal or peer-reviewed paper.
• No independent reproduction; selected SSOG rows have standard deviations, but complete seed records, uncertainty for ImageNet SDPA and other single-run rows, and hyperparameter-search accounting are absent.
• Small benchmark set and classification-only evaluation.
• No strong closest-prior-art baselines.
• Public code does not reproduce ImageNet or resolution transfer and has no detected license.
• Blog mixture-softmax equation and released factorized operator differ.
• FLOP reductions are not accompanied by optimized wall-clock, memory, or throughput results.
• “Zero-shot resolution transfer” requires an external position-embedding resize, degrades sharply beyond $288^2$, and has no matched SDPA transfer curve in the blog.
• The time-series/world-model connection is a research hypothesis, not demonstrated transfer.

Links Into The Wiki

• SSOG
• Time-Series Scaling And Efficiency
• Foundation Time-Series Model Research Agenda
• TSMixer
• Tiny Time Mixers
• MiniMax Sparse Attention
• Oryx

Open Questions

• Is the CIFAR gain preserved against tuned convolutional, windowed/local, positional-only, PerViT/LocAt-style, AFT, and deformable baselines?
• Does bounded steering matter after sufficient model scale, or is the fixed geometric field doing nearly all useful work?
• Which operator should be evaluated: the blog’s global mixture-softmax, the released per-atom-normalized factorized path, or both?
• Can one optimized kernel beat FlashAttention-style dense attention in wall-clock and memory at realistic image sizes and batch sizes?
• Does the mixer preserve long-range correspondence on detection, segmentation, video, or tracking tasks?
• Can causal multi-scale lag atoms improve multivariate time-series modeling without erasing rare delayed events and intervention effects?