Alex Open Research Wiki

Small-Scale Experiments

2 min read

Small-Scale Experiments

Summary

Small-Scale Experiments is a model-centric research protocol for comparing model families before paying for large training runs. Its central premise is that small models can expose reliable scaling structure, but only after much more aggressive hyperparameter search than is customary. The protocol combines a tuned-frontier scaling law, noisy-quadratic search diagnostics, fixed-data loss-to-capability checks, and held-out larger scales.

Interface

model family + fixed pretraining data
  -> broad random search at several small scales
  -> verify the near-optimum score tail
  -> fit the best-loss frontier
  -> validate on a held-out larger scale
  -> check that equal loss still implies comparable capability
  -> compare near the measured range
  -> scale the winner with light retuning

The method does not treat one far-extrapolated loss as decisive. It asks whether tuning reached the local optimum, whether hyperparameter sensitivity falls with scale, whether pretraining loss remains a valid comparison proxy, and whether the fitted frontier survives a held-out scale.

Evidence Boundary

  • The introducing source is arXiv v1 from 2026-08-12, not peer reviewed.
  • The experiments cover decoder-only Llama variants from about 4M to 268M effective parameters on a fixed FineWeb-Edu subset.
  • The paper reports that scaling structure is absent with 4 or 16 configurations per scale, visible at 64, and accurate in the tested setup at 256.
  • Fitted local loss-surface dimension falls with model size, but the estimate is conditional on the search parameterization and noisy-quadratic asymptotic regime.
  • Far extrapolation remains unstable because the fitted irreducible error is weakly identified before loss saturation.
  • Public Lingua and OPDA artifacts exist, but no paper-specific training repository, full configuration release, logs, or checkpoints were found during ingest.

Role In The Wiki

This entity is the operational protocol card for time-series and world-model scaling work. It upgrades “train a few smaller models and extrapolate” into a falsifiable experiment:

  • reserve at least one larger scale for validation or test;
  • allocate enough compute to search the sensitive small-scale frontier;
  • distinguish near-data interpolation/ranking from irreducible-error-driven extrapolation;
  • keep corpus composition and every representation-changing data policy fixed when using loss as a capability proxy;
  • carry capability-aware probes for rare regimes, event streams, native multivariate coupling, context, exogenous variables, and action-conditioned rollout.

Official Artifacts

Graph View

Backlinks