--- abstract: | Self-supervised learning (SSL) has driven substantial progress in audio representation learning, though existing methods have increasingly relied on elaborate pre-training recipes to reach competitive performance. A markedly different pre-training philosophy underpins the most influential progress in language modeling and, more recently, in visual representation learning: rather than train encoders as static feature extractors, models are trained to predict the next element, a discrete token or a continuous embedding, from the preceding context. Autoregressive prediction thereby provides a unified pre-training interface that transfers across modalities, compelling the model to learn the underlying data distribution. We ask whether such a simple causal paradigm can yield strong audio learners, given that audio's temporal structure makes autoregressive prediction of patch embeddings a natural fit. We introduce NAPE (**N**ext-**A**udio-**P**atch-**E**mbedding prediction), a self-supervised framework in which a causal Transformer predicts each next patch embedding of a log-mel spectrogram from the previous ones, using causal masking and stop-gradient as its sole training signal. The design is intentionally minimalist, avoiding reconstruction decoders, acoustic tokenizers, student-teacher setups, and auxiliary regularization losses. Across six audio and speech benchmarks, NAPE achieves state-of-the-art fine-tuning performance on several tasks, scales consistently across encoder sizes, and yields strong linear-probing results. NAPE also produces structured attention patterns without explicit supervision. author: - Umberto Cappellazzo - Xubo Liu - Stavros Petridis - Maja Pantic date: 2026-08-20 title: "Listening Forward: Next Patch Embedding Prediction Enables Scalable Audio Learners" --- # Abstract Self-supervised learning (SSL) has driven substantial progress in audio representation learning, though existing methods have increasingly relied on elaborate pre-training recipes to reach competitive performance. A markedly different pre-training philosophy underpins the most influential progress in language modeling and, more recently, in visual representation learning: rather than train encoders as static feature extractors, models are trained to predict the next element, a discrete token or a continuous embedding, from the preceding context. Autoregressive prediction thereby provides a unified pre-training interface that transfers across modalities, compelling the model to learn the underlying data distribution. We ask whether such a simple causal paradigm can yield strong audio learners, given that audio's temporal structure makes autoregressive prediction of patch embeddings a natural fit. We introduce NAPE (**N**ext-**A**udio-**P**atch-**E**mbedding prediction), a self-supervised framework in which a causal Transformer predicts each next patch embedding of a log-mel spectrogram from the previous ones, using causal masking and stop-gradient as its sole training signal. The design is intentionally minimalist, avoiding reconstruction decoders, acoustic tokenizers, student-teacher setups, and auxiliary regularization losses. Across six audio and speech benchmarks, NAPE achieves state-of-the-art fine-tuning performance on several tasks, scales consistently across encoder sizes, and yields strong linear-probing results. NAPE also produces structured attention patterns without explicit supervision. # Introduction {#sec:introduction} Self-supervised learning (SSL) has emerged as a foundational paradigm for representation learning across modalities, delivering strong transfer performance without the cost of human annotations [@radford2018improving; @devlin2019bert; @brown2020language; @chen2021simsiam; @he2022masked; @oquab2023dinov2; @baevski2020wav2vec; @kong2024audio; @chen2024eat]. In audio, SSL methods largely adapt paradigms first developed in vision. Vision Transformer backbones [@dosovitskiy2020image] are pre-trained on spectrograms either through *masked-spectrogram modeling* [@niizumi2022masked; @huang2022masked; @chong2023masked; @chen2022beats; @dinkel2024scaling; @niizumi2026rethinking] or through *student-teacher distillation* objectives [@ahmed2024asit; @chen2024eat; @alex2025sslam; @yang2025spear], with domain-specific augmentations tailored to the time-frequency structure of audio. Most of these methods pre-train on AudioSet [@gemmeke2017audio] and are evaluated primarily by fine-tuning performance on downstream classification tasks. These paradigms, however, rely on increasingly elaborate pre-training recipes. Existing methods typically depend on reconstruction decoders that map latent features back to raw mel content [@huang2022masked; @chen2024eat], separately-trained acoustic tokenizers that supply discrete semantic targets [@chen2022beats], teacher-student setups with exponential moving average (EMA)-updated encoders [@chen2024eat; @alex2025sslam; @ahmed2024asit], auxiliary regularization losses that stabilize training [@fei2023jepa; @alex2025sslam], and multi-codebook vector quantisation [@yang2025spear]. Recently, a distinct line of work has begun to reshape SSL by shifting away from reconstruction toward the direct prediction of latent embeddings. This shift extends a paradigm long established in language modeling, where models are trained not as static feature extractors, but as predictive systems that model the data distribution through a single causal objective. Autoregressive prediction has thereby provided a unified pre-training interface across modalities, from discrete tokens in language to continuous embeddings in vision. Two variants of this philosophy have gained traction. *Joint-embedding predictive* approaches [@assran2023self; @oquab2023dinov2; @fei2023jepa; @bardes2024revisiting; @balestriero2025lejepa; @yuksel2025wavjepa; @huang2026text; @wu2026visreg] predict the latent embeddings of masked regions from a context view, produced by a target branch that is typically maintained via an EMA of the online encoder or stabilized by auxiliary regularization losses. While these methods are scalable and achieve strong performance, most of them rely on heavy heuristics (e.g., EMA, frozen layers, teacher-student architectures) to ensure training stability. *Autoregressive next-embedding* approaches [@teoh2025next; @xu2025nepa; @bredis2026next; @yao2026rethinking; @maes2026leworldmodel], in contrast, predict each latent embedding directly from the preceding ones, mirroring the causal next-token objective that drives modern large language models [@radford2018improving; @brown2020language; @liu2024deepseek; @yang2025qwen3]. Both variants have quickly become a promising direction for representation learning in *vision*. Despite this progress, predictive next-embedding methods *remain absent* from audio SSL, and joint-embedding predictive approaches themselves have seen *only limited exploration* in the audio and speech settings [@fei2023jepa; @yuksel2025wavjepa; @tuncay2025audio]. The absence of any next-embedding autoregressive method for audio is particularly striking because audio is arguably the modality *most* naturally suited to this paradigm. Unlike images, whose 2D structure is approximately isotropic and admits no canonical ordering, audio signals unfold along a well-defined temporal axis and sequential structure is intrinsic to the signal, not imposed on it. Predicting the next patch of a spectrogram from the past ones mirrors both how acoustic events emerge in time and how modern language models learn from sequential data. Motivated by this natural alignment, we seek a next-embedding prediction framework for audio that is deliberately minimalist while still delivering state-of-the-art downstream performance across audio and speech benchmarks. We therefore introduce NAPE (**N**ext-**A**udio-**P**atch-**E**mbedding prediction), a self-supervised framework that brings the causal next-embedding prediction paradigm to audio. Given a log-mel spectrogram, NAPE first applies a patch embedding layer that splits the spectrogram into non-overlapping patches and projects each of them into a $d$-dimensional embedding, producing a 2D grid of patch embeddings. Since the causal Transformer that follows operates on a 1D sequence, this grid must be traversed under a *scanning order*. This choice is a design axis specific to audio: unlike static images, spectrograms have a strong temporal axis and a qualitatively different frequency axis, so the order in which the grid is linearized determines both the causal context available at each prediction step and the structural inductive bias of the model. We consider four scanning strategies, depicted in Figure `\ref{fig:patch_type}`{=latex}. A causal Transformer encoder then processes the linearized sequence, and a lightweight predictor head produces an estimate of the next patch embedding, analogous to next-token prediction in language modeling, but operating in continuous embedding space rather than over a discrete vocabulary. Prediction quality is measured by negative cosine similarity to the target patch embedding under a stop-gradient. This objective requires no reconstruction decoder, no acoustic tokenizer, no student-teacher setup, and no auxiliary regularization losses: the entire learning signal comes from the model's ability to anticipate the next embedding in the sequence. Extensive experiments on standard audio and speech benchmarks, including AudioSet [@gemmeke2017audio], ESC-50 [@piczak2015esc], Speech Commands V1 and V2 [@warden2018speech], and IEMOCAP [@busso2008iemocap], show that NAPE achieves state-of-the-art performance on several tasks. NAPE exhibits favorable scaling properties across three encoder sizes: *Small*, *Base*, and *Large*, with 19/85/303 million parameters, respectively. Beyond fine-tuning, NAPE delivers strong linear-probing results despite being a purely predictive model whose objective is not aligned with linear separability, indicating that the learned representations remain discriminative even under strict feature-freeze evaluation. A qualitative analysis of NAPE's attention patterns and embedding-space structure further confirms that NAPE learns meaningful, structured features from audio. Our main contributions are: - We introduce NAPE, the first self-supervised audio framework built around causal next-patch-embedding prediction, offering a substantially simpler alternative to reconstruction-based, masked-modeling, and joint-embedding predictive approaches for audio. - Through systematic ablations, we identify the design axes that make next-embedding prediction work in the audio setting---scanning order, predictor head, prediction target, patch embedding layer, and the three key components of causality, prediction shift, and stop-gradient---and converge on an optimal configuration for the framework. - Across six audio and speech benchmarks, NAPE achieves strong downstream results at three encoder scales, exhibits favorable scaling behavior, and delivers competitive linear-probing performance. - We provide qualitative evidence that NAPE learns structured representations, with attention patterns that reason jointly about the current spectral context and the frequency-consistent temporal history, and embedding-space behavior that groups acoustically-similar patches without any explicit labels. # The NAPE Framework {#sec:methodology} ![**Left:** NAPE's overview. The input spectrogram is split into patches and embedded into a sequence of embeddings. At each step, the model predicts the embedding of the [next patch]{style="color: red"} (red border) using only the embeddings of the [preceding patches]{style="color: greenpatch"}; patches at [future positions]{style="color: darkgray"} are hidden by the causal attention mask. **Middle:** The NAPE pipeline: patch embeddings $z$ are processed by the causal encoder $h$ and predictor $g$ to produce predictions $\hat{z}$, which are compared against the targets $z$ under stop-gradient using a similarity function $\mathcal{D}$ (i.e., negative cosine similarity). **Right:** Encoder architecture ($h$): multiple stacked Transformer layers with pre-norm design, causal self-attention, LayerScale, and query-key normalization.](Figures/Architecture.png){#fig:architecture width="100%"} In this section, we describe the NAPE framework in detail. NAPE is a self-supervised pre-training method that trains a causal Transformer to predict the embedding of the next patch of a log-mel spectrogram from the preceding ones, relying only on causal masking and stop-gradient. We describe NAPE's main components and downstream adaptation in the next subsections. Figure `\ref{fig:architecture}`{=latex} depicts the overall framework. ## Spectrogram Patchification and Scanning Order {#sec:nape-patches} #### Input Representation. Given a raw waveform, we compute a log-mel spectrogram $x \in \mathbb{R}^{1 \times F \times T_{\text{frames}}}$ with $F$ frequency bins and $T_{\text{frames}}$ time frames. We split $x$ into non-overlapping square patches of size $P \times P$, yielding a 2D grid of $N = T_F \cdot T_T$ patches with $T_F = F/P$ and $T_T = T_{\text{frames}}/P$. Each patch is projected to a $d$-dimensional embedding by a patch embedding layer $f$, producing the sequence $\{z_1, \ldots, z_N\} \in \mathbb{R}^{N \times d}$. We use a standard Conv2d patch embedding layer by default, but we also consider a deeper convolutional stem with batch normalization (*convstem*) [@xiao2021early] and a speech-oriented stem that treats the mel axis as feature channels and applies temporal-only convolutions (*speechstem*) [@team2026gemma]. We refer to Section `\ref{sec:ablations}`{=latex} for details and results about the choice of $f$. #### Scanning Order. Transformer encoders operate on 1D sequences, so a 2D patch grid must be linearized before it can be processed. Under bidirectional attention, as used in prior audio SSL methods [@gong2021ast; @huang2022masked; @chen2024eat], the choice of linearization is not a functional design choice: self-attention is permutation-equivariant given positional embeddings, so any consistent ordering yields the same representations. NAPE's causal formulation, however, breaks this equivariance as the ordering determines which patches are \`\`past" (visible to a given position) and which are \`\`future" (to be predicted), and therefore imposes a substantive inductive bias on the model. Since spectrograms have a strong temporal axis and a qualitatively different frequency axis, unlike natural images, whose 2D structure is approximately isotropic, the choice of scanning order for causal prediction is particularly consequential. Thus, we consider four orderings, illustrated in Figure `\ref{fig:patch_type}`{=latex}. ***Raster*** (left-to-right, bottom-to-top) sweeps time before advancing in frequency and is the standard patch ordering used in vision and audio Transformer models, where each prediction is conditioned on the entire past time axis at every frequency traversed so far. ***Time-major*** (bottom-to-top within each time column, then advance in time) sweeps frequency before advancing in time, so each prediction is conditioned on the full frequency profile of every past time step. ***Zigzag*** is an alternating-direction variant of raster: rows alternate left-to-right and right-to-left, so consecutive patches in the sequence remain spatially adjacent even across row transitions. This avoids the spatial discontinuity raster incurs when jumping from the end of one row to the start of the next. ***Diagonal*** sweeps patches along anti-diagonals of the grid, mixing time and frequency progression at every step and thereby encoding 2D spatial priors more uniformly than the other three orders. ![**Patch scanning orders.** NAPE linearizes the 2D spectrogram patch grid into a 1D causal sequence in one of four ways: *raster* (left-to-right, bottom-to-top), *time-major* (bottom-to-top within each time column, then advance in time), *zigzag* (raster with alternating row directions), and *diagonal* (sweep by frequency-plus-time index). The numbers indicate the position of each patch in the resulting sequence.](Figures/Patch_processing_type.png){#fig:patch_type width="100%"} ## Next-Audio-Patch-Embedding Prediction {#sec:nape-prediction} #### Prediction Task. Given the embedding sequence $z = \{z_1, \ldots, z_N\}$ produced by $f$ in the chosen scanning order, NAPE jointly trains an encoder $h$ and a lightweight predictor head $g$ so that, at each position $t$, the model produces an estimate of the next patch embedding $z_{t+1}$ using only the previous patches: $$\hat{z}_{t+1} = g\!\left(h(z_{\leq t})\right), \label{eq:nape-prediction}$$ where $z_{\leq t} = \{z_1, \ldots, z_t\}$ denotes the patch embeddings at all positions up to and including $t$. The restriction to the past patches is enforced by a *causal attention mask* that prevents each position from attending to patches at later positions in the sequence. This is directly analogous to next-token prediction in language modeling, but operates in a continuous embedding space rather than over a discrete vocabulary. #### Loss Function. Following SimSiam [@chen2021simsiam], we measure prediction quality by the negative cosine similarity between the target embedding $z_{t+1}$ and the predicted embedding $\hat{z}_{t+1}$: $$\mathcal{D}(z_{t+1}, \hat{z}_{t+1}) \;=\; -\, \frac{z_{t+1}}{\lVert z_{t+1} \rVert_2} \cdot \frac{\hat{z}_{t+1}}{\lVert \hat{z}_{t+1} \rVert_2}, \label{eq:cosine-similarity}$$ where $\lVert \cdot \rVert_2$ is the $\ell_2$-norm. Applying stop-gradient (`stopgrad`) on the target and averaging over all valid prediction positions yields the NAPE objective: $$\mathcal{L} \;=\; \frac{1}{N-1} \sum_{t=1}^{N-1} \mathcal{D}\!\left(\texttt{stopgrad}(z_{t+1}), \; \hat{z}_{t+1}\right). \label{eq:nape-loss}$$ The `stopgrad` operator treats the target embedding as a constant, so gradients flow only through the predicted side. Cosine similarity is magnitude-invariant, which prevents the trivial solution of shrinking both sides of the objective toward zero norm. We compare cosine similarity against alternative similarity functions in Section `\ref{sec:ablations}`{=latex} and find it to yield the optimal results. #### Causal Transformer Encoder ($h$). We use a Vision Transformer backbone [@dosovitskiy2020image; @wang2026vit] with pre-norm design and a causal attention mask during pre-training (see Figure `\ref{fig:architecture}`{=latex}, right panel). For stability at depth, we adopt Rotary Position Embedding (RoPE) [@su2024roformer] applied independently along the frequency and time axes, LayerScale [@touvron2021going] on residual branches, and parameter-free query-key normalization [@henry2020query] on the per-head query and key projections. We instantiate three configurations of increasing size to test NAPE's scalability: *Small* ($d=384$, $12$ layers, $6$ heads; $\sim$``{=html}19M parameters), *Base* ($d=768$, $12$ layers, $12$ heads; $\sim$``{=html}85M parameters), and *Large* ($d=1024$, $24$ layers, $16$ heads; $\sim$``{=html}303M parameters). #### Predictor Head ($g$). The predictor head $g$ decouples the representation space of $h$ from the space in which predictions are made. Predicting directly with the encoder output forces the encoder to place its representations in the same space as the prediction targets, which can restrict feature richness. This asymmetric predictor design, together with the stop-gradient applied to the target branch, gives NAPE a structural resemblance to SimSiam [@chen2021simsiam]: representation collapse is avoided without contrastive negatives or an EMA teacher, relying instead on the asymmetry between the two branches. NAPE differs from SimSiam in what the branches encode: SimSiam compares two augmented views of the same input under a symmetric encoder, whereas NAPE compares a prediction of the next patch to that patch's actual embedding within a single autoregressive sequence, replacing view augmentation and siamese symmetry with temporal prediction as the source of the learning signal. We study multiple predictor styles in Section `\ref{sec:ablations}`{=latex}. ```{=latex} \begin{wrapfigure}{r}{0.47\textwidth} \begin{algorithm}[H] \caption{NAPE pre-training algorithm.} \label{alg:nape} \begin{lstlisting}[style=pseudocode,linewidth=0.46\textwidth] #f: Patch Embedding Layer #h: Causal Transformer Encoder #g: Predictor for x in loader: # x: [B,1,F,T_frames] z = f(x) # embeddings [B,T,D] z_hat = g(h(z)) # predictions [B,T,D] loss = D(z, z_hat) loss.backward(); # update(f,h,g) def D(z, z_hat): target = z[:,1:, :].detach() # stop-grad pred = z_hat[:,:-1, :] # AR shift pred = normalize(pred, dim=-1) target = normalize(target, dim=-1) return -(pred*target).sum(-1).mean() \end{lstlisting} \end{algorithm} \end{wrapfigure} ``` #### NAPE's Key Components. Three complementary mechanisms prevent NAPE from converging to trivial solutions and together define its training regime. (i) ***causality***: enforced by the causal attention mask that restricts each position to attend only to prior patches, it prevents the encoder from attending to the target patch when producing its prediction, blocking the trivial identity mapping. (ii) The ***prediction shift*** between input and target positions ensures that the model at position $t$ predicts the embedding at position $t+1$ rather than the embedding at its own position, so causality alone cannot be side-stepped by copying the current input through. (iii) ***stop-gradient*** on the target embedding prevents gradients from flowing through both sides of the loss simultaneously, which would otherwise allow the encoder to collapse all embeddings toward a shared constant. Together, these three ingredients constitute the core of NAPE's self-supervised recipe, and we analyze the individual contribution of each in Section `\ref{sec:ablations}`{=latex}. The pseudocode of NAPE's pre-training is given in Algorithm `\ref{alg:nape}`{=latex}. # Experiments {#sec:experiments} In this section, we carry out extensive experiments to evaluate NAPE at different scales and configurations. **(1)** We first perform several ablations on key design choices to converge on the best NAPE configuration (Section `\ref{sec:ablations}`{=latex}); **(2)** we then compare that configuration against state-of-the-art methods at multiple scales (Section `\ref{sec:scaling}`{=latex} and  `\ref{sec:sota}`{=latex}); **(3)** we assess linear separability of the learned features (Section `\ref{sec:linear-probe}`{=latex}), and **(4)** we finally analyze how NAPE organizes acoustic information through its attention patterns and the structure of its learned embeddings (Section `\ref{sec:qualitative}`{=latex}). ## Experimental Setup {#sec:setup} #### Pre-training Data. We pre-train NAPE on AudioSet [@gemmeke2017audio] without labels, combining the unbalanced and balanced training splits. We obtained and processed $1{,}964{,}222$ clips from the unbalanced split, $20{,}961$ clips from the balanced split, and $18{,}900$ clips from the evaluation split, consistent with prior work [@chen2022beats; @chen2024eat; @alex2025sslam]. All input waveforms are resampled to mono at $16$ kHz and converted into log-mel spectrograms with $128$ mel bands using a $25$ ms Hanning window and $10$ ms hop size. A $10$-second clip yields a spectrogram of size $1 \times 128 \times 1008$ (channel, frequency, time), corresponding to $8 \times 63 = 504$ non-overlapping $16 \times 16$ patches per clip (as in  [@gong2021ast; @chen2022beats; @chen2024eat]). #### Downstream Benchmarks. For downstream evaluation, we fine-tune on **AudioSet-2M** (*AS-2M*, unbalanced) and **AudioSet-20K** (*AS-20K*, balanced), applying the weighted sampling strategy of @huang2022masked on AS-2M to mitigate class imbalance. We further evaluate on **ESC-50** [@piczak2015esc], a $50$-class environmental sound classification benchmark of $2000$ clips at $5$s each, using $5$-fold cross-validation; **Speech Commands V1 and V2** (*KS1*, *KS2*) [@warden2018speech], keyword spotting tasks with $12$ and $35$ classes respectively; and **IEMOCAP** (*ER*) [@busso2008iemocap], a $4$-class speech emotion recognition benchmark with $5$-fold cross-validation. Together these tasks span both audio and speech domains. We report mean average precision (mAP) on the multi-label AudioSet tasks and top-1 classification accuracy on the single-label tasks; for ESC-50 and IEMOCAP we report the mean across cross-validation folds. #### Pre-training Details. We pre-train with the AdamW optimizer [@loshchilov2017decoupled] using a base learning rate of $5 \times 10^{-3}$ with cosine decay, a weight decay of $0.05$, $\beta_1 = 0.9$, $\beta_2 = 0.95$, and a batch size of $256$ for the Small and Base configurations, and $128$ for NAPE Large. We pre-train NAPE Small and NAPE Large for $25$ epochs and NAPE Base for $30$ epochs, with a warmup ratio of $10\%$. All models are trained using HuggingFace `Trainer` with distributed data parallelism (DDP) on NVIDIA L40s GPUs (46GB). #### Fine-tuning Details. For each downstream task, we initialize from the pre-trained encoder and attach a linear classifier on top of mean-pooled patch tokens. We disable the causal attention mask during fine-tuning so that attention is bidirectional over the full patch sequence. We provide an ablation study on the optimal pooling method and whether to use causal/bidirectional attention in the Appendix `\ref{sec:additional_ablations}`{=latex}. We use AdamW with $\beta_2 = 0.999$, cosine learning-rate decay, and layer-wise learning-rate decay [@clark2020electra; @bao2021beit]. We use binary cross-entropy for AudioSet and soft-target cross-entropy for the single-label tasks. Following prior work [@gong2021ast; @chen2024eat; @chen2022beats], we apply a standard augmentation stack (SpecAugment [@park2019specaugment], Mixup [@zhang2017mixup], CutMix [@yun2019cutmix], DropPath [@huang2016deep], temporal roll, additive noise, and label smoothing [@szegedy2016rethinking]). For AS-2M and AS-20K we additionally maintain an exponential moving average (EMA) [@polyak1992acceleration] of the fine-tuning weights (decay $0.99995$ and $0.999$ respectively); no EMA is used for the other tasks. Full task-specific hyperparameters are reported in the Appendix `\ref{sec:hyperparameters}`{=latex}. #### Linear Probing Details. For linear probing, we freeze the pre-trained encoder and train only the classifier plus a preceding LayerNorm [@ba2016layer]. All augmentations are disabled. #### Baselines. We primarily compare against recent in-domain self-supervised methods, including Audio-MAE [@huang2022masked], BEATs [@chen2022beats], A-JEPA [@fei2023jepa], ASiT [@ahmed2024asit], EAT [@chen2024eat], SSLAM [@alex2025sslam], and SPEAR [@yang2025spear]. We also include results from out-of-domain and in-domain supervised pre-training. For all baseline methods, fine-tuning results are taken from the original papers. ## NAPE's Optimal Configuration {#sec:ablations} In this section, we ablate the main design axes of NAPE to identify its optimal configuration. Unless otherwise specified, all ablations use the NAPE *Base* model and the raster scanning order. Each ablation modifies a single design axis at a time, holding the rest of the configuration fixed at NAPE's defaults. #### `Prediction Shift/Stop-gradient/Causality.`{=latex} Table `\ref{tab:maincomponents}`{=latex} disentangles the three key mechanisms of NEPA: the *prediction shift*, the *stop-gradient* on the target, and the *causal attention mask*. Removing either the prediction shift or the stop-gradient causes pre-training to diverge, matching the analysis in Section `\ref{sec:nape-prediction}`{=latex}: without the shift, the model at position $t$ can trivially satisfy the objective by copying its own input embedding through; without the stop-gradient, gradients flow through both sides of the loss and the encoder collapses all embeddings toward a shared constant. ```{=latex} \begin{wraptable}{r}{0.54\textwidth} \small %\renewcommand{\arraystretch}{0.9} % \renewcommand{\tabcolsep}{0.5mm} \caption{\textbf{Ablation on main NAPE's design elements}: \emph{prediction shift}, \emph{stop-gradient}, and \emph{causal objective}.} \label{tab:maincomponents} \begin{tabular}{ccccccccc} \toprule \textbf{Pred} & \textbf{stop} & \textbf{causal} & \textbf{Audio Tasks} & \textbf{Speech Tasks}\\ \textbf{shift} & \textbf{grad} & \textbf{mask} & \emph{AS-2M} & \emph{AS-20K} & \emph{ESC-50} & \emph{KS1} & \emph{KS2} & \emph{ER} \\ \midrule \ding{56} & \ding{51} & \ding{51} & \color{red}\textbf{Diverge} \\ \ding{51} & \ding{56} & \ding{51} & \color{red}\textbf{Diverge} \\ \hline \ding{51} & \ding{51} & \ding{56} & 41.8 & 24.8 & 68.9 & 96.1 & 97.3 & 57.0 \\ \ding{51} & \ding{51} & \ding{51} & \textbf{49.6} & \textbf{39.1} & \textbf{94.2} &\textbf{97.9} & \textbf{98.8} & \textbf{64.9} \\ \bottomrule \end{tabular} \end{wraptable} ``` Removing the causal mask, in contrast, does not diverge: the prediction shift alone continues to define a nominal prediction target, but degrades downstream performance sharply, with the largest drops on the audio benchmarks ($-7.8$ mAP on AS-2M, $-14.3$ on AS-20K, and $-25.4$ points on ESC-50; smaller but consistent drops on the speech tasks). Without causality, the encoder can attend to the target patch while producing its prediction, so the objective is trivially satisfied by a near-identity mapping that routes each target back to its predicted position (the loss saturates near $-1$ within a few thousand steps of pre-training, see Appendix `\ref{sec:loss_visualization}`{=latex}): the loss decreases during pre-training, but the encoder is not forced to learn useful structure. All three mechanisms are therefore jointly necessary, none can be dropped without either destabilizing training or degrading the learned representations to a degree that fine-tuning cannot recover. ```{=latex} \begin{wraptable}{r}{0.49\textwidth} \small \renewcommand{\tabcolsep}{0.5mm} \caption{Ablation on the patch embedding layer.} %\resizebox{0.999\linewidth}{!}{ \begin{tabular}{lcccccc} \toprule \textbf{Patch Emb.} & \textbf{Audio Tasks} & \textbf{Speech Tasks}\\ \textbf{Layer} & \emph{AS-2M} & \emph{AS-20K} & \emph{ESC-50} & \emph{KS1} & \emph{KS2} &\emph{ER} \\ \midrule Convstem & 46.7 & 34.3 & 89.1 & 97.4 & 98.3 & 63.6 \\ Speechstem & 47.6 & 33.1 & 88.4 & \textbf{98.1} & \textbf{98.9} & 63.0 \\ \hline \addlinespace[2pt] \textbf{Conv2d} & \textbf{49.6} & \textbf{39.1} & \textbf{94.2} & 97.9 & 98.8 & \textbf{64.9} \\ \bottomrule \end{tabular}%} \label{tab:patchemb} \end{wraptable} ``` #### `Patch Embedding Layer $f$.`{=latex} Table `\ref{tab:patchemb}`{=latex} compares three patchifiers, all configured to produce the same sequence length (504 tokens per $10$s clip). The default *Conv2d* is a single strided convolution with kernel and stride $16 \times 16$, which treats time and frequency as symmetric 2D axes and matches the standard ViT design [@dosovitskiy2020image]. We also consider two alternatives motivated by observations from computer vision and speech literature. The *Convstem*, following @xiao2021early, replaces the single Conv2d with four $3 \times 3$ stride-2 convolutions interleaved with batch normalization with the channel count growing progressively toward $d$; deeper convolutional stems have been shown to stabilize optimization and improve downstream performance in ViT-based image models. The *Speechstem*, inspired by the audio front-end of speech-oriented models such as Gemma 3n and Gemma 4 [@team2026gemma], flattens the mel axis into feature channels and applies temporal-only $3 \times 3$ convolutions with a temporal downsampling factor of $2$, producing a 1D time-only sequence of $504$ tokens that reflects the frame-based processing typical of speech recognition systems. Both alternatives underperform Conv2d across all benchmarks, with the largest gaps on AudioSet: on AS-20K, Convstem loses $4.8$ mAP points and Speechstem loses $6.0$ mAP spoints relative to Conv2d. Convstem's added non-linearity and speechstem's temporal-first inductive bias therefore do not translate into gains for our causal spectrogram prediction objective, treating frequency and time as symmetric 2D axes at the patchification stage is important for the pretraining signal that NAPE exploits. Conv2d remains NAPE's default. ```{=latex} \begin{wraptable}{r}{0.555\textwidth} \small \renewcommand{\tabcolsep}{0.5mm} \caption{Ablation on the predictor-style variants.} %\resizebox{0.999\linewidth}{!}{ \begin{tabular}{lccccccc} \toprule \textbf{Predictor} & \textbf{\#Par.} & \textbf{Audio Tasks} & \textbf{Speech Tasks}\\ \textbf{Style} & & \emph{AS-2M} & \emph{AS-20K} & \emph{ESC-50} & \emph{KS1} & \emph{KS2} &\emph{ER} \\ \midrule \textcolor{darkgray}{None} & - & 48.7 & 37.8 & 93.3 & 98.1 & \textbf{98.8} & 64.2 \\ 2-MLP & 1.2M & 49.4 & 38.5 & 93.6 & 98.0 & \textbf{98.8} & 64.2 \\ Transformer & 14.2M & 49.2 & 38.4 & 93.0 & \textbf{98.2} & 98.7 & \textbf{65.0} \\ \textbf{SimSiam} & 1.8M & \textbf{49.6} & \textbf{39.1} & \textbf{94.2} & 97.9 & \textbf{98.8} & 64.9 \\ \bottomrule \end{tabular}%} \label{tab:predictor_table} \end{wraptable} ``` #### `Predictor $g$.`{=latex} Table `\ref{tab:predictor_table}`{=latex} compares four predictor variants: no predictor (encoder output used directly as the prediction), a two-layer MLP (*2-MLP*; $\text{Linear}(d,d) \to \text{GELU} \to \text{Linear}(d,d)$), a *SimSiam*-style [@chen2021simsiam] three-layer predictor with intermediate LayerNorms ($\text{Linear}(d,d) \to \text{LN} \to \text{GELU} \to \text{Linear}(d,d) \to \text{LN} \to \text{GELU} \to \text{Linear}(d,d)$), and a *Transformer* predictor in the style of JEPA-family methods: a 2-layer causal Transformer ($16$ heads) operating in the encoder's hidden dimension (we apply a causal mask to prevent future leaking as for the encoder). Adding any predictor improves over using the encoder output directly, confirming that decoupling the representation and prediction spaces benefits the learned features. Among the three predictor styles, the SimSiam variant performs best on the audio benchmarks and on IEMOCAP, while remaining on par on the keyword-spotting tasks. Notably, the lightweight SimSiam predictor outperforms the Transformer predictor despite being nearly $8\times$ smaller in parameters, indicating that additional predictor capacity is not the bottleneck for NAPE, consistent with the observation in the siamese self-supervised setting [@chen2021simsiam] that a compact MLP predictor suffices when combined with a well-designed encoder and stop-gradient target. The SimSiam-style predictor is NAPE's default. #### `Patch Scannning Variants.`{=latex} Figure `\ref{fig:variants}`{=latex} compares the four scanning orders introduced in Section `\ref{sec:nape-patches}`{=latex} (we use the 2-MLP predictor style). *Diagonal*, *raster*, and *zigzag* all perform comparably well across the six benchmarks, with diagonal and raster slightly ahead of zigzag on most tasks. In contrast, *time-major* consistently underperforms the other three orders across all tasks. We interpret this pattern as reflecting the temporal structure of audio: diagonal, raster, and zigzag all advance in time as the causal sequence progresses, allowing the model to accumulate temporal context in a way that matches how acoustic events unfold. Time-major, instead, exhausts each frequency column before advancing in time, so predictions early in the sequence are conditioned on rich instantaneous spectra but only limited temporal context, a mismatch with the temporal nature of audio classification. Given their superior performance, we retain both raster and diagonal in the main comparison against state-of-the-art methods in Section `\ref{sec:sota}`{=latex}. ![`NAPE's performance across four scan orders on six benchmarks.`{=latex}](Figures/variants.png){#fig:variants width="100%"} ```{=latex} \begin{wraptable}{r}{0.505\textwidth} \small \renewcommand{\tabcolsep}{0.5mm} \caption{Ablation on NAPE's target to predict.} %\resizebox{0.999\linewidth}{!}{ \begin{tabular}{lcccccc} \toprule \textbf{Predicted} & \textbf{Audio Tasks} & \textbf{Speech Tasks}\\ \textbf{Target} & \emph{AS-2M} & \emph{AS-20K} & \emph{ESC-50} & \emph{KS1} & \emph{KS2} &\emph{ER} \\ \midrule 1st enc. layer & \color{red}\textbf{Diverge} \\ \hline \addlinespace[2pt] Raw Mel & \textbf{49.7} & 38.0 & \textbf{94.8} & 97.7 & 98.6 & 64.2 \\ \textbf{Patch embed} & 49.6 & \textbf{39.1} & 94.2 & \textbf{97.9} & \textbf{98.8} & \textbf{64.9} \\ \bottomrule \end{tabular}%} \label{tab:target} \end{wraptable} ``` #### `Prediction Target.`{=latex} Table `\ref{tab:target}`{=latex} compares three choices for the target of the auto-regressive prediction: (i) the *patch embedding* $z_{t+1}$ produced by the shared embedding layer $f$ (the default); (ii) the *raw mel* content of the next patch i.e., the flattened mel-spectrogram values inside the patch, following the target formulation used by masked reconstruction methods such as Audio MAE [@huang2022masked]; (iii) the output of the *first encoder layer*, in the style of JEPA methods that use deeper encoder features as prediction targets. For the raw mel variant, we add a linear projection on top of the predictor $g$ to map its output from $d$ dimensions to the raw patch dimensionality. Patch embedding and raw mel yield comparable results across all benchmarks, indicating that both are valid target choices for NAPE. Using the first encoder layer as target, however, causes pre-training to diverge: the target itself depends on the encoder being trained, and stop-gradient alone is insufficient to prevent the encoder from collapsing both sides of the loss to a shared constant. JEPA-family methods circumvent this instability with additional regularization such as EMA teachers [@assran2023self; @fei2023jepa], variance-covariance regularizers such as VISReg [@wu2026visreg] and sketched isotropic gaussian regularizers such as SIGReg [@balestriero2025lejepa], which we do not employ here. Since the patch embedding approach on average performs better than raw mel and it is adopted in  [@xu2025nepa] as well, we retain the patch embedding as NAPE's default. ```{=latex} \begin{wraptable}{r}{0.5\textwidth} \small \renewcommand{\tabcolsep}{0.5mm} \caption{Ablation on the similarity function.} %\resizebox{0.999\linewidth}{!}{ \begin{tabular}{lcccccc} \toprule \textbf{Similarity} & \textbf{Audio Tasks} & \textbf{Speech Tasks}\\ \textbf{Function} & \emph{AS-2M} & \emph{AS-20K} & \emph{ESC-50} & \emph{KS1} & \emph{KS2} &\emph{ER} \\ \midrule L1 & \color{red}\textbf{Diverge} \\ L2 & \color{red}\textbf{Diverge} \\ \hline \addlinespace[2pt] Cross-entropy & 48.8 & 37.4 & 93.6 & \textbf{98.0} & 98.7 & 64.5 \\ \textbf{Cosine} & \textbf{49.6} & \textbf{39.1} & \textbf{94.2} & 97.9 & \textbf{98.8} & \textbf{64.9} \\ \bottomrule \end{tabular}%} \label{tab:loss} \end{wraptable} ``` #### `Similarity Function ($\mathcal{D}$).`{=latex} In Table `\ref{tab:loss}`{=latex} we compare four choices for the similarity function $\mathcal{D}$ in Eq. `\ref{eq:cosine-similarity}`{=latex}: the *negative cosine similarity* (NAPE's default), a soft *cross-entropy* formulation that treats prediction and target as distributions over the $d$ channels after applying a softmax (as in [@chen2021simsiam]), and the $\ell_1$ and $\ell_2$ distances between the two $d$-dimensional vectors. Both $\ell_1$ and $\ell_2$ cause pre-training to diverge: unlike cosine similarity, which is magnitude-invariant, these distance-based objectives can be trivially minimized by shrinking the norm of both predicted and target embeddings toward zero, a form of representation collapse in which the encoder outputs converge to a shared low-magnitude constant. The cross-entropy variant trains stably (softmax normalization implicitly bounds the target magnitude) but underperforms cosine on all benchmarks except the keyword-spotting tasks, where the two are on par. Cosine similarity, combining magnitude-invariance with a directionally informative loss signal, yields the best overall results and is retained as NAPE's default. ```{=latex} \begin{boxaccent}{NAPE's Optimal Configuration} (i) \textbf{Conv2d} patch embedding layer, (ii) \textbf{raster}/\textbf{diagonal} scanning order, (iii) \textbf{SimSiam-style predictor} with three-layer MLP and intermediate LayerNorms, (iv) \textbf{patch embedding} as the prediction target, and (v) \textbf{negative cosine similarity} loss with stop-gradient on the target branch. \end{boxaccent} ``` **Additional Ablation Results.** We refer to Appendix `\ref{sec:additional_ablations}`{=latex} for additional ablations studies on: **1)** the use of normalization layers (LayerNorm vs RMSNorm), **2)** freezing/unfreezing the patch embedding layer, **3)** the optimal positional encoding (absolute encoding vs RoPE), **4)** the optimal pooling method during fine-tuning, **5)** the optimal attention type during fine-tuning (causal vs bidirectional), **(6)** the additional use of random masking during pre-training, and **7)** NAPE's performance in terms of different compute budgets. ## Scaling NAPE {#sec:scaling} ```{=latex} \begin{wrapfigure}{r}{0.45\textwidth} \includegraphics[width=\linewidth]{Figures/nape_vs_audiomae.pdf} \caption{Scaling comparison between Audio-MAE and NAPE, raster.} \label{fig:audiomae_vs_nape} \end{wrapfigure} ``` Figure `\ref{fig:main_results}`{=latex} shows downstream performance for the raster and diagonal variants of NAPE at three encoder scales: *Small* ($\sim$``{=html}19M parameters, NAPE-S), *Base* ($\sim$``{=html}85M, NAPE-B), and *Large* ($\sim$``{=html}303M, NAPE-L). NAPE scales positively across the board: on every one of the six benchmarks, NAPE-L improves over NAPE-B, which in turn improves over the small version. While this improvement tends to diminish as we scale from the base to the large model, we observe consistent scaling gains for most of the tasks, while the keyword-spotting tasks show smaller absolute gains, reflecting their already-saturated accuracy levels. Raster and diagonal track each other closely at the Small scale, with essentially identical performance across all six tasks. At the Base and Large scales, diagonal slightly outperforms raster on most benchmarks, while raster achieves the strongest result on AS-2M with NAPE-L reaching $50.18$ mAP. In Figure `\ref{fig:audiomae_vs_nape}`{=latex}, we compares NAPE (raster) and Audio-MAE [@huang2022masked] at three encoder scales on AS-2M and AS-20K. NAPE outperforms Audio-MAE at every scale, with particularly large margins at the Small size ($+2.6$ mAP on AS-2M, $+4.1$ mAP on AS-20K). NAPE also exhibits more favorable scaling behavior from Base to Large. ![NAPE's results at different scales under raster and diagonal scan orders.](Figures/6plots.png){#fig:main_results width="100%"} ```{=latex} \small ``` ```{=latex} \renewcommand{\tabcolsep}{1.5mm} ``` +:-----------------------------------------------------------------------+:------------------------------:+:--------------------------------:+:-------------------------------:+:-------------------------------:+:-------------------------------:+:-------------------------------:+:-------------------------------:+:-------------------------------:+ | **Model** | **#Par.** | **Pre-train** | **Audio Tasks** | **Speech Tasks** | | | | | +------------------------------------------------------------------------+--------------------------------+----------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+ | | | **Data** | *AS-2M* | *AS-20K* | *ESC-50* | *KS1* | *KS2* | *ER* | +------------------------------------------------------------------------+--------------------------------+----------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+ | ***Out-of-domain Supervised Pre-training*** | | | | | | | | | +------------------------------------------------------------------------+--------------------------------+----------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+ | `\addlinespace`{=latex}[PSLA]{style="color: darkgray"} [@gong2021psla] | [14M]{style="color: darkgray"} | [IN]{style="color: darkgray"} | [44.4]{style="color: darkgray"} | [31.9]{style="color: darkgray"} | [-]{style="color: darkgray"} | [-]{style="color: darkgray"} | [96.3]{style="color: darkgray"} | [-]{style="color: darkgray"} | +------------------------------------------------------------------------+--------------------------------+----------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+ | [AST]{style="color: darkgray"} [@gong2021ast] | [86M]{style="color: darkgray"} | [IN]{style="color: darkgray"} | [45.9]{style="color: darkgray"} | [34.7]{style="color: darkgray"} | [88.7]{style="color: darkgray"} | [95.5]{style="color: darkgray"} | [98.1]{style="color: darkgray"} | [56.0]{style="color: darkgray"} | +------------------------------------------------------------------------+--------------------------------+----------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+ | [HTS-AT]{style="color: darkgray"} [@chen2022hts] | [31M]{style="color: darkgray"} | [IN]{style="color: darkgray"} | [47.1]{style="color: darkgray"} | [-]{style="color: darkgray"} | [-]{style="color: darkgray"} | [-]{style="color: darkgray"} | [98.0]{style="color: darkgray"} | [-]{style="color: darkgray"} | +------------------------------------------------------------------------+--------------------------------+----------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+ | [Audio-CLIP]{style="color: darkgray"} [@guzhov2022audioclip] | [93M]{style="color: darkgray"} | [TI+AS]{style="color: darkgray"} | [25.9]{style="color: darkgray"} | [-]{style="color: darkgray"} | [96.7]{style="color: darkgray"} | [-]{style="color: darkgray"} | [-]{style="color: darkgray"} | [-]{style="color: darkgray"} | +------------------------------------------------------------------------+--------------------------------+----------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+ | ```{=latex} | | | | | | | | | | \addlinespace | | | | | | | | | | ``` | | | | | | | | | | ```{=latex} | | | | | | | | | | \hline | | | | | | | | | | ``` | | | | | | | | | | ```{=latex} | | | | | | | | | | \addlinespace | | | | | | | | | | ``` | | | | | | | | | | ***In-domain Supervised Pre-training*** | | | | | | | | | +------------------------------------------------------------------------+--------------------------------+----------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+ | ```{=latex} | [86M]{style="color: darkgray"} | [IN+AS]{style="color: darkgray"} | [45.9]{style="color: darkgray"} | [-]{style="color: darkgray"} | [95.6]{style="color: darkgray"} | [-]{style="color: darkgray"} | [97.9]{style="color: darkgray"} | [-]{style="color: darkgray"} | | \addlinespace | | | | | | | | | | ``` | | | | | | | | | | [AST]{style="color: darkgray"} [@gong2021ast] | | | | | | | | | +------------------------------------------------------------------------+--------------------------------+----------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+ | [HTS-AT]{style="color: darkgray"} [@chen2022hts] | [31M]{style="color: darkgray"} | [IN+AS]{style="color: darkgray"} | [47.1]{style="color: darkgray"} | [-]{style="color: darkgray"} | [97.0]{style="color: darkgray"} | [-]{style="color: darkgray"} | [-]{style="color: darkgray"} | [-]{style="color: darkgray"} | +------------------------------------------------------------------------+--------------------------------+----------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+ | [Audio-MAE]{style="color: darkgray"} [@huang2022masked] | [86M]{style="color: darkgray"} | [AS]{style="color: darkgray"} | [-]{style="color: darkgray"} | [-]{style="color: darkgray"} | [97.4]{style="color: darkgray"} | [-]{style="color: darkgray"} | [-]{style="color: darkgray"} | [-]{style="color: darkgray"} | +------------------------------------------------------------------------+--------------------------------+----------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+ | ```{=latex} | | | | | | | | | | \addlinespace | | | | | | | | | | ``` | | | | | | | | | | ```{=latex} | | | | | | | | | | \hline | | | | | | | | | | ``` | | | | | | | | | | ```{=latex} | | | | | | | | | | \addlinespace | | | | | | | | | | ``` | | | | | | | | | | ***Self-Supervised Pre-training*** | | | | | | | | | +------------------------------------------------------------------------+--------------------------------+----------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+ | ```{=latex} | 89M | AS+LS | \- | 31.0 | 88.8 | 96.0 | 98.0 | 59.6 | | \addlinespace | | | | | | | | | | ``` | | | | | | | | | | SS-AST [@gong2022ssast] | | | | | | | | | +------------------------------------------------------------------------+--------------------------------+----------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+ | MAE-AST [@baade2022mae] | 86M | AS+LS | \- | 30.6 | 90.0 | 95.8 | 97.9 | 59.8 | +------------------------------------------------------------------------+--------------------------------+----------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+ | CAV-MAE [@gong2022contrastive] | 86M | IN+AS | 44.9 | 34.2 | \- | \- | \- | \- | +------------------------------------------------------------------------+--------------------------------+----------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+ | Audio-MAE [@huang2022masked] | 86M | AS | 47.3 | 37.1 | 94.1 | 96.9 | 98.3 | \- | +------------------------------------------------------------------------+--------------------------------+----------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+ | Audio-MAE L [@huang2022masked] | 304M | AS | 47.4 | 37.7 | \- | \- | \- | \- | +------------------------------------------------------------------------+--------------------------------+----------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+ | data2vec [@baevski2022data2vec] | 94M | AS | \- | 34.5 | \- | \- | \- | \- | +------------------------------------------------------------------------+--------------------------------+----------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+ | MaskSpec [@chong2023masked] | 86M | AS | 47.1 | 32.3 | 89.6 | \- | 97.7 | \- | +------------------------------------------------------------------------+--------------------------------+----------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+ | BEATs$_{iter3}$ [@chen2022beats] | 90M | AS | 48.0 | 38.3 | 95.6 | 97.7 | 98.3 | 64.5 | +------------------------------------------------------------------------+--------------------------------+----------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+ | A-JEPA [@fei2023jepa] | 86M | AS | 48.6 | 38.4 | **96.3** | 97.7 | 98.5 | \- | +------------------------------------------------------------------------+--------------------------------+----------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+ | ASiT [@ahmed2024asit] | 86M | AS | 48.0 | 38.6 | 95.3 | [98.2]{.underline} | **98.9** | \- | +------------------------------------------------------------------------+--------------------------------+----------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+ | EAT [@chen2024eat] | 88M | AS | 48.6 | 40.2 | 95.9 | \- | 98.3 | \- | +------------------------------------------------------------------------+--------------------------------+----------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+ | SSLAM [@alex2025sslam] | 88M | AS | **50.2** | **40.9** | [96.2]{.underline} | **98.8** | 98.1 | \- | +------------------------------------------------------------------------+--------------------------------+----------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+ | SPEAR$_a$ Large [@yang2025spear] | 327M | AS | 49.7 | 39.3 | \- | \- | \- | \- | +------------------------------------------------------------------------+--------------------------------+----------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+ | ```{=latex} | 85M | AS | 49.6 | 39.1 | 94.2 | 97.9 | [98.8]{.underline} | 64.9 | | \addlinespace | | | | | | | | | | ``` | | | | | | | | | | ```{=latex} | | | | | | | | | | \hline | | | | | | | | | | ``` | | | | | | | | | | ```{=latex} | | | | | | | | | | \addlinespace | | | | | | | | | | ``` | | | | | | | | | | **[NAPE-B raster]{style="color: nepapink"}** | | | | | | | | | +------------------------------------------------------------------------+--------------------------------+----------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+ | **[NAPE-B diagonal]{style="color: nepadiagonal"}** | 85M | AS | 49.7 | 39.2 | 94.8 | 97.9 | 98.6 | 67.1 | +------------------------------------------------------------------------+--------------------------------+----------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+ | **[NAPE-L raster]{style="color: nepapink"}** | 303M | AS | **50.2** | [40.5]{.underline} | 96.0 | 97.9 | [98.8]{.underline} | [68.0]{.underline} | +------------------------------------------------------------------------+--------------------------------+----------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+ | **[NAPE-L diagonal]{style="color: nepadiagonal"}** | 303M | AS | [50.0]{.underline} | 40.4 | [96.2]{.underline} | [98.2]{.underline} | **98.9** | **68.8** | +------------------------------------------------------------------------+--------------------------------+----------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+ : Comparison with audio methods on audio and speech downstream tasks. IN, AS, and LS denote the ImageNet, AudioSet, and LibriSpeech datasets, respectively. TI denotes the 400M text-image pairs for CLIP pre-training. We [gray-out]{style="color: darkgray"} the models and results with additional supervised training on external datasets. Best results are in **bold**, second-best are [underlined]{.underline}. `\label{tab:main_table}`{=latex} ## Comparison with State-of-the-Art {#sec:sota} We compare NAPE against prior audio pre-training methods in Table `\ref{tab:main_table}`{=latex}. We use the best configuration identified in Section `\ref{sec:ablations}`{=latex}, reporting both raster and diagonal variants at the Base and Large scales. NAPE-B with diagonal scan delivers strong performance across all six benchmarks, matching or exceeding every self-supervised competitor of comparable size. When we scale it further, NAPE-L raster attains results on par with the strongest baseline, SSLAM [@alex2025sslam], tying it on AS-2M ($50.2$ mAP) and approaching it closely on AS-20K ($40.5$ vs $40.9$ mAP) and ESC-50 ($96.0$ vs $96.2$%). What is notable about these results is the simplicity of underlying NAPE's recipe. Unlike SSLAM, which relies on audio-mixture supervision, a student-teacher architecture, and a reconstruction decoder, NAPE requires none of these ingredients: its pre-training objective consists of a stop-gradient and a negative cosine similarity between the predicted next patch embedding and its target. NAPE also outperforms A-JEPA [@fei2023jepa] on $5$ benchmarks despite A-JEPA relying on an auxiliary target encoder updated via EMA, predicting multiple masked patches in parallel rather than causally, and adopting a regularized masking strategy during fine-tuning. Finally, NAPE transfers particularly well to speech tasks: on IEMOCAP, NAPE-L reaches $68.0$% accuracy, a $+3.5$-point improvement over the strongest baseline result reported at any scale (BEATs$_{iter3}$ at $64.5$%), suggesting that NAPE learns representations that generalize well beyond acoustic-event classification. ## Linear Probing Results {#sec:linear-probe} #### `Which Layer to Probe?`{=latex} Linear probing measures the linear separability of the features produced by the pretrained encoder, treating the encoder as a fixed feature extractor and training only a linear classification head. Following recent observations that the most classification-relevant features in deep Transformer models often lie in intermediate rather than final layers [@skean2025layer; @bolya2026perception], we begin our linear probing study by measuring downstream performance as a function of encoder depth. ```{=latex} \begin{wrapfigure}{r}{0.45\textwidth} \includegraphics[width=\linewidth]{Figures/layerwise_as20k.pdf} \caption{Layer-wise linear probing analysis.} \label{fig:linearprobing_plot} \end{wrapfigure} ``` Figure `\ref{fig:linearprobing_plot}`{=latex} reports AS-20K mAP obtained by linearly probing each layer of NAPE-S, NAPE-B, and NAPE-L, using the best raster configuration identified in Section `\ref{sec:ablations}`{=latex}. Across all three scales, the best probing layer lies at roughly the middle of the encoder: layer $2$ for NAPE-S, layer $6$ for NAPE-B, and layer $11$ for NAPE-LL. Beyond this mid-network optimum, performance declines steadily toward the final layer, dropping by roughly $3$-$5$ mAP points. This pattern is consistent with the interpretation that the top layers of a NAPE-pretrained encoder specialize for the next-patch-embedding prediction objective, while the mid-layers retain more general and classification-relevant information. ```{=latex} \begin{wraptable}{r}{0.37\textwidth} \small \renewcommand{\tabcolsep}{0.5mm} \caption{Linear probing results.} %\resizebox{0.999\linewidth}{!}{ \begin{tabular}{lcccc} \toprule \textbf{Model} & \textbf{Layer} & \emph{AS-2M} & \emph{AS-20K} & \emph{ESC-50}\\ \midrule Small & 2nd & 23.2 & 18.9 & 79.8 \\ Base & 6th& 25.0 & 19.7 & 81.7 \\ \textbf{Large} & 11th &\textbf{27.1} &\textbf{20.4} & \textbf{83.5} \\ \bottomrule \end{tabular}%} \label{tab:linearprobing_table} \end{wraptable} ``` #### `Linear Probing Across Scales and Tasks.`{=latex} We then use the best probing layer identified per model to compare NAPE-S, NAPE-B, and NAPE-L on AS-2M, AS-20K, and ESC-50. Table `\ref{tab:linearprobing_table}`{=latex} reports the results. NAPE scales positively under linear probing: larger models yield stronger probes on every task, from AS-2M ($+3.9$ mAP from small to large) to ESC-50 ($+3.7$ in accuracy). This trend reinforces the scaling behavior observed under fine-tuning (Section `\ref{sec:scaling}`{=latex}). At the same time, the absolute linear-probing numbers are noticeably lower than their fine-tuning counterparts. This gap is expected for prediction-based self-supervised methods: the pretraining objective encourages the encoder to learn features that support the next-patch prediction task, which do not necessarily align with the linear separability needed for classification. ## Qualitative Results {#sec:qualitative} To gain insight into what NAPE learns, we complement the quantitative benchmarks with two qualitative analyses of a pretrained NAPE-L (raster) on the AudioSet evaluation set. Both use the model after pretraining, with no fine-tuning. More qualitative results can be found in the Appendix `\ref{sec:additional_qualitative}`{=latex}. #### `Prediction Fidelity.`{=latex} We measure the cosine similarity between the predicted embeddings $\hat{z}_{t+1}$ and the true patch embeddings $z_{t+1}$. Figure `\ref{fig:qualitative}`{=latex} (top left) shows the similarity averaged across $500$ held-out AudioSet clips, and the top middle and right panels report the same measurement for two individual clips: NAPE predicts the next patch embedding accurately almost everywhere on the grid, with similarity close to the ceiling of $1.0$ both on average and per clip. The remaining low-similarity regions have clear structural explanations. The very first patch has the lowest similarity, since no previous context is available for the prediction to condition on. The first mel row is harder on average, since it combines limited past context with low-frequency patches. Finally, the rightmost patches show slightly lower similarity because the last time columns correspond to zero-padded frames appended to reach the target clip length. Away from these boundary regions, NAPE satisfies its pre-training objective on unseen audio.
Top: We report the cosine similarity between the predicted and true patch embeddings, averaged over 500 audio clips (left) and for two individual clips (middle, right). Bottom: Left: selected query patch; middle: attention map showing the patches NAPE attends to when predicting the next patch; right: embedding-similarity map between the predicted next-patch embedding ẑt + 1 and every actual patch embedding in the spectrogram.
#### `Attention and Embedding Analysis.`{=latex} To understand *how* NAPE arrives at its predictions, we select a query patch (marked in red in Figure `\ref{fig:qualitative}`{=latex}, bottom left) and analyze the attention and embedding structure it induces. The attention map (bottom middle), which conveys the aggregated attention from the query position to every other patch averaged over all layers and heads, reveals a highly structured pattern with two distinct components: NAPE attends strongly to the current time column, integrating the full spectral profile of the current moment, and to same-mel-frequency patches earlier in the clip, tracking the temporal evolution of the frequency it is about to predict. The embedding-similarity map (bottom right), which compares the predicted embedding $\hat{z}_{t+1}$ against every actual patch embedding in the clip, shows that $\hat{z}_{t+1}$ is most similar to patches that share acoustic structure and energy with the query, regions of the spectrogram carrying comparable spectral content, with similarity gradually decaying at the temporal and spectral extremes. This grouping into coherent acoustic components emerges without any explicit labels or region annotations, suggesting that despite being trained only on a local next-patch objective, NAPE develops representations that capture the broader acoustic structure of the clip. # Conclusion {#sec:conclusion} We presented NAPE, a self-supervised framework for audio representation learning based on causal next patch embedding prediction. Departing from the reconstruction- and masking-based approaches that dominate audio SSL, NAPE relies on a deliberately minimalist recipe: a single causal Transformer encoder, a lightweight predictor head, and a negative cosine similarity loss with stop-gradient on the target branch. Across six audio and speech benchmarks, NAPE achieves strong downstream performance while relying on a substantially simpler pre-training recipe. NAPE also exhibits favorable scaling behavior across three encoder sizes and delivers competitive linear-probing results. A qualitative analysis further shows that the model develops structured attention patterns and organizes its learned embeddings into acoustically coherent regions without any explicit supervision. Together, these results establish autoregressive next-embedding prediction as a simple, scalable, and effective self-supervised objective for audio, and open a direct path to bringing the causal predictive paradigm into audio representation learning. ### Acknowledgments {#acknowledgments .unnumbered} We thank Andrew Rouditchenko (Nvidia) for his insightful and valuable discussions. # References
Achiam, Josh, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, et al. 2023. “Gpt-4 Technical Report.” *arXiv Preprint arXiv:2303.08774*.
Ahmed, Sara Atito Ali, Muhammad Awais, Wenwu Wang, Mark D Plumbley, and Josef Kittler. 2024. “Asit: Local-Global Audio Spectrogram Vision Transformer for Event Classification.” *IEEE/ACM Transactions on Audio, Speech, and Language Processing* 32: 3684–93.
Alex, Tony, Sara Atito, Armin Mustafa, Muhammad Awais, and Philip Jackson. 2025. “Sslam: Enhancing Self-Supervised Models with Audio Mixtures for Polyphonic Soundscapes.” In *International Conference on Learning Representations*, 2025:22608–26.
Assran, Mahmoud, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. 2023. “Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture.” In *2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 15619–29. IEEE.
Ba, Jimmy Lei, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. “Layer Normalization.” *arXiv Preprint arXiv:1607.06450*.
Baade, Alan, Puyuan Peng, and David Harwath. 2022. “Mae-Ast: Masked Autoencoding Audio Spectrogram Transformer.” *arXiv Preprint arXiv:2203.16691*.
Baevski, Alexei, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. 2022. “Data2vec: A General Framework for Self-Supervised Learning in Speech, Vision and Language.” In *International Conference on Machine Learning*, 1298–1312. PMLR.
Baevski, Alexei, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. “Wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations.” *Advances in Neural Information Processing Systems* 33: 12449–60.
Balestriero, Randall, and Yann LeCun. 2025. “Lejepa: Provable and Scalable Self-Supervised Learning Without the Heuristics.” *arXiv Preprint arXiv:2511.08544*.
Bao, Hangbo, Li Dong, Songhao Piao, and Furu Wei. 2021. “Beit: Bert Pre-Training of Image Transformers.” *arXiv Preprint arXiv:2106.08254*.
Bardes, Adrien, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mahmoud Assran, and Nicolas Ballas. 2024. “Revisiting Feature Prediction for Learning Visual Representations from Video.” *arXiv Preprint arXiv:2404.08471*.
Bardes, Adrien, Jean Ponce, and Yann LeCun. 2021. “Vicreg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning.” *arXiv Preprint arXiv:2105.04906*.
Bolya, Daniel, Po-Yao Huang, Peize Sun, Jang Hyun Cho, Andrea Madotto, Chen Wei, Tengyu Ma, et al. 2026. “Perception Encoder: The Best Visual Embeddings Are Not at the Output of the Network.” *Advances in Neural Information Processing Systems* 38: 60884–937.
Bredis, George, Nikita Balagansky, Daniil Gavrilov, and Ruslan Rakhimov. 2026. “Next Embedding Prediction Makes World Models Stronger.” *arXiv Preprint arXiv:2603.02765*.
Brown, Tom, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, et al. 2020. “Language Models Are Few-Shot Learners.” *Advances in Neural Information Processing Systems* 33: 1877–1901.
Busso, Carlos, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008. “IEMOCAP: Interactive Emotional Dyadic Motion Capture Database.” *Language Resources and Evaluation* 42 (4): 335–59.
Chen, Ke, Xingjian Du, Bilei Zhu, Zejun Ma, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. 2022. “HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection.” In *ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, 646–50. IEEE.
Chen, Mark, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. 2020. “Generative Pretraining from Pixels.” In *International Conference on Machine Learning*, 1691–1703. PMLR.
Chen, Sanyuan, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, and Furu Wei. 2022. “Beats: Audio Pre-Training with Acoustic Tokenizers.” *arXiv Preprint arXiv:2212.09058*.
Chen, Wenxi, Yuzhe Liang, Ziyang Ma, Zhisheng Zheng, and Xie Chen. 2024. “EAT: Self-Supervised Pre-Training with Efficient Audio Transformer.” *arXiv Preprint arXiv:2401.03497*.
Chen, Xinlei, and Kaiming He. 2021. “Exploring Simple Siamese Representation Learning.” In *2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 15745–53. IEEE.
Chong, Dading, Helin Wang, Peilin Zhou, and Qingcheng Zeng. 2023. “Masked Spectrogram Prediction for Self-Supervised Audio Pre-Training.” In *ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, 1–5. IEEE.
Clark, Kevin, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020. “Electra: Pre-Training Text Encoders as Discriminators Rather Than Generators.” *arXiv Preprint arXiv:2003.10555*.
Deng, Jia, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. “Imagenet: A Large-Scale Hierarchical Image Database.” In *2009 IEEE Conference on Computer Vision and Pattern Recognition*, 248–55. Ieee.
Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. “Bert: Pre-Training of Deep Bidirectional Transformers for Language Understanding.” In *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)*, 4171–86.
Dinkel, Heinrich, Zhiyong Yan, Yongqing Wang, Junbo Zhang, Yujun Wang, and Bin Wang. 2024. “Scaling up Masked Audio Encoder Learning for General Audio Classification.” *arXiv Preprint arXiv:2406.06992*.
Dosovitskiy, Alexey, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, et al. 2020. “An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale.” *arXiv Preprint arXiv:2010.11929*.
Elizalde, Benjamin, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. 2023. “Clap Learning Audio Concepts from Natural Language Supervision.” In *ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, 1–5. IEEE.
Fei, Zhengcong, Mingyuan Fan, and Junshi Huang. 2023. “A-Jepa: Joint-Embedding Predictive Architecture Can Listen.” *arXiv Preprint arXiv:2311.15830*.
Gemmeke, Jort F, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. 2017. “Audio Set: An Ontology and Human-Labeled Dataset for Audio Events.” In *2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, 776–80. IEEE.
Gong, Yuan, Yu-An Chung, and James Glass. 2021a. “Ast: Audio Spectrogram Transformer.” In *INTERSPEECH*.
———. 2021b. “Psla: Improving Audio Tagging with Pretraining, Sampling, Labeling, and Aggregation.” *IEEE/ACM Transactions on Audio, Speech, and Language Processing* 29: 3292–3306.
Gong, Yuan, Cheng-I Lai, Yu-An Chung, and James Glass. 2022. “Ssast: Self-Supervised Audio Spectrogram Transformer.” In *Proceedings of the AAAI Conference on Artificial Intelligence*, 36:10699–709. 10.
Gong, Yuan, Andrew Rouditchenko, Alexander H Liu, David Harwath, Leonid Karlinsky, Hilde Kuehne, and James Glass. 2022. “Contrastive Audio-Visual Masked Autoencoder.” *arXiv Preprint arXiv:2210.07839*.
Guzhov, Andrey, Federico Raue, Jörn Hees, and Andreas Dengel. 2022. “Audioclip: Extending Clip to Image, Text and Audio.” In *ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, 976–80. IEEE.
He, Kaiming, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. 2022. “Masked Autoencoders Are Scalable Vision Learners.” In *2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 15979–88. IEEE.
Henry, Alex, Prudhvi Raj Dachapally, Shubham Shantaram Pawar, and Yuxuan Chen. 2020. “Query-Key Normalization for Transformers.” In *Findings of the Association for Computational Linguistics: EMNLP 2020*, 4246–53.
Huang, Chen, Xianhang Li, Vimal Thilak, Etai Littwin, and Josh Susskind. 2026. “Text-Conditional JEPA for Learning Semantically Rich Visual Representations.” *arXiv Preprint arXiv:2605.03245*.
Huang, Gao, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. 2016. “Deep Networks with Stochastic Depth.” In *European Conference on Computer Vision*, 646–61. Springer.
Huang, Po-Yao, Hu Xu, Juncheng Li, Alexei Baevski, Michael Auli, Wojciech Galuba, Florian Metze, and Christoph Feichtenhofer. 2022. “Masked Autoencoders That Listen.” *Advances in Neural Information Processing Systems* 35: 28708–20.
Kong, Zhifeng, Arushi Goel, Rohan Badlani, Wei Ping, Rafael Valle, and Bryan Catanzaro. 2024. “Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities.” *arXiv Preprint arXiv:2402.01831*.
Liu, Aixin, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, et al. 2024. “Deepseek-V3 Technical Report.” *arXiv Preprint arXiv:2412.19437*.
Loshchilov, Ilya, and Frank Hutter. 2017. “Decoupled Weight Decay Regularization.” *arXiv Preprint arXiv:1711.05101*.
Maes, Lucas, Quentin Le Lidec, Luiz Facury, Nassim Massaudi, Ayush Chaurasia, Francesco Capuano, Richard Gao, et al. 2026. “Stable-Worldmodel: A Platform for Reproducible World Modeling Research and Evaluation.” *arXiv Preprint arXiv:2605.21800*.
Maes, Lucas, Quentin Le Lidec, Damien Scieur, Yann LeCun, and Randall Balestriero. 2026. “Leworldmodel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels.” *arXiv Preprint arXiv:2603.19312*.
Niizumi, Daisuke, Daiki Takeuchi, Yasunori Ohishi, Noboru Harada, and Kunio Kashino. 2022. “Masked Spectrogram Modeling Using Masked Autoencoders for Learning General-Purpose Audio Representation.” In *HEAR: Holistic Evaluation of Audio Representations*, 1–24. PMLR.
Niizumi, Daisuke, Daiki Takeuchi, Masahiro Yasuda, Binh Thien Nguyen, Noboru Harada, and Nobutaka Ono. 2026. “Rethinking Masking Strategies for Masked Prediction-Based Audio Self-Supervised Learning.” *arXiv Preprint arXiv:2603.23810*.
Oquab, Maxime, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, et al. 2023. “Dinov2: Learning Robust Visual Features Without Supervision.” *arXiv Preprint arXiv:2304.07193*.
Park, Daniel S, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le. 2019. “Specaugment: A Simple Data Augmentation Method for Automatic Speech Recognition.” *arXiv Preprint arXiv:1904.08779*.
Piczak, Karol J. 2015. “ESC: Dataset for Environmental Sound Classification.” In *Proceedings of the 23rd ACM International Conference on Multimedia*, 1015–18.
Polyak, Boris T, and Anatoli B Juditsky. 1992. “Acceleration of Stochastic Approximation by Averaging.” *SIAM Journal on Control and Optimization* 30 (4): 838–55.
Radford, Alec, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, et al. 2021. “Learning Transferable Visual Models from Natural Language Supervision.” In *International Conference on Machine Learning*, 8748–63. PmLR.
Radford, Alec, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. “Improving Language Understanding by Generative Pre-Training.”
Skean, Oscar, Md Rifat Arefin, Dan Zhao, Niket Patel, Jalal Naghiyev, Yann LeCun, and Ravid Shwartz-Ziv. 2025. “Layer by Layer: Uncovering Hidden Representations in Language Models.” *arXiv Preprint arXiv:2502.02013*.
Su, Jianlin, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. “Roformer: Enhanced Transformer with Rotary Position Embedding.” *Neurocomputing* 568: 127063.
Szegedy, Christian, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. “Rethinking the Inception Architecture for Computer Vision.” In *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*, 2818–26.
Team, Gemma, Sherif El Abd, Vaibhav Aggarwal, Robin Algayres, Alek Andreev, Olivier Bachem, Ian Ballantyne, et al. 2026. “Gemma 4 Technical Report.” *arXiv Preprint arXiv:2607.02770*.
Teoh, Jayden, Manan Tomar, Kwangjun Ahn, Edward S Hu, Tim Pearce, Pratyusha Sharma, Akshay Krishnamurthy, Riashat Islam, Alex Lamb, and John Langford. 2025. “Next-Latent Prediction Transformers Learn Compact World Models.” *arXiv Preprint arXiv:2511.05963*.
Touvron, Hugo, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. 2021. “Going Deeper with Image Transformers.” In *2021 IEEE/CVF International Conference on Computer Vision (ICCV)*, 32–42. IEEE.
Tuncay, Ludovic, Etienne Labbé, Emmanouil Benetos, and Thomas Pellegrini. 2025. “Audio-Jepa: Joint-Embedding Predictive Architecture for Audio Representation Learning.” *arXiv Preprint arXiv:2507.02915*.
Wang, Feng, Sucheng Ren, Tiezheng Zhang, Predrag Neskovic, Anand Bhattad, Cihang Xie, and Alan Yuille. 2026. “Vit-5: Vision Transformers for the Mid-2020s.” *arXiv Preprint arXiv:2602.08071*.
Warden, Pete. 2018. “Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition.” *arXiv Preprint arXiv:1804.03209*.
Wu, Haiyu, Randall Balestriero, and Morgan Levine. 2026. “VISReg: Variance-Invariance-Sketching Regularization for JEPA Training.” *arXiv Preprint arXiv:2606.02572*.
Wu, Ho-Hsiang, Prem Seetharaman, Kundan Kumar, and Juan Pablo Bello. 2022. “Wav2clip: Learning Robust Audio Representations from Clip.” In *ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, 4563–67. IEEE.
Xiao, Tete, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Dollár, and Ross Girshick. 2021. “Early Convolutions Help Transformers See Better.” *Advances in Neural Information Processing Systems* 34: 30392–400.
Xu, Sihan, Ziqiao Ma, Wenhao Chai, Xuweiyi Chen, Weiyang Jin, Joyce Chai, Saining Xie, and Stella X Yu. 2025. “Next-Embedding Prediction Makes Strong Vision Learners.” *arXiv Preprint arXiv:2512.16922*.
Yang, An, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, et al. 2025. “Qwen3 Technical Report.” *arXiv Preprint arXiv:2505.09388*.
Yang, Xiaoyu, Yifan Yang, Zengrui Jin, Ziyun Cui, Wen Wu, Baoxiang Li, Chao Zhang, and Phil Woodland. 2025. “Spear: A Unified Ssl Framework for Learning Speech and Audio Representations.” *arXiv Preprint arXiv:2510.25955*.
Yao, Yumeng, Jingzhi Dong, Haowen Gu, Tao Chen, Zonghan Wu, Xiaoshui Huang, and Yazhou Yao. 2026. “Rethinking Point Clouds as Sequences: A Causal Next-Token Predictive Learning Framework.” *arXiv Preprint arXiv:2605.17566*.
Yuksel, Goksenin, Pierre Guetschel, Michael Tangermann, Marcel van Gerven, and Kiki van der Heijden. 2025. “WavJEPA: Semantic Learning Unlocks Robust Audio Foundation Models for Raw Waveforms.” *arXiv Preprint arXiv:2509.23238*.
Yun, Sangdoo, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. 2019. “Cutmix: Regularization Strategy to Train Strong Classifiers with Localizable Features.” In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, 6023–32.
Zhang, Biao, and Rico Sennrich. 2019. “Root Mean Square Layer Normalization.” *Advances in Neural Information Processing Systems* 32.
Zhang, Hongyi, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. 2017. “Mixup: Beyond Empirical Risk Minimization.” *arXiv Preprint arXiv:1710.09412*.
# Appendix {#sec:appendix} # Related Work {#sec:relatedwork} #### Supervised Audio Pre-training. Supervised pre-training for audio has been mainly explored in two regimes. *Out-of-domain* approaches adapt models originally trained on labeled image datasets such as ImageNet [@deng2009imagenet] to spectrogram inputs, typically by modifying the input layer from three RGB channels to a single-channel spectrogram. Early work of this kind used CNN backbones such as EfficientNet [@gong2021psla]; more recent work adopts Transformer-based architectures, notably AST [@gong2021ast] and HTS-AT [@chen2022hts], which have driven substantial gains on audio classification benchmarks. *In-domain* approaches instead pre-train directly on the target modality. CLAP [@elizalde2023clap] adapts the CLIP [@radford2021learning] recipe to audio via contrastive language-audio alignment on supervised text-audio pairs, while Audio-CLIP [@guzhov2022audioclip] and Wav2clip [@wu2022wav2clip] extends the CLIP backbone with an additional audio encoder trained on AudioSet [@gemmeke2017audio]. Despite delivering strong results, these methods rely on large quantities of labeled data, which are expensive and time-consuming to obtain in practice. #### Self-Supervised Audio Pre-training. Self-supervised learning has driven substantial progress in audio representation learning, largely by transferring ideas developed in the vision domain. Most recent methods extract log-mel spectrograms as input and follow the masked image modeling paradigm [@he2022masked]: Audio-MAE [@huang2022masked], MaskSpec [@chong2023masked] and MSM-MAE [@niizumi2022masked] directly applie the MAE reconstruction objective to spectrogram patches; and BEATs [@chen2022beats] instead trains an iterative acoustic tokenizer to provide discrete semantic prediction targets. Waveform-based approaches such as wav2vec 2.0 [@baevski2020wav2vec] and data2vec [@baevski2022data2vec] bypass spectrogram preprocessing altogether, and predict latent representations of the raw audio signal. More recent efforts push these directions further by combining ideas from multiple objectives. EAT [@chen2024eat] combines an MAE-style reconstruction with the data2vec latent-target formulation. A-JEPA [@fei2023jepa] adapts the joint-embedding predictive paradigm to audio, using a target encoder updated via EMA and predicting masked patch embeddings. SSLAM [@alex2025sslam] extends EAT with an additional source retention loss trained on artificially mixed audio to improve robustness to polyphonic content. Recently, SPEAR [@yang2025spear] unifies speech and general audio representation learning through the distillation of complementary knowledge from specialized teacher models. Despite their diversity, these methods all rely on bidirectional prediction of masked or corrupted audio, and typically require a reconstruction decoder, a student-teacher setup, or auxiliary regularization losses to achieve competitive performance. #### Next-token/embedding Prediction. Predictive learning has long been a central principle in representation learning across modalities. In language, GPT-style models [@radford2018improving; @radford2021learning; @achiam2023gpt] established autoregressive next-token prediction as a scalable pre-training objective, with subsequent work confirming that the same paradigm transfers to vision (Image-GPT [@chen2020generative]) and beyond. More recent efforts move away from predicting raw signal tokens toward predicting embeddings directly. A first line of work adopts an *autoregressive* formulation: NEPA [@xu2025nepa] introduces next-embedding predictive autoregression for visual representation learning; PointNTP [@yao2026rethinking] adapts causal next-token predictive learning to 3D point clouds; and LeWorldModel [@maes2026leworldmodel] together with stable-worldmodel [@maes2026stable] extend the paradigm to next-frame prediction for control tasks. A second line of work adopts a *joint-embedding* formulation: I-JEPA [@assran2023self] and A-JEPA [@fei2023jepa] predict latent representations of masked regions from context using dual encoders and an EMA teacher, while LeJEPA [@balestriero2025lejepa] dispenses with masking and instead enforces invariance across augmented views with a variance-covariance regularizer. Together, these works suggest that predicting future tokens or embeddings can serve as a unified and scalable pre-training principle across modalities. In audio, however, this paradigm has remained largely unexplored: existing SSL methods rely almost exclusively on bidirectional masked modeling, and it is unclear *a priori* whether causal, per-position prediction is compatible with the non-stationary, temporally structured nature of audio signals. NAPE is, to the best of our knowledge, the first method to demonstrate that causal next-embedding prediction is a competitive and scalable pre-training objective for audio spectrograms, matching or surpassing more elaborate masked-modeling and joint-embedding methods with a substantially simpler recipe. # Full Hyperparameter List {#sec:hyperparameters} ```{=latex} \small ``` ```{=latex} \renewcommand{\tabcolsep}{0.4mm} ``` -------------------------------------- -------------------------------------- ------------------------------------- ----------------- ----------------- ----------------------- ----------------- ----------------- **Hyperparameters** `\cellcolor`{=latex}**Pre-training** `\cellcolor`{=latex}**Fine-tuning** *AS-2M* *AS-2M* *AS-20K* *ESC-50* *KS1* *KS2* *ER* Optimizer AdamW Opt. Momentum ($\beta_1$, $\beta_2$) (0.9,0.95) (0.9,0.999) Weight Decay 0.05 Learning Rate Scheduler Cosine Decay Layer-Wise LR Decay 1.0 0.7/0.9 0.8/0.9 0.7/0.9 0.7/0.8 0.7/0.8 0.7/0.9 Base Learning Rate 5e-3 1.25e-3 Epochs 30/25 20/15 30/20 100 50 50 50 Warm-up Epochs 3 4/3 6/5 10 5 5 5 Batch Size 256/128 64 GPUs 8 4 4 1 1 1 4 Weighted sampling `\ding`{=latex} `\ding`{=latex} `\ding`{=latex} `\ding`{=latex} `\ding`{=latex}$\ast$ `\ding`{=latex} `\ding`{=latex} Multilabel N/A `\ding`{=latex} `\ding`{=latex} `\ding`{=latex} `\ding`{=latex} `\ding`{=latex} `\ding`{=latex} EMA Decay Rate 0.9999 0.99995 0.999 `\ding`{=latex} `\ding`{=latex} `\ding`{=latex} `\ding`{=latex} Label Smoothing N/A 0\. 0\. 0.1 0.1 0\. 0.1 Roll Augmentation `\ding`{=latex} `\ding`{=latex} `\ding`{=latex} `\ding`{=latex} `\ding`{=latex} `\ding`{=latex} `\ding`{=latex} Drop Path 0\. 0.1 SpecAug (time/freq) N/A (96,16) (24,16)/(96,16) (96,24)/(24,16) (24,16) (24,16) (48,24) Mixup (alpha/prob.) N/A (0.8,1.0) (0.8,0.8) (0.8,0.5) (0.8,0.8) (0.8,0.8) (0.8,0.5) Cutmix (alpha/prob.) N/A (1.0,1.0) (1.0,0.8) (1.0,0.5) (1.0,0.8) (1.0,0.8) (1.0,0.5) Noise Augmentation `\ding`{=latex} `\ding`{=latex} `\ding`{=latex} `\ding`{=latex} `\ding`{=latex} `\ding`{=latex} `\ding`{=latex} Loss Function Neg Cos Sim BCE BCE CE BCE BCE CE Dataset Mean for Norm. -6.84 -6.84 -6.84 -6.84 -9.11 -9.16 -13.74 Dataset Std for Norm. 5.38 5.38 5.38 5.38 4.53 4.61 3.88 -------------------------------------- -------------------------------------- ------------------------------------- ----------------- ----------------- ----------------------- ----------------- ----------------- : **Hyperparameter list**. When an hyperparameter $h$ varies between the base ($h_b$) and large ($h_l$) model, we include both values like ($h_b$/$h_l$). $^\ast$Following [@chen2022beats], we balance each class to $50$% of the size of the unknown class for each training epoch. `\label{tab:hyperparameters}`{=latex} #### Fine-tuning Setting. We report the full hyperparameter list for fine-tuning NEPA Base and Large in Table `\ref{tab:hyperparameters}`{=latex}. Those values refer to the Raster scan variant, with the Diagonal variant having almost the same hyperparameters (the only difference is in the number of epochs needed to converge for *AS-20K*, which can vary of only a few epochs). Regarding NEPA Small, we used the same hyperparameters as NEPA Base, with the only difference being the value of the layer-wise learning rate decay ($LLRD$) for some of the downstream tasks as this hyperparameter depends on the number of layers of the model. Specifically, we set $LLRD = 0.6$ for *AS-2M* and $LLRD = 0.6$ for *KS1*/*KS2*. No other changes have been made. #### Linear Probing Setting. For the linear probing experiments, we made the following changes (all other settings are the same as the fine-tuning hyperparameters): we increase the learning rate to $1e-2$ and we disable all augmentation techniques. # Additional Ablation Studies {#sec:additional_ablations} In this section, we report additional ablations studies on NAPE. For all experiments we use the raster scan variant. We use the SimSiam predictor for all ablations except for the ablation on the pre-training budget where we use the 2-MLP predictor. #### `Freezing the Patch Embedding Layer.`{=latex} Table `\ref{tab:patchembfreeze}`{=latex} examines whether freezing the patch embedding layer during fine-tuning affects downstream performance. In [@xu2025nepa], freezing the embedding layer yields significant improvements. However, in our setting, the two configurations are essentially indistinguishable on all three tasks tested. We therefore leave the patch embedding layer trainable during fine-tuning, matching the standard practice of prior audio Transformer methods. #### `Positional Encoding.`{=latex} Table `\ref{tab:pos}`{=latex} compares learned absolute positional encodings with Rotary Position Embedding (RoPE) [@su2024roformer], applied independently along the frequency and time axes. RoPE substantially outperforms absolute encodings on both AudioSet benchmarks, with gains of $+1.8$ mAP on AS-2M and $+2.4$ mAP on AS-20K. Beyond the accuracy improvement, RoPE also offers a practical benefit: because it encodes relative positions rather than absolute ones, it transfers naturally to clips of different lengths at inference time, whereas absolute encodings require interpolation. We therefore adopt RoPE as NAPE's default positional encoding. ```{=latex} \begin{wraptable}{r}{0.395\textwidth} \small \renewcommand{\tabcolsep}{0.5mm} \caption{Ablation on the pooling method and attention type at fine-tuning.} %\resizebox{0.999\linewidth}{!}{ \begin{tabular}{lcccc} \toprule \textbf{Attention} & \textbf{Pooling} & \cellcolor{yellow}\textbf{Task} \\ \textbf{Type} & \textbf{Mode} & \emph{AS-2M} & \emph{AS-20K} & \emph{KS2} \\ \midrule Bidirec & CLS Tok & \textbf{49.7} & 38.9 & 98.7 \\ Bidirec & Last Tok & 49.6 & 38.7 & \textbf{98.8} \\ \textbf{Bidirec} & \textbf{Avg Pool} & 49.6 & \textbf{39.1} & \textbf{98.8} \\ \hline \addlinespace[2pt] Causal & Last Tok & 49.4 & 38.9 & \textbf{98.8} \\ \bottomrule \end{tabular}%} \label{tab:causalbid_table} % \end{wraptable} ``` #### `Attention Type and Pooling.`{=latex} Table `\ref{tab:causalbid_table}`{=latex} jointly ablates the attention mask used during fine-tuning and the pooling strategy for producing the clip-level representation. Under causal attention, the natural pooling choice is the last token: only the final position has access to the full preceding context, and the CLS token functions more as a BOS-like anchor at the start of the sequence than as a readout summary of the clip (this is in line with [@xu2025nepa]). Under bidirectional attention, however, every position sees the entire sequence, so CLS, last-token, and mean pooling all in principle have access to the same information. This is reflected in the numbers: the three bidirectional variants are essentially on par on the three tested tasks. Keeping causal attention with last-token pooling remains competitive but slightly worse than its bidirectional counterpart, with a $0.2$ mAP drop on both AS-2M and AS-20K. We therefore use bidirectional attention with mean pooling as NAPE's default fine-tuning configuration. ```{=latex} \renewcommand{\tabcolsep}{0.5mm} ``` ::: {#tab:pos} **Freeze emb** *AS-2M* *AS-20K* *KS2* ----------------- --------- ---------- ------- `\ding`{=latex} 49.61 39.14 98.73 `\ding`{=latex} 49.58 39.08 98.80 : `Ablation on positional encoding.`{=latex} ::: ```{=latex} \hfill ``` ::: {#tab:pos} **Positional Enc.** *AS-2M* *AS-20K* --------------------- ---------- ---------- Absolute 47.8 36.7 **RoPE** **49.6** **39.1** : `Ablation on positional encoding.`{=latex} ::: ```{=latex} \begin{wraptable}{r}{0.445\textwidth} \small \renewcommand{\tabcolsep}{0.5mm} \caption{Ablation on normalization layer style.} %\resizebox{0.999\linewidth}{!}{ \begin{tabular}{lcccccc} \toprule \textbf{Norm} & \textbf{Audio Tasks} & \textbf{Speech Tasks}\\ \textbf{Type} & \emph{AS-2M} & \emph{AS-20K} & \emph{ESC-50} & \emph{KS1} & \emph{KS2} &\emph{ER} \\ \midrule \textbf{LN} & 49.6 & 39.1 & 94.2 & 97.9 & 98.8 & 64.9 \\ RMS & 49.5 & 38.8 & 94.4 & 97.8 & 98.8 & 65.2 \\ \bottomrule \end{tabular}%} \label{tab:norm} % \end{wraptable} ``` #### `Normalization Layer.`{=latex} Table `\ref{tab:norm}`{=latex} compares LayerNorm [@ba2016layer] (LN) and RMSNorm [@zhang2019root] in the encoder. RMSNorm drops the mean-centering step of LayerNorm and normalizes each feature vector by its root-mean-square, which has become common in recent language and vision Transformers. In our setting, the two variants are effectively on par, with small variations across the six downstream tasks. For this reason, we retain LayerNorm as NAPE's default for consistency with prior audio Transformer works. ```{=latex} \begin{wrapfigure}{r}{0.5\textwidth} \includegraphics[width=\linewidth]{Figures/epochs.pdf} \caption{Pre-training budget ablation. AS-2M (left) and AS-20K (right) mAP trend as a function of the number of epochs the model is trained on.} \label{fig:epochs_trend} \end{wrapfigure} ``` #### `Pre-training Budget.`{=latex} Figure `\ref{fig:epochs_trend}`{=latex} reports downstream mAP on AS-2M and AS-20K when fine-tuning from NAPE-Base and NAPE-Large checkpoints obtained after $6$, $12$, $18$, and $30$ epochs of pre-training. Performance improves monotonically with the pre-training budget for both scales and both benchmarks: on AS-2M, NAPE-Large improves from $49.45$ mAP after $6$ epochs to $49.89$ after $30$; on AS-20K, from $37.15$ to $39.94$. Notably, NAPE already delivers strong results after only a few epochs of pre-training. After just $6$ epochs, NAPE-Large reaches $49.45$ mAP on AS-2M --- already matching or exceeding most published baselines (cf. Table `\ref{tab:main_table}`{=latex}) --- making NAPE an appealing choice under tight compute budgets. Longer pre-training continues to yield returns at $30$ epoch, and NAPE-Large consistently benefits more than NAPE-Base from a larger budget, consistent with the scaling behavior discussed in Section `\ref{sec:scaling}`{=latex}. ```{=latex} \begin{wraptable}{r}{0.43\textwidth} \small \renewcommand{\tabcolsep}{0.5mm} \caption{Ablation on random masking applied to the input embeddings.} %\resizebox{0.999\linewidth}{!}{ \begin{tabular}{ccccccc} \toprule \textbf{Mask} & \textbf{Audio Tasks} & \textbf{Speech Tasks}\\ \textbf{Ratio} & \emph{AS-2M} & \emph{AS-20K} & \emph{ESC-50} & \emph{KS1} & \emph{KS2} &\emph{ER} \\ \midrule \textbf{0} & \textbf{47.6} & \textbf{36.2} & \textbf{92.9} & 97.4 & 98.3 & \textbf{64.6} \\ 20 & 47.3 & \textbf{36.2} & 92.2 & \textbf{97.6} & \textbf{98.5} & 63.9 \\ 50 & 47.2 & 35.2 & 91.7 & 97.5 & 98.3 & 63.9 \\ \bottomrule \end{tabular}%} \label{tab:mask} \end{wraptable} ``` #### `Random Masking.`{=latex} Table `\ref{tab:mask}`{=latex} examines whether adding random masking to the input embeddings during pre-training, in the style of masked modeling methods, helps NAPE. We consider three masking ratios: no masking ($0\%$, the default), $20\%$, and $50\%$. We use NAPE small with raster scan order. Across all six benchmarks, masking either has no effect or slightly degrades downstream performance, with the largest drop on AS-20K ($-1.0$ mAP at $50\%$) and ESC-50 ($-1.2$ points at $50\%$). This differs from methods such as Audio-MAE [@huang2022masked] or BEATs [@chen2022beats], where high masking ratios are central to the pre-training signal: the model is asked to reconstruct or predict the masked content from the visible context. In NAPE, the causal attention mask already prevents each position from accessing its target, so additional input masking removes useful context without changing the difficulty of the prediction task. We therefore leave the input unmasked in NAPE's default configuration. # NAPE vs JEPA vs LeWorldModel Formulation {#sec:napevsjepa} ## NAPE vs JEPA-style Methods NAPE and JEPA methods [@assran2023self; @fei2023jepa] learn representations by predicting one set of patch embeddings from another, therefore it is worth clarifying how the two paradigms differ structurally. Figure `\ref{fig:nape_jepa_leworldmodel}`{=latex} (left and right) illustrates the two frameworks. JEPA-style methods use *two* encoders: a context encoder $h$ that processes the visible portion of the input $x$, and a target encoder that produces the prediction targets from the complementary region $y$. The target encoder is not trained by gradient descent; instead, its weights are updated as an EMA of the context encoder's weights, so that its output remains a moving target that the predictor tries to match. The predictor itself is a Transformer that receives both the context representation and an additional variable $c$, which encodes the target positions to be predicted, and produces the predicted target embeddings $\hat{z}_y$ in parallel. The context encoder operates with bidirectional attention over the visible patches. To prevent representation collapse under this regime, JEPA methods typically require additional machinery beyond stop-gradient: an EMA teacher (I-JEPA [@assran2023self], A-JEPA [@fei2023jepa]), auxiliary variance-covariance regularizers such as VICReg [@bardes2021vicreg], or masking strategies carefully tuned so that the two views do not degenerate into trivial solutions. NAPE, in contrast, uses a *single* encoder ($h$) and takes its prediction targets directly from the shared patch embedding layer $f$. Because $f$ is a shallow, non-recurrent module, its outputs provide a stable target signal that does not need to be maintained by a separate EMA-tracked encoder. NAPE also features a lightweight MLP head ($g$) that operates on a single position at a time, rather than a Transformer that jointly reasons about context and target positions: a structurally simpler design that reflects NAPE's per-position causal prediction task. There is no masking: the model processes the full input as a causal sequence, and predictions are made one patch at a time under a causal attention mask. Collapse prevention relies solely on the three key components discussed in Section `\ref{sec:nape-prediction}`{=latex}: causality, prediction shift, and stop-gradient, with no auxiliary regularizers or student-teacher setups. ## NAPE vs LeWorldModel Methods A closely related recent method is LeWorldModel [@maes2026leworldmodel], a JEPA designed for world modeling that also autoregressively predicts next-frame embeddings from raw pixels (Figure `\ref{fig:nape_jepa_leworldmodel}`{=latex}, middle). Like NAPE, LeWM aims to simplify the JEPA recipe and dispenses with EMA teachers and pretrained encoders, training its encoder and predictor jointly end-to-end. Unlike NAPE, LeWM relies on the SIGReg regularizer [@balestriero2025lejepa], an auxiliary loss that projects the learned embeddings onto random directions and enforces Gaussian-distributed marginals, to prevent collapse, whereas NAPE requires only stop-gradient on the target branch. LeWM uses MSE as its prediction loss (kept stable by SIGReg), whereas NAPE uses cosine similarity combined with stop-gradient, which together avoid the shrink-to-zero collapse mode observed with $\ell_1$ and $\ell_2$ under a magnitude-sensitive loss (Section `\ref{sec:ablations}`{=latex}). Finally, the two methods target different problems: LeWM predicts across time steps of an action-conditioned trajectory for planning in latent space, while NAPE predicts across patches within a single input for representation learning applicable to classification tasks. NAPE therefore constitutes a distinct point in the JEPA design space: one that shares LeWM's end-to-end simplicity but arrives at collapse prevention through a fundamentally different mechanism. ![Comparison between JEPA, LeWorldModel, and NAPE architectures.](Figures/NAPE_JEPA_LeWorld.png){#fig:nape_jepa_leworldmodel width="100%"} # Training Loss Visualizations {#sec:loss_visualization} To better understand NAPE's training dynamics, in Figure `\ref{fig:loss_curves}`{=latex} we report the pre-training loss curves of six NAPE runs across multiple scales and configurations. The top row reports runs of the default NAPE configuration (raster scanning, SimSiam predictor, cosine similarity, all three key components enabled) at the three model scales --- **Small** (top left), **Base** (top middle), and **Large** (top right). All three curves follow the same qualitative trajectory: a rapid descent from around $-0.3$ to below $-0.9$ within the first $\sim 20{,}000$ steps, followed by a smooth and steady decrease that asymptotes just below $-0.98$, close to the theoretical minimum of $-1$. The Large model runs for roughly twice as many steps as Small and Base, its batch size is halved to fit into memory, but reaches a comparable final loss along a trajectory of the same shape, indicating that the NAPE objective is well-conditioned across encoder capacities. The bottom row reports three ablation runs at the Base scale. **Without the causal mask** (bottom left), the loss collapses almost immediately, saturating near $-1.0$ within $\sim 2{,}000$ steps and remaining there. The model has found the trivial identity mapping enabled by bidirectional attention: the objective is satisfied without learning any structure that transfers downstream, as reflected in the sharp mAP drop reported in Table `\ref{tab:maincomponents}`{=latex}. The configuration with **L1 loss** (bottom middle) diverges: the loss drops rapidly to $\sim 0.05$ within a few hundred steps, plateaus briefly, and then climbs steadily. The initial descent reflects the model finding low-norm solutions in which both predicted and target embeddings shrink toward zero, since L1 is magnitude-sensitive and trivially reduced by scaling the outputs; once the encoder approaches this degenerate regime, the objective becomes ill-conditioned and the loss reverses. The **cross-entropy loss** (bottom right), in contrast, converges to a reasonable minimum around step $\sim 70{,}000$ before gradually overfitting. Unlike L1, CE does not collapse: this suggests that collapse prevention is not exclusive to cosine similarity. In our setting, however, CE still yields worse downstream performance than cosine (Table `\ref{tab:loss}`{=latex}), and its tendency to overfit after the initial descent makes it a less robust choice for NAPE pre-training. # Additional Qualitative Results {#sec:additional_qualitative} We include additional qualitative results for NAPE-L with raster order in Figure `\ref{fig:qualitative_2_raster}`{=latex}) and NAPE-L with diagonal order in Figure `\ref{fig:qualitative_2_diagonal}`{=latex}).
Pre-training loss curves for multiple NEPA models across scales and configurations.
```{=latex} \makeatletter ``` ```{=latex} \setlength{\@fptop}{0pt} ``` ```{=latex} \makeatother ```
Additional qualitative results for NAPE-L with raster order.
Additional qualitative results for NAPE-L with diagonal order.