--- bibliography: - reference.bib --- ```{=latex} \newcommand{\promptbox}[3]{% \colorlet{promptbg}{#3}% \colorlet{promptframe}{#3}% \par\noindent \begingroup \setlength{\fboxsep}{6pt}% \colorbox{promptframe!85!black}{% \parbox{\dimexpr\linewidth-2\fboxsep\relax}{% \color{white}\bfseries\small \rule[-4pt]{0pt}{14pt}#1\strut}% }\par\nointerlineskip \endgroup \lstinputlisting[style=gepaprompt]{#2}% \medskip% } ``` ```{=latex} \providecommand{\methodname}{\textsc{FST}\xspace} ``` ```{=latex} \newcommand{\todo}[1]{\textcolor{red}{[\textsc{todo}: #1]}} ``` ```{=latex} \newcommand{\NUM}[1]{\textcolor{orange}{#1}} ``` ```{=latex} \newcommand{\devvrit}[1]{\textcolor{red}{Devvrit: #1}} ``` ```{=latex} \begin{tcolorbox}[abstractbox, width=\textwidth]\centering {\LARGE\titleFont \textcolor{titleblue}{Learning, Fast and Slow: Towards LLMs\\ That Adapt Continually}\par} \vspace{0.5em} {\normalsize\authorFont \textbf{Rishabh Tiwari}$^{*\,1, 4}$\hspace{0.2em} \textbf{Kusha Sareen}$^{*\,2}$\hspace{0.2em} \textbf{Lakshya A Agrawal}$^{*\,1}$\\[0.2em] \textbf{Joseph E. Gonzalez}$^{\,1}$\hspace{0.2em} \textbf{Matei Zaharia}$^{\,1}$\hspace{0.2em} \textbf{Kurt Keutzer}$^{\,1}$\hspace{0.2em} \textbf{Inderjit S Dhillon}$^{\,3}$\\[0.2em] \textbf{Rishabh Agarwal}$^{\dagger\,2, 5}$\hspace{0.2em} \textbf{Devvrit Khatri}$^{\dagger\,3, 6}$ }\\[0.8em] {\normalsize\authorFont $^{1}${UC Berkeley}\hspace{1.5em}$^{2}${Mila}\hspace{1.5em}$^{3}${UT Austin}\hspace{1.5em}$^{4}${Eragon}\hspace{1.5em}$^{5}${Periodic Labs}\hspace{1.5em}$^{6}${Mirendil}}\\[0.3em] {\normalsize\authorFont\hypersetup{urlcolor=titleblue} \href{https://rishabhtiwari.ai/projects/fst/video.mp4}{Video} \hspace{0.5em}|\hspace{0.5em} \href{https://gepa-ai.github.io/gepa/blog/2026/05/11/learning-fast-and-slow/}{Blog} \hspace{0.5em}|\hspace{0.5em} \href{https://rishabhtiwari.ai/projects/fst/code/}{Code}}\\[0.1em] \vspace{0.1em} \begin{justify} \authorFont \setstretch{1.1} \setlength{\parindent}{0pt} % \begin{abstract} % %Training large language models usually optimizes a single object: the model parameters. Yet deployed LLM systems are controlled by both slowly changing parameters and rapidly changing textual state, including prompts, instructions, and task context. % The process of training a large language model optimizes a single object: the model's parameters. % However, deployed LLM systems are controlled by both slowly changing parameters and rapidly changing textual state, including prompts, instructions, and task context. % %We argue that effective training should jointly optimize these two channels. % We argue that effective training should jointly optimize these two components. % XXXX % Alternative: % XXXX % Inspired by neuroscience and classical machine learning work, we introduce a fast-slow learning framework for LLM post-training, where model parameters act as slow weights and optimized textual context as fast weights. The key advantage is that fast-weights absorb part of the domain-specific learning burden that would otherwise be forced into the parameters. % % , as happens in the currently used paradigm. % Context optimization enables rapid task-level adaptation by learning from text feedback, while slow weights can remain closer to the base model and focus on consolidating more persistent reasoning behaviors. We study this principle in reinforcement learning with verifiable rewards by interleaving slow-weight RL updates with fast-weight GEPA-style context optimization. % Our method FULL NAME (\methodname{}) is up to $3\times$ more sample-efficient across math, code, and scientific reasoning tasks, while consistently reaching a higher performance asymptote. % XX SOME NUMBERS HERE, NOT SAMPLE EFFICIENCY XX. % % Our method leads to more sample-efficient learning across math, code, and scientific reasoning tasks and results in a higher performance asymptote. % % : fast weights quickly improve the training trajectory, allowing the system to reach higher reward with fewer slow-weight updates. % Since performance is achieved with less parameter movement, the resulting models % % remain closer to the base checkpoint and thus % exhibit (up to $X\%$) lower KL to the base policy. % % , higher entropy, and stronger out-of-distribution generalization than RL-only baselines. % Preserving slow weights thereby improves future adaptability: after training on one task, fast-slow trained models retain greater plasticity and achieve larger gains when subsequently trained on a new task. % % Moreover, the fast-slow trained model exhibit continual learning . % % Finally, we investigate fast-to-slow distillation and show that improvements discovered in fast weights can be partially consolidated into parameters, making our approach complementary to recent self-distillation methods. % % Finally, we exhibit strong continual learning ability of our method when task domains change on the fly. % Finally, we observe strong continual learning ability. When task domains change on the fly, our method continues to acquire each new task while parameter-only RL stalls. % More broadly, our work motivates a new paradigm of multi-timescale optimization, where fast textual adaptation and slow parameter learning are optimized jointly end-to-end rather than sequentially. % % \todo{Is there anything sharper we can say about the advantages of co-evolving? anything more quantitative?} % \end{abstract} % \begin{abstract} Large language models (LLMs) are trained for downstream tasks by updating their parameters (e.g., via RL). However, updating parameters % typically optimizes only the model’s parameters. % This enables the weights to forces them to absorb task-specific information, which can result in catastrophic forgetting and loss of plasticity. In contrast, in-context learning with fixed LLM parameters can cheaply and rapidly \emph{adapt} to task-specific requirements (e.g., prompt optimization), but cannot by itself typically match the performance gains available through updating LLM parameters. % We argue that effective training should jointly optimize these two components. % XXXX % Alternative: % XXXX %We argue that effective post-training should jointly optimize these two components. There is no good reason for \emph{restricting} learning to being in-context or in-weights. Moreover, humans also likely learn at different time scales (e.g., System 1 vs 2). To this end, we introduce a fast-slow learning framework for LLMs, with model parameters as ``slow'' weights and optimized context as ``fast'' weights. These fast ``weights'' can learn from textual feedback to absorb the task-specific information, while allowing slow weights to stay closer to the base model and persist general reasoning behaviors. % Inspired by neuroscience and classical machine learning work, we introduce a fast-slow learning framework for LLM post-training, where model parameters act as slow weights and optimized textual context as fast weights. The key advantage is that fast-weights absorb part of the domain-specific learning burden that would otherwise be forced into the parameters. % , as happens in the currently used paradigm. % Context optimization enables rapid task-level adaptation by learning from text feedback, while slow weights can remain closer to the base model and focus on consolidating more persistent reasoning behaviors. We study this principle in reinforcement learning with verifiable rewards by interleaving slow-weight RL updates with fast-weight GEPA-style context optimization. \textbf{Fast-Slow Training} (\methodname{}) is up to $3\times$ more sample-efficient than only slow learning (RL) across reasoning tasks, while consistently reaching a higher performance asymptote. Moreover, FST-trained models remain closer to the base LLM (up to $70\%$ less KL divergence), resulting in less catastrophic forgetting than RL-training. % Since performance is achieved with less parameter change, the resulting models % remain closer to the base checkpoint and thus% exhibit (up to $X\%$) lower KL to the base policy. % , higher entropy, and stronger out-of-distribution generalization than RL-only baselines. This reduced drift also preserves plasticity: after training on one task, FST trained models adapt more effectively to a subsequent task than parameter-only trained models. % Preserving slow weights thereby improves future adaptability: after training on one task, fast-slow trained models retain greater plasticity and achieve larger gains when subsequently trained on a new task. % Moreover, the fast-slow trained model exhibit continual learning . % Finally, we investigate fast-to-slow distillation and show that improvements discovered in fast weights can be partially consolidated into parameters, making our approach complementary to recent self-distillation methods. % Finally, we exhibit strong continual learning ability of our method when task domains change on the fly. In continual learning scenarios, where task domains change on the fly, FST continues to acquire each new task while parameter-only RL stalls. % As such, FST offers strong continual learning ability. % Overall, our work motivates a new paradigm where fast context adaptation and slow parameter learning happens jointly, which we believe open up new avenues for LLMs that adapt continually. % end-to-end rather than sequentially. % \todo{Is there anything sharper we can say about the advantages of co-evolving? anything more quantitative?} % \end{abstract} \end{justify} \vspace{0.3em} {\centering\small\authorFont \textbf{Correspondence}: \texttt{\href{mailto:rishabhtiwari@berkeley.edu,lakshyaaagrawal@berkeley.edu}{\{rishabhtiwari, lakshyaaagrawal\}@berkeley.edu}, \href{mailto:kusha.sareen@mila.quebec}{kusha.sareen@mila.quebec}, \href{mailto:devvrit.03@gmail.com,rishabhagarwal.467@gmail.com}{\{devvrit.03, rishabhagarwal.467\}@gmail.com}\\[0.5em]$^{*}$Equal contribution\hspace{0.1em} $^{\dagger}$Equal advising.}\par} \end{tcolorbox} ``` ```{=latex} \vspace{1.0em} ``` ```{=latex} \vspace{-1.5mm} ``` Introduction ============ ```{=latex} \vspace{-1.5mm} ``` ```{=latex} \centering ``` ![**Fast-Slow Training (`\methodname{}`{=latex}) learns faster, stays adaptable, and forgets less.** Comparison of `\methodname{}`{=latex} (slow-weight RL interleaved with fast-weight prompt optimization), RL alone (slow weights only), and GEPA alone (fast weights only), averaged over `CodeIO`, `Math (Polaris)`, and `HoVer-hard`. **Left:** Evaluation accuracy on the trained task as training samples accumulate. `\methodname{}`{=latex} reaches RL's peak with substantially fewer samples and converges to a higher ceiling than either RL or GEPA alone. **Middle:** Plasticity, the model's remaining ability to learn a new skill. After training on a first task, we continue with a fresh round of RL on a second task and report the final accuracy from each initialization. The RL-trained checkpoint barely learns the new task, while the `\methodname{}`{=latex}-trained checkpoint roughly matches the base model. **Right:** How far each training run drifts from the base model, measured by KL$(\pi_{\mathrm{train}}\,\|\,\pi_{\mathrm{base}})$. Smaller drift correlates with less forgetting of prior abilities; at matched accuracy, `\methodname{}`{=latex} sits well to the left of RL.](paper_plots/fig_main_hero_v4.png "fig:"){#fig:hero width="\\linewidth"} `\vspace{-4.5mm}`{=latex} ```{=latex} \centering ``` ![**Slow weights and fast weights co-evolve through interleaved updates.** The slow loop (top) updates $\theta$ from the scalar reward alone ($\theta_c \to \theta_{c+1}$). The fast loop (bottom) updates $\Phi$ via reflective optimization, additionally consuming the rollout's full text including thoughts, tool calls, errors, and rich feedback ($\Phi_c \to \Phi_{c+1}$). Maintaining $\Phi$ as a Pareto-frontier population (rather than a single best prompt) preserves diversity: different contexts specialize to different problem slices exposing the slow update rule to rich conditioning during training. ](figures/D9.png "fig:"){#fig:method width="\\linewidth"} `\vspace{-4.5mm}`{=latex} Large language models (LLMs) are commonly adapted through supervised finetuning (SFT) or reinforcement learning (RL), both of which modify the model parameters, to specialized domains such as math and coding [@deepseek_r1; @tulu3; @deepseekmath; @deepseekcoder; @instructgpt]. However, treating parameter updates as the sole mechanism of adaptation creates a fundamental bottleneck: every improvement, whether it be a reusable reasoning skill, a task-specific heuristic or a transient lesson from recent rollouts, must be written into the same persistent set of model weights. Since the entire policy is parameterized by these weights, an update that improves in-domain reward simultaneously moves the model away from its base behavior [@gao2022scalinglawsrewardmodel; @mitigatingalignmenttaxrlhf], reducing entropy [@entropymechanism; @li2026choicedivergenceneglectedkey], hurting out-of-distribution generalization [@instructgpt; @scalinglawsforgettingfinetuning; @empiricalstudycatastrophicforgetting; @beyondreasoninggains], and degrading its ability to adapt to future tasks, known as plasticity loss [@Frati_2024; @understandingpreventingcapacityloss; @Dohare2024; @mitigatingplasticitylosscontinual]. LLM systems also possess another powerful adaptation mechanism: prompts, instructions, and contextual information [@languagemodelsfewshotlearners; @cot]. Unlike model parameters, these textual components can be modified cheaply, frequently, and per task. Prompt optimization methods demonstrate that substantial behavioral improvements can be obtained by improving the textual context under which the model operates [@ape; @opro; @dspy; @miprov2; @gepa]. In this work, we introduce **Fast-Slow Training (FST)**, where we view LLM adaptation as occurring through two complementary components (Figure `\ref{fig:method}`{=latex}). The first is a *slow parametric component*: the model weights, which are expensive to update, persist across tasks, and encode long-lived behavior. The second is a *fast textual component*: prompts, instructions, and task context, which can be changed cheaply and frequently, influence behavior immediately, capturing task-level adaptation without permanently modifying the model. The fast-slow distinction we draw above has a long history in neural networks [@hinton1987using; @schmidhuber1992learning; @ba2016usingfastweightsattend; @anand2023predictioncontrolcontinualreinforcement], motivated by separating temporary, task-specific adaptations in fast-weights from persistent, broadly useful behaviors in slow-weights. We instantiate this idea in RLVR [@deepseekmath; @scalerl] by interleaving slow reinforcement learning updates with fast context optimization using GEPA [@gepa]. Rather than first training a policy and then optimizing a prompt for the final checkpoint, our method allows the context and the policy to co-evolve. The fast textual weights quickly incorporate lessons from rollouts, steering the model toward better reasoning behavior, while the slow parametric weights are updated under this evolving context. This produces a training process in which performance gains are distributed appropriately across both elements, instead of being forced entirely into the model parameters. This division of labor has several consequences, which we evaluate in RLVR settings spanning math, code, and general reasoning tasks. 1. **Fast textual adaptation improves data efficiency.** Fast weights incorporate task-level signal rapidly, so the system improves without waiting for slow parameter updates. Empirically, fast-slow training matches RL reward with up to $3\times$ fewer rollouts and consistently reaches a higher performance ceiling (Section `\ref{sec:de}`{=latex} Advantages 1 and 2). 2. **Fast-slow training induces smaller slow-weight displacement.** With the textual channel carrying part of the adaptation, the parameters need not move as far from the base policy. At matched reward, our models have up to 70% lower KL to the base policy than RL-only baselines. (Section `\ref{sec:displacement}`{=latex} Advantage 3) 3. **Fast-slow training preserves plasticity.** We test this by training on one task using RL-only and FST, then continuing training on a second task from the resulting checkpoints; fast-slow trained models adapt effectively in the second phase while RL trained models collapse to near $0\%$ - suggesting FST retains greater capacity for future learning (Section `\ref{sec:plasticity}`{=latex} Advantage 4). 4. **Fast-slow training enables continual learning.** We test our method in setting where tasks change on the fly. We observe our method is able to adapt more quickly to changing objectives (Section `\ref{sec:cl}`{=latex} Advantage 5). Overall, our results suggest that effective LLM post-training should not be viewed as parameter learning followed by prompt tuning. Instead, it should be viewed as optimization over multiple adaptation channels, where fast textual weights and slow parametric weights are trained together to achieve rapid and task-specific improvements while preserving the generality and plasticity of the base model. Preliminaries {#sec:prelim} ============= #### Fast and slow weights: a general framework We model the *slow weights* (model parameters) as $\theta$, and *fast weights* (textual scaffolds) as $\phi$ drawn from a discrete text space $\Sigma^\ast$. Given a query $x$, the system produces a response by sampling $$y \sim \pi_\theta(\cdot \mid x, \phi), \label{eq:policy}$$ where $\pi_\theta(y \mid x, \phi)$ denotes the policy induced by parameters $\theta$ when conditioned on textual context $\phi$ and query $x$. For a task distribution $\mathcal{D}$ and reward $r$, the natural joint objective is $$\max_{\theta,\, \phi}\;\; J(\theta, \phi) \;=\; \mathbb{E}_{x \sim \mathcal{D},\; y \sim \pi_\theta(\cdot \mid x, \phi)}\!\big[\,r(x, y)\,\big]. \label{eq:joint-base}$$ Each factor admits many concrete optimizers. On the slow side, $\theta$ can be updated by SFT, preference optimization [@dpo], or policy-gradient methods such as PPO [@ppo] and GRPO [@deepseekmath], frequently under verifiable rewards [@tulu3]. On the fast side, $\phi$ can be updated by automated prompt-optimization methods such as APE [@ape], OPRO [@opro], DSPy/MIPROv2 [@dspy; @miprov2], and GEPA [@gepa]. Our framework is agnostic to these choices; we instantiate it with RL with verifiable rewards (RLVR) for $\theta$ and reflective evolutionary prompt optimization (GEPA) for $\phi$. #### Slow weights: RL with verifiable rewards We follow the ScaleRL recipe [@scalerl] for slow-weight updates. The reward $r(x, y) \in [0, 1]$ is given by an automatic verifier on $(x, y)$ [@tulu3] (e.g., rule-based correctness for math, code, and science tasks). For each query $x$, the current policy generates a *group* of $G$ rollouts $\{y_i\}_{i=1}^G$ under the current $(\theta, \phi)$, from which group-relative advantages [@deepseekmath] are computed, $$A_i \;=\; \frac{r(x, y_i) \;-\; \bar r_g}{\sigma_g \;+\; \varepsilon}, \qquad \bar r_g \;=\; \tfrac{1}{G}\!\sum_{j=1}^{G} r(x, y_j), \qquad \sigma_g^2 \;=\; \tfrac{1}{G}\!\sum_{j=1}^{G}\!\big(r(x, y_j) - \bar r_g\big)^2, \label{eq:adv}$$ and normalized at the batch level. The policy is updated using the truncated importance-sampling REINFORCE objective [cispo]{.smallcaps} [@minimaxm1; @scalerl], $$\mathcal{L}_{\textsc{cispo}}(\theta) \;=\; -\,\mathbb{E}\!\left[\, \mathrm{sg}\!\big(\min(\rho_t,\,\tau)\big)\,\cdot\, A \,\cdot\, \nabla_\theta \log \pi_\theta(y_t \mid x, \phi, y_{ ```{=latex} > \small > ``` > ```{=latex} > \ttfamily > ``` > Given a bi-directional graph in the form of space separated edges, output a path from source node to the destination node in the form of comma separated integers.\ > For this question the graph is {graph}\ > The source node is {source}\ > The destination node is {destination}\ > Please reason step by step, and put your final answer within \\boxed{}. The seed system prompt used during RL and as the initial GEPA candidate is: > `\small`{=latex} *You are solving a graph path-finding task. You will be given a list of edges and a source and destination node. Output one valid path from source to destination. Inspect the source node's neighbors first, identify which neighbor leads to the destination via a sequence of valid edges, then commit to that branch. Each consecutive pair in your output path must be a valid edge in the graph. Put your final answer comma-separated inside boxed braces.* #### Scoring. The reward function extracts the contents of the last `\boxed{…}` from the post-`` body of the rollout, strips whitespace, and applies an exact-match comparison against the gold path $s, v_1, \ldots, v_{p-2}, g$ rendered as a comma-separated string. The reward is $1.0$ on exact match and $0.0$ otherwise. The task admits no partial credit, so even a single wrong intermediate node zeros the reward. #### Splits used in the paper. We sweep difficulty by varying $(d, p, n)$. The headline experiments use $(d,p,n) = (25, 20, 500)$ with $10{,}000$ training examples and $200$ held-out test examples. Hyperparameters and compute {#app:hyperparams} =========================== This appendix lists all training hyperparameters needed to reproduce the numbers in the main paper. Settings shared across domains are described first; per-domain overrides follow in Tables `\ref{tab:hyper-rl}`{=latex}--`\ref{tab:hyper-gepa}`{=latex}. #### Shared RL configuration. All RL runs are GRPO [@deepseekmath] with the [cispo]{.smallcaps} surrogate loss [@minimaxm1; @scalerl] ($\text{clip\_low}=1.0$, $\text{clip\_high}=3.0$), advantage normalization by per-group standard deviation, and a small KL-to-reference penalty ($\text{coef}=10^{-3}$). The actor optimizer is AdamW (PyTorch defaults $\beta_1=0.9$, $\beta_2=0.999$, weight decay $0$) at learning rate $10^{-6}$ with a $10$-step linear warm-up; we use no learning-rate decay. Each RL step samples $G=8$ rollouts per problem with `train_batch_size`$\,=32$ problems (so $256$ rollouts per step), runs PPO updates with `ppo_mini_batch_size`$\,=32$, and uses tensor-parallel size $1$ for the rollout engine (vLLM). At evaluation time we report mean@$4$ over four rollouts per validation prompt at temperature $0.6$, top-$p$ $0.95$. We checkpoint every $50$ training steps and keep all checkpoints. All runs use Qwen3 *thinking* mode except star-graph (which uses the Instruct base, no thinking) and the no-thinking baselines reported in the main text. #### Shared GEPA configuration. GEPA cycles use the same settings for all domains: cycle length $T=6$ RL steps between GEPA optimizations (equivalently, $\texttt{warmstart\_steps}=\texttt{rl\_steps\_per\_cycle}=6$); all $K$ population prompts are scored on every question (`prompts_per_question`$\,=K$) with advantage grouping *Option B* (`advantage_grouping="question"`), so a problem's $G$ rollouts are split $G/K$ per prompt and a single group statistic $(\bar r_g, \sigma_g)$ is computed across all of them. The reflection LM is OpenAI `gpt-5.2`, accessed through LiteLLM. Per-cycle GEPA budgets are $\texttt{num\_eval\_examples}=192$ and $\texttt{max\_metric\_calls}=960$ across all domains except polaris ($96$ and $1500$) and star-graph ($200$ and $960$). Candidate prompts are injected as a system message; for datasets whose raw prompt already contains a system message (HoVer, Physics, Math) we *merge* the GEPA prompt into the existing system role rather than stack a second one. Each GEPA cycle outputs a Pareto-frontier population of size $K$ which seeds the next cycle. #### Per-domain overrides. Tables `\ref{tab:hyper-rl}`{=latex} and `\ref{tab:hyper-gepa}`{=latex} summarize the parameters that vary across training domains. ```{=latex} \centering ``` ```{=latex} \small ``` ```{=latex} \setlength{\tabcolsep}{4pt} ``` ::: {#tab:hyper-rl} Domain Base model $L_{\text{ctx}}$ / $L_p$ / $L_r$ Batch Rollouts/step GPU util. ---------------- -------------------------------- ----------------------------------------- ------- --------------- ----------- HoVer-hard Qwen3-8B (think) 18944 / 4096 / 8192 32 256 0.6 Physics Qwen3-8B (think) 18944 / 4096 / 8192 32 256 0.7 CodeIO Qwen3-8B (think) 18944 / 4096 / 8192 32 256 0.6 Math (Polaris) Qwen3-8B-SFT$^\dagger$ (think) 12288 / 4096 / 8192 64 512 0.7 Star-graph Qwen3-4B-Instruct (no think) `\phantom{0}`{=latex}8192 / 4096 / 4096 32 256 0.6 : Per-domain RL hyperparameters and base models. $L_{\text{ctx}}$/$L_p$/$L_r$ are the maximum context, prompt, and response lengths (tokens). Train batch is the number of problems per RL step; rollouts/step is $\text{batch} \times G$. ::: \ `\footnotesize `{=latex}$^\dagger$ Polaris uses our own continued-SFT base (Qwen3-8B base further SFT'd on Nemotron to recover math performance, since Qwen3-8B-Instruct is already saturated on math). For Polaris only, GRPO advantages are not normalized by group std (`norm_adv_by_std_in_grpo=false`) and the train batch is doubled to $64$ problems/step. ```{=latex} \centering ``` ```{=latex} \small ``` ::: {#tab:hyper-gepa} Domain $K$ $G/K$ Cycle $T$ Eval set Metric calls Reflection LM ---------------- ----- ------- ----------- ------------------------- -------------------------- --------------- HoVer-hard 8 1 6 192 `\phantom{0}`{=latex}960 `gpt-5.2` Physics 4 2 6 192 `\phantom{0}`{=latex}960 `gpt-5.2` CodeIO 8 1 6 192 `\phantom{0}`{=latex}960 `gpt-5.2` Math (Polaris) 4 2 6 `\phantom{0}`{=latex}96 1500 `gpt-5.2` Star-graph 8 1 6 200 `\phantom{0}`{=latex}960 `gpt-5.2` : Per-domain GEPA hyperparameters. $K$ is the population size; $G/K$ is the number of rollouts each candidate prompt produces per problem within an RL group. Eval set / metric calls are the per-cycle GEPA budgets. ::: #### Compute and wall-clock. All runs are submitted to a SLURM cluster with $8 \times$ H100 (80GB) per node. The headline runs in the main paper are single-node ($1\times8$ GPU); the Polaris $K=8$ ablation is the only multi-node configuration ($4\times8 = 32$ GPUs). Mean per-RL-step wall-clock under the headline configuration is $\sim 60$ s for RL-only and $\sim 100$ s for `\methodname{}`{=latex} without rollout reuse (HoVer-hard, $K=8$). Enabling rollout reuse (Section `\ref{app:rollout-reuse}`{=latex}) brings the RL-step cost down to $\sim 47$ s, slightly faster than RL-only at the step level, because $\sim 1/3$ of the RL group's rollouts are served from the GEPA evaluation cache rather than freshly generated. This figure covers the RL training loop only: `\methodname{}`{=latex} additionally runs periodic GEPA cycles (rollouts for $K$ candidate prompts plus a reflection call), which add real wall-clock on top, so end-to-end a `\methodname{}`{=latex} run is more expensive than an RL-only run of equal step count. RL training runs go to either $1500$ steps or until validation accuracy saturates (whichever comes first); a single headline RL+`\methodname{}`{=latex} run consumes on the order of $25$--$40$ H100-GPU-hours, of which the GEPA cycles are a sizable fraction. GEPA reflection calls to `gpt-5.2` are billed separately; at the per-cycle metric-call budget above this is $\lesssim$ \$10 per training run. Design ablations {#app:codeio-ablations} ================ This appendix collects the four design-choice ablations. All sweeps are run on CodeIO (Qwen3-8B thinking, light-recipe defaults from Appendix `\ref{app:hyperparams}`{=latex}) and reported at a matched RL step ($500$) on the held-out CodeIO mean@$4$. The unmodified RL-only baseline at the same step ($\mathbf{39.65\%}$) is included in every panel for reference. ```{=latex} \centering ``` ![CodeIO design ablations (val mean@$4$ at training step $500$). Sage bars mark the headline configuration in each panel; gray bars are the alternatives swept; the dashed line indicates RL-only at the same matched step. (**a**) Population size $K$. (**b**) Advantage baseline at $K{=}2$: per-prompt (Prompt baseline) vs. per-problem (Problem baseline). (**c**) Cycle length $T$ at $K{=}8$, Problem baseline. (**d**) Light vs. full GEPA recipe.](paper_plots/fig_codeio_ablation_panels.png){#fig:codeio-ablations width="\\linewidth"} #### Population size $K$ (Fig. `\ref{fig:codeio-ablations}`{=latex}a). We sweep $K \in \{1, 2, 4, 8\}$ holding the rest of the recipe at light / Problem baseline / cycle $T{=}6$. Every $K \geq 1$ improves on RL-only ($39.65\%$), indicating that even a single optimized prompt carries useful task signal into RL. Performance is non-monotonic in $K$: $K{=}1$ already buys $+1.5$ pp ($41.10\%$), $K{=}2$ Problem baseline and $K{=}4$ are roughly tied at $\sim$$40.4\%$, and the gain saturates at $K{=}8$ with $\mathbf{42.84\%}$ (cycle $6$), the headline configuration. Larger $K$ widens the within-group prompt distribution and gives GRPO advantages a richer signal to compare against, but only when paired with a tight cycle (see panel c). $K{=}8$ with cycle$=12$ recovers most but not all of the gain ($41.13\%$). #### Advantage baseline at $K{=}2$ (Fig. `\ref{fig:codeio-ablations}`{=latex}b). With a fixed population of $K{=}2$ prompts, the choice of GRPO advantage baseline matters. The *Prompt baseline* (per-prompt: rollouts under each $\phi^{(k)}$ are normalized against their own group mean and std) yields $39.30\%$, slightly *below* RL-only. The *Problem baseline* (per-problem: all $G$ rollouts under both prompts share a single group statistic) yields $\mathbf{40.65\%}$, $+1.4$ pp over the Prompt baseline and $+1.0$ pp over RL-only. The intuition: the prompt baseline makes prompts compete only with themselves and discards the cross-prompt comparison entirely, while the problem baseline exposes the policy gradient to which prompt a stronger response came from on the same problem. The problem baseline is the default for all main-text experiments. #### Cycle length $T$ at $K{=}8$ (Fig. `\ref{fig:codeio-ablations}`{=latex}c). Holding $K{=}8$ Problem-baseline fixed, we compare $T{=}6$ vs. $T{=}12$ RL steps between successive GEPA optimizations. Cycle $T{=}6$ reaches $\mathbf{42.84\%}$; doubling the cycle to $T{=}12$ drops mean@$4$ by $1.7$ pp to $41.13\%$, giving up more than half of the advantage over RL-only. This is the expected staleness story: as $\theta$ moves between GEPA cycles, the prompts in $\Phi$ become increasingly mistuned to the current policy, and the per-question rollout-group signal degrades. $T{=}6$ is short enough that the population $\Phi_c$ remains close to optimal across the cycle, though it requires twice as many GEPA optimizations as cycle 12. #### Light vs. full GEPA recipe (Fig. `\ref{fig:codeio-ablations}`{=latex}d). The \`\`light" recipe uses $K{=}4$ candidates, a smaller per-cycle GEPA budget (`num_eval_examples`$=192$, `max_metric_calls`$=960$), and a proposer prompt that asks `gpt-5.2` for incremental edits to the current best prompt rather than rewrites from scratch. The \`\`full" recipe is the original configuration from @gepa: $K{=}1$, doubled metric budget (`max_metric_calls`$=1922$), and an open-ended proposer that allows full rewrites. Within our setup the full recipe gives essentially no lift over RL-only ($39.85\%$ vs. $39.65\%$); the light recipe at the same $K{=}1$ yields $+1.5$ pp ($41.10\%$); and scaling light to $K{=}8$ gives a further $+1.7$ pp ($\mathbf{42.84\%}$). Two factors contribute. First, full's $K{=}1$ pinches the population channel that turns out to matter most in panel a. Second, the open-ended rewrite proposer is more prone to drift on a moving policy: incremental edits keep the population's induction biases coherent across cycles, while wholesale rewrites tend to discard structure each round. The combined effect is a recipe that runs on roughly half the GEPA budget and outperforms the original by $\sim$$3$ pp on this task. Rollout reuse: same accuracy at lower wall-time and rollout cost {#app:rollout-reuse} ================================================================ ```{=latex} \centering ``` ![Rollout reuse on HoVer-hard, training step $\le$300. **Left:** mean wall-time per RL step. `\methodname{}`{=latex}+reuse drops from $\sim$66 s to $\sim$47 s (about $30\%$ faster); the saving comes almost entirely from the generation phase. **Middle:** cumulative RL trajectories. The shaded region splits each step's $256$ trajectories into live-generated (lavender) and from-cache (amber). By step 300 about $\sim$17,000 of the $\sim$77,000 RL trajectories were served from cache. **Right:** HoVer-hard val accuracy mean\@4. `\methodname{}`{=latex}+reuse tracks `\methodname{}`{=latex}; the no-prompt RL baseline lags both.](paper_plots/fig_rollout_reuse.png){#fig:rollout-reuse width="\\linewidth"} #### Why reuse is possible. `\methodname{}`{=latex} interleaves GEPA prompt optimization with RL updates: every $T$ RL steps GEPA runs a cycle that scores each candidate prompt $\phi^{(k)}$ on a small evaluation pool drawn from the same training distribution the next RL phase will see. Each evaluation entry is a tuple $(p, \phi^{(k)}, y, r)$ -- a problem $p$, the prompt $\phi^{(k)}$ used, the sampled response $y$, and its scalar reward $r$. Without reuse, those tuples are thrown away once GEPA picks a new population $\Phi_{c+1}$; the next RL step then re-rolls $G$ fresh trajectories per problem from scratch under the new prompts. But because the population $\Phi_{c+1}$ is a Pareto-frontier subset of the prompts GEPA *already evaluated*, many of those discarded tuples are perfectly valid samples from the current policy under one of the current prompts. The natural optimization is to splice them into the RL group instead of regenerating them. #### Cache mechanics. We maintain a per-(problem, prompt) cache of GEPA evaluation tuples produced during the most recent cycle. When the RL phase forms its rollout group of size $\text{batch} \times G$ across the prompts in $\Phi_{c+1}$, it first claims any cached $(p, \phi, y, r)$ matching a (problem, prompt) slot and only generates the remaining slots live with vLLM. Cached and live trajectories are then concatenated into a single GRPO group; the policy gradient does not distinguish the two sources because both were sampled under prompts in the active population. The cache is cleared when GEPA produces the next $\Phi_{c+2}$, so reused trajectories are at most $T$ RL steps old ( $T{=}6$ in our headline configuration). #### Empirical effect. Figure `\ref{fig:rollout-reuse}`{=latex} measures the impact on HoVer-hard. We compare two otherwise identical `\methodname{}`{=latex} runs -- one with reuse enabled and one without -- through the first $300$ training steps, with a no-prompt RL baseline as a third reference. Wall-time per RL step (left panel) drops from $\sim$66 s to $\sim$47 s, a $\sim$$29\%$ speedup that comes almost entirely out of the generation phase: GEPA's own per-cycle cost is unchanged, but a substantial fraction of the following RL phase's rollouts no longer need to be sampled. The middle panel decomposes the cumulative trajectory budget into the live and from-cache components: by step $300$, roughly $17$k of the $77$k total trajectories ($\sim$$22\%$) were served from cache, with the cache hit rate concentrated in the first few RL steps after each GEPA cycle and tapering as the live group fills in problems not in GEPA's evaluation pool. The right panel shows that this comes at no accuracy cost: the reuse and non-reuse `\methodname{}`{=latex} curves are within sampling noise of each other on HoVer-hard val mean@$4$, and both lead the no-prompt RL baseline throughout. KL-vs-reward, full four-task results {#app:polaris-kl} ==================================== ```{=latex} \centering ``` ![Validation reward versus $\mathrm{KL}(\pi_{\mathrm{train}}\,\|\,\pi_{\mathrm{base}})$ on all four training tasks: `CodeIO`, `Math` (Polaris), `HoVer`, and `Physics`. Same axes, smoothing, and conventions as Figure `\ref{fig:kl-vs-acc}`{=latex} in the main text. The three-task variant in the main text drops the Polaris panel for the reasons discussed below.](paper_plots/fig_kl_vs_acc_v2.png){#fig:kl-vs-acc-full width="\\linewidth"} The Polaris (math) trajectories in Figure `\ref{fig:kl-vs-acc-full}`{=latex} look qualitatively different from the other three tasks: RL and `\methodname{}`{=latex} sit on top of each other in $(\mathrm{KL}, \text{reward})$-space rather than `\methodname{}`{=latex} pulling the frontier to the left. We attribute this to the base model used for the Polaris runs. Unlike the other three tasks, which start from `Qwen3-8B` (an instruction-tuned model with strong format following), Polaris was trained on top of a model built by SFT'ing `Qwen3-8B-Base` on Nemotron, since the public `Qwen3-8B` is already saturated on Polaris. That custom SFT base has noticeably weaker instruction-following than the public Instruct checkpoint: the GEPA-evolved prompts that drive `\methodname{}`{=latex}'s KL gain on the other three tasks rely on the policy actually following format and self-checking instructions in the prompt, and on this base much of that signal is lost. The model still learns the math task from RL reward, so the reward axis behaves normally; it simply does not benefit from the prompt channel as strongly, and the two trajectories collapse onto each other. We expect the Polaris curve to look more like the other three with a stronger instruction-tuned base, but isolating that requires retraining and is left for future work. Explicit fast-to-slow distillation {#app:explicit-distill} ================================== The ceiling claim in Section `\ref{sec:abl:explicit-distill}`{=latex} raises a natural question: can the gains from the fast textual channel be folded into the parameters explicitly, without ever doing RL on the slow weights? We test this by replacing the slow-weight policy-gradient update with an on-policy reverse-KL distillation loss $$\label{eq:distill} \mathcal{L}_{\text{distill}}(\theta) \;=\; \mathbb{E}_{x \sim \mathcal{D},\; y \sim \pi_\theta(\cdot \mid x)} \!\left[\, \sum_{t=1}^{|y|} \mathrm{KL}\!\Big(\pi_\theta(\cdot \mid x, y_{