Forget about credit assignment in long-horizon agentic post-training

Evolution Strategies (ES) fine-tuning of Qwen3.5-4B outperformed the policy-gradient recipe Tmax on TBench in preliminary experiments, according to a blog post arguing that zeroth-order methods should be revisited for long-horizon agentic post-training. The post cites Qiu et al. 2026 and Liang et al. 2026 in reporting that ES does not suffer the curse of dimensionality when fine-tuning billion-parameter LLMs, while policy-gradient RL suffers a "curse-of-horizon" because every generated token enters the gradient under outcome rewards. On a synthetic task ES beat policy gradients at long horizons on sample efficiency, though across 20 real tasks policy gradients were mostly ahead or tied because current benchmarks sit at short effective horizons.

TL;DR Evolution Strategies ES is a scalable zeroth-order alternative to policy-gradient PG RL for agentic post-training: - ES surprisingly does not suffer from the curse of dimensionality in parameter count when finetuning billion-parameter LLMs Qiu et al. 2026 https://arxiv.org/abs/2509.24372 , Liang et al. 2026 https://arxiv.org/abs/2602.00170 . - PG suffers from the curse of horizon: with outcome rewards, every generated token enters its gradient, so long agentic tasks need heavy credit-assignment engineering Tmax https://arxiv.org/abs/2606.23321 . - On a synthetic task, PG wins at short horizons and ES wins at long horizons in terms of sample efficiency . On 20 real tasks, PG is mostly ahead or tied: current benchmarks sit at short effective horizons, where PG is expected to be cheaper, so ES’s advantage should show up on longer-horizon tasks. - In preliminary experiments, fine-tuning Qwen3.5-4B with ES already outperforms the PG recipe Tmax on TBench So freaking simple, embarrassingly parallelizable, no credit assignment, no critic or masking engineering. We think it’s time to revisit zeroth-order methods for long-horizon agentic post-training. Why ask this now Fashion is a cycle; zeroth-order weight-space methods are making a comeback. Sometimes, simple methods just work, and surprisingly well at that another bitter lesson yet . Policy-gradient-based methods PG like GRPO have dominated RL fine-tuning, but they struggle on long agentic tasks with issues like credit assignment over long rollouts, training instability, sensitivity to hyperparameters, etc. Recently, ES has been shown to work for finetuning billion-parameter models, and its performance has been shown to be comparable to that of SOTA PG methods https://arxiv.org/abs/2509.24372 . But if the performance is roughly similar, it is unclear what the key advantage of such zeroth-order methods is, especially considering the fact that in some cases they can be less sample efficient. In this blog post, we revisit the long-standing comparison between ES and PG, particularly in terms of their exploration currency weight-space exploration vs. action-space exploration and the tradeoff in the variances of their gradient estimates, to determine in which regime one surpasses the other. Not to jump the gun, but it actually turns out that ES has many desirable qualities for finetuning on long-horizon tasks. We build on the following observations: - Long-horizon agents are hard for PG. Tmax https://arxiv.org/abs/2606.23321 , T1 https://arxiv.org/abs/2609.11042 and TRACE https://arxiv.org/abs/2607.13988 each needed extra engineering masking, actor-critic, per-action credit to make PG work on Terminal-Bench https://arxiv.org/abs/2601.11868 -style tasks. - Under terminal credit, every generated token enters the PG gradient, including tokens that do not affect the reward; we refer to this as the “ curse-of-horizon .” - ES at Scale Qiu et al., 2026 showed ES can fine-tune 7B models, with mixed results against RL, and follow-up work Liang et al., 2026 posits that the fact that ES can work in such high dimensionality indicates a lack of the traditional “ curse-of-dimensionality .” - The fact that ES bypasses the need for credit assignment, hence avoiding the curse of horizon, and that it does not suffer from the dimensionality curse makes it a stellar candidate for long-horizon agentic post-training. We break down the theory and show concrete results in support of this thesis next. Parameter-space vs. action-space exploration Let’s start by revisiting algos to gain some theoretical intuition behind ES and PG. Assume that you have some underlying reward landscape that you want to hill-climb on. The reward landscape is jagged, and you have two viable options for approximating the gradient or direction s that improve s reward in general; it doesn’t have to be the precise gradient , as illustrated in Figure 1: Figure 1: Parameter-space vs. action-space exploration. a A jagged reward landscape . b ES perturbs the weights, which amounts to climbing a Gaussian-smoothed version of the landscape, too large a σ over-smooths it . c PG keeps the weights fixed and samples actions, which gives noisy returns at a fixed θ. 1 evolution strategies ES : you perturb the weights isotropic Gaussian noise, full-rank or low-rank N times, which forms a “population,” compute the reward corresponding to each perturbed candidate, and perform a weighted average of the perturbations by the outcome rewards to form an update step an estimate of the gradient . Here is the math: where θ are the model weights, σ is the perturbation scale, and is the outcome reward of the i-th perturbed model. 2 policy gradient PG : you sample a trajectory or a bunch of them , and compute the gradient of the output tokens’ log probabilities, weighted by their advantages. Here is the math: where is the j-th trajectory sampled from the current policy with the weights held fixed , G is the number of sampled trajectories, T is the number of generated tokens, and is the advantage of token t in trajectory j. With an outcome reward e.g. GRPO , every token in a trajectory gets the same advantage, i.e. the trajectory’s reward minus the group-mean reward GRPO also divides by the group’s standard deviation . Now if you treat both methods as estimators of one underlying true gradient, see below , that converge in expectation an approximation we will go along with here for the sake of further analysis , then the bulk of the workload shifts to analyzing the variances of the two estimators. In particular, we are interested in how the two variances scale wrt different properties of the tasks. where is the expected reward and is its Gaussian-smoothed version Figure 1b . Why variance analysis If treated as gradient estimators, variances of the methods essentially dictate the sample efficiency/convergence speed of the learning methods. It’s about the signal-to-noise ratio of the learning methods. Assuming the same optimum is attainable by both methods assume this for now, but it’s actually generally not true lol , the variance determines which method is cheaper. Figure 2: The naive curses. a ES variance grows linearly with the parameter count D. b PG variance grows linearly with the number of generated tokens T. c The per-token noise of PG score energy κ stays below 1 for a categorical policy like an LLM, whatever the vocabulary size, but grows with the action dimension for continuous actions. Naively, ES variance grows with parameter count and PG variance grows with sequence length Figure 2 : To put it quite plainly, ES is expected to suffer from the curse of dimensionality while PG suffers from the curse of horizon. However, in practice, this is more nuanced. In fact, only noise injected along directions that change the reward slows learning, so both costs shrink to effective counts. Previously, it was long thought that because of the curse of dimensionality, there is no hope of applying ES to tune LLMs at the billion-parameter level. However, this has been shown not to be true by ES at Scale https://arxiv.org/abs/2509.24372 and a series of follow-up works Sarkar et al., 2026 https://arxiv.org/abs/2511.16652 ; Liang et al., 2026 https://arxiv.org/abs/2602.00170 ; Schweighofer et al., 2026 https://arxiv.org/abs/2605.30148 ; Gan et al., 2026 https://arxiv.org/abs/2603.12228 . Indeed, many of these works point to one reality, which is that finetuning LLMs is surprisingly simple and low-dimensional , and practically many methods can work. So concern for the curse of dimensionality is out the door, great However, things are not looking so rosy for the horizon curse in the case of PG. Even though a reward-irrelevant token adds PG noise only if it gets credit from the reward and its gradient overlaps the reward-relevant directions, in an LLM the overlap is almost to be expected since we generate autoregressively. Moreover, the effective horizon is further determined by how well you can do credit assignment, which becomes increasingly difficult as your horizon grows, unboundedly so for some of the modern agentic tasks. A combination lock to pull the two costs apart To pull the two costs dimension and horizon apart controllably, we design a toy multi-combo lock task. The task is very simple: it is essentially a non-autoregressive -digit multi-combination lock with slots. The policy network here an MLP that takes in a skill-specific position code and outputs logits over -vocab just has to memorize length- sequences; the remaining F slots are filler distractor tokens that do not affect the outcome rewards. The task is illustrated below: Figure 3: The multi-skill combination lock. Each rollout samples one of r independent skills, and a policy with shared weights emits T = k + F tokens from a vocabulary of size V. The reward is the fraction of the k scored decisions that are correct, while the F filler tokens are not scored but still enter the PG gradient. Essentially, by varying number of combos , number of scored slots and number of filler slots , we get to controllably vary the effective dimension and horizon, which allows us to identify regimes under which one method dominates the other in cost. The outcome is quite clean: Figure 4: Which method is cheaper across the number of skills r and the filler length F, with both methods tuned in every cell. Color shows the ratio of the sample costs of PG and ES red: PG cheaper, blue: ES cheaper , and the dashed line marks where the two tie. The tie moves to longer horizons as the number of skills grows. Here, we tune both methods in every setting and record the number of samples each method needs to reach 90% performance. PG is cheaper at short horizons and ES is cheaper at longer horizons. The crossover between the costs of the two methods is determined by a bunch of factors, but, to put it plainly, can be summarized by and : where γ comes from weight sharing, and β from credit assignment. The intuition behind the γ and the β factors is the following: γ measures how much the representations of the filler tokens and the decision tokens overlap, and β measures how much the filler tokens get credited that is, misattributed . | | γ → 0 no overlap | γ 0 shared weights | |---|---|---| | β → 0 per-token credit | free | free | | β ≈ 1 terminal credit | free | paid for | So the horizon tax incurred by the filler tokens is actually discounted by both factors above . Note that this is a simplified expression assuming each filler position contributes uniformly, which is most definitely not the case in general. However, the general-case analysis is of a similar spirit and does not change the conclusions. We also left out a bunch of detailed analysis for the sake of keeping the blog post simple, so interested readers can find out more in our paper link coming soon . Where real tasks sit on the horizon axis Now the real question is where the real tasks sit on this dimension v. horizon phase diagram, and whether their relative positions predict their ES-PG head-to-head performance comparison. To answer it, we needed to approximately measure the effective dimension and effective horizon on these tasks with respect to some base models. Measuring both quantities reliably turns out to be tricky, especially the effective dimensionality. One measure is to approximate the effective dimension by the effective rank of the token log-likelihood gradients on the reference solutions mostly written by humans . This approach, however, picks up mostly on the stylistic mismatch between the base model and the reference solutions. Sparing the details, we therefore only show the horizon of these tasks panel a below and the relative performance diff between ES- and PG-tuned solutions on these tasks vs. their task horizons panel b . Figure 5: Real tasks on the horizon axis. a Sampled tokens per rollout tick and effective horizon marker of each task. b Best held-out score of ES minus that of PG at an equal sample budget, against . Filled markers are our own head-to-heads step size or learning rate tuned for each method , and hollow ones are reported by ES at Scale with their own budgets. PG’s largest leads are at the shortest horizons. We measured the effective horizon of 20 tasks in the setup each one is trained in. The task set includes 12 NLP tasks, 2 agentic environments ALFWorld, WebShop , plus Countdown and five math benchmarks from ES at Scale. All use 7B Qwen2.5 models instruct or math . Terminal-Bench is not included here; we come back to it at the end of this section. Here is how the horizon measurement works: we count each sampled token by how much of its gradient lies in the directions that change the reward, which gives the effective horizon the marker, a lower bound ; the raw generated token count sets the upper cap the tick . Note that this measure is also approximate and is strongly coupled to the base model performance on the particular task, so we use it to rank tasks, not to draw a hard boundary of when one method surpasses the other. As panel a shows, multiple-choice tasks imdb, winogrande, agnews, boolq, arc, and csqa typically require a few effective tokens, other NLP trivia, squad, webq, translate, xsum, hellaswag up to a few dozen, action-only agents webshop, alfworld up to about a hundred, and math a few hundred countdown, minerva, amc, aime, math500, olympiad . Notably, even the longest of them reach only short-to-moderate effective horizons, and panel b shows PG ahead or tied on nearly all of them. This is consistent with the mixed ES-vs-PG GRPO/PPO results reported previously, if these benchmarks sit at effective horizons before the onset of a true ES advantage. We hence expect ES to have more advantage in the longer-horizon regimes. Our first stop on a much longer-horizon task is Terminal-Bench TBench , where an agent works through each task over many turns of reasoning, shell commands and their outputs. We compare ES against Tmax https://arxiv.org/abs/2606.23321 , the open-source recipe for training terminal agents with PG. Note that this comparison is about attainable performance rather than sample efficiency: the two runs did not use matched sample budgets. One catch here is that the Qwen2.5 7B models from Figure 5 cannot or only very, very rarely solve any of the TBench tasks, which leaves neither method a reward signal to learn from, so for this comparison we switch to Qwen3.5-4B. With literally the most basic, vanilla version of ES no credit assignment, no critic, no masking , ES lifts the success rate of Qwen3.5-4B from 12.1% to 21.1%, ahead of the RL checkpoint released with Tmax same base model at 18.2%. ES was trained on the same training tasks as Tmax setup details in the appendix details-on-the-terminal-bench-21-result . Figure 6: Terminal-Bench 2.1 https://www.tbench.ai/news/terminal-bench-2-1 success rate of Qwen3.5-4B before training, after ES fine-tuning, and after RL training with the Tmax https://arxiv.org/abs/2606.23321 recipe the checkpoint released with Tmax . Each checkpoint is evaluated five times on all TBench 2.1 tasks; bars show the mean and error bars the spread over the five evaluations. When to go for ES Other reasons that you might consider ES - It uses only forward passes, stores no activations, and runs perturbed models in parallel on inference engines. - It also repairs confident wrong decisions that PG’s gradient struggles to fix the lock-repair appendix appendix . - It needs no credit assignment, critic, or masking. These properties make ES a promising candidate for post-training in the agentic era. Links and credits Qiyao Liang, Kajetan Schweighofer, Xin Qiu and Risto Miikkulainen Cognizant AI Lab . A preliminary version of this work appeared at the SciFM workshop best paper runner-up at COLM 2026. For more details and citations, please refer to our manuscript coming soon . To cite this blog post in the meantime: @misc{liang2026forget, title = {Forget about credit assignment in long-horizon agentic post-training}, author = {Liang, Qiyao and Schweighofer, Kajetan and Qiu, Xin and Miikkulainen, Risto}, year = {2026}, month = nov, note = {Blog post} } References - Gan, Y., & Isola, P. 2026 . Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights. arXiv:2603.12228 https://arxiv.org/abs/2603.12228 - Ivison, H., Yin, J. O., Shao, R., Xiao, T., Lambert, N., & Hajishirzi, H. 2026 . Tmax: A Simple Recipe for Terminal Agents. arXiv:2606.23321 https://arxiv.org/abs/2606.23321 - Liang, Q., Song, J., Liu, Y., Gore, J., Fiete, I., Miikkulainen, R., & Qiu, X. 2026 . The Blessing of Dimensionality in LLM Fine-tuning: A Variance-Curvature Perspective. arXiv:2602.00170 https://arxiv.org/abs/2602.00170 - Merrill, M. A., Shaw, A. G., Carlini, N., Li, B., Raj, H., et al. 2026 . Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces. ICLR 2026. arXiv:2601.11868 https://arxiv.org/abs/2601.11868 - Qiu, X., Gan, Y., Hayes, C. F., Liang, Q., Xu, Y., Dailey, R., Meyerson, E., Hodjat, B., & Miikkulainen, R. 2026 . Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning. ICML 2026. arXiv:2509.24372 https://arxiv.org/abs/2509.24372 - Qwen Team 2024 . Qwen2.5 Technical Report. arXiv:2412.15115 https://arxiv.org/abs/2412.15115 - Qwen Team 2026 . Qwen3.5-4B. Model card https://huggingface.co/Qwen/Qwen3.5-4B - Sarkar et al. 2026 . Evolution Strategies at the Hyperscale. ICML 2026. arXiv:2511.16652 https://arxiv.org/abs/2511.16652 - Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. 2017 . Proximal Policy Optimization Algorithms. arXiv:1707.06347 https://arxiv.org/abs/1707.06347 - Schweighofer, K., Hayes, C. F., Dailey, R., Miikkulainen, R., & Qiu, X. 2026 . Overcoming Forgetting in LLM Fine-Tuning with Evolution Strategies. arXiv:2605.30148 https://arxiv.org/abs/2605.30148 - Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y. K., Wu, Y., & Guo, D. 2024 . DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300 https://arxiv.org/abs/2402.03300 - Tao, L., Peng, B., Yao, W., Ge, T., Cheng, H., Wang, M. H., Gao, J., & Li, S. 2026 . TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agents. arXiv:2607.13988 https://arxiv.org/abs/2607.13988 - Terminal-Bench Team 2026 . Terminal-Bench 2.1: A revision of Terminal-Bench 2.0 that fixes 28 tasks. Announcement https://www.tbench.ai/news/terminal-bench-2-1 - Yang, J., Shi, Y., Li, Z., Wang, R., Li, Z., Mi, H., & Liang, L. 2026 . T1: Terminal Agent Reinforcement Learning for Long-Horizon Tasks. arXiv:2609.11042 https://arxiv.org/abs/2609.11042 - Yin, O., & Ivison, H. 2026 . TMax v1.1: Improved Data and Stable RL. Blog post https://wai-org.com/blog/tmax-v1-1/ Appendix Lock-and-repair mechanism of ES So one really interesting phenomenon that we noticed was the decision commitment of PG vs. ES. PG methods are known to suffer from mode collapse , which means they lock into policies that emit wrong decision tokens confidently and consistently such that the diversity of output is murdered and there is little chance of getting unstuck. On the synthetic task, where we are able to systematically vary the horizon length and can cheaply afford to tune the HPs of each method, we found that this locking phenomenon is increasingly prominent for task settings with longer horizon higher F and higher dimension higher r . The learning dynamics one color per skill are plotted below for all PG settings, along with their ES counterparts ES only has one row because it does not care about the horizon length . Figure 7: Learning dynamics on the combination lock, with both methods tuned in every setting. The top three rows are PG at increasing filler length F, and the bottom row is ES, which does not depend on F. Columns increase the number of skills r. In each panel, thin colored lines are individual skills, the thick line is their mean, and the dotted line marks the 0.9 target. The orange strip below each panel shows the fraction of decisions locked onto a wrong token. PG’s locked fraction grows with F and r and stays there, while ES’s rises early and then decays as it unlocks them. The only defense for PG in this case is actually to reduce the step size , to slow down the rate of accumulating these locked decision tokens, which inevitably slows down the progress of learning. ES, on the other hand, has the ability to correct itself over the course of learning as shown by the orange panels, which track the fraction of locked tokens over training , which makes it more robust to the HP settings on this task. This indicates that exploration in parameter space vs. action space can allow the two methods to discover fundamentally different solutions. To circle back to the discussion in the main text, variance comparisons only matter for the cost comparison of the methods, but the exploration mechanisms matter for determining the final attainable performance outcomes. Details on the Terminal-Bench 2.1 result The TBench comparison Figure 6 uses Qwen3.5-4B https://huggingface.co/Qwen/Qwen3.5-4B as the base model, the RL checkpoint trained from the same base model and released with Tmax https://arxiv.org/abs/2606.23321 , and our own ES checkpoint. Each checkpoint is evaluated five times on all TBench 2.1 https://www.tbench.ai/news/terminal-bench-2-1 tasks, using Amazon EKS as the sandbox service and serving the models through vLLM. All evaluations use the Vanillux harness from Tmax, without the artificial cap on the number of interaction rounds per task, with a maximum sequence length of 64k tokens and at most 16k per turn. For the RL training setup, see Tmax https://arxiv.org/abs/2606.23321 . ES is trained on the same training tasks TMax-15K https://huggingface.co/datasets/allenai/TMax-15K , also on Harbor-Hub , with the binary pass/fail signal of each task’s verifier scripts as the reward. We use the ES implementation of ES at Scale https://arxiv.org/abs/2509.24372 with population size 32, learning rate α = 0.001, σ = 0.0005 and batch size 256. Training ran on an H200 cluster for 200 iterations, and we select the checkpoint by its performance on a held-out set of 512 Tmax tasks.