# Mo’ turns, mo’ problems

> Source: <https://ianbarber.blog/2026/10/15/mo-turns-mo-problems/>
> Published: 2026-10-16 02:46:12+00:00

Doing reinforcement learning for agentic use-cases is, conceptually, not all that complicated. For each problem you set up a VM with a harness and a fresh model context, send the prompt, let the model make tool calls, score the whole thing at the end, and update the weights.

The important assumption in there is that the model generating the rollout and the model you’re updating are the *same model*, which, for reasons, is referred to as being “on policy”. Most RL algorithms use PPO’s importance ratio: `policy/old_policy`. This scaled the update based on how close the model is to the version that produced the data. On the first update that ratio should be exactly 1: it’s the same weights, seeing the same tokens, so we should get the same answer. Differences between the two makes [Horace sad](https://x.com/cHHillee/status/2109835504506245247).

Also, *agentic* post-training means there is more than the model to consider. Rollouts come from the model, in an inference engine, but also a harness and sandbox environment. These can also produce some weird differences, and as session get longer the chances of hitting a meaningful difference increase. Sessions are getting pretty long these days: In Alibaba’s [QwenGyre](https://arxiv.org/abs/2609.33848) paper rollouts on their 122B model went for around 2 hours on average, while for the 2.4T flagship many of them ran past four hours.

How you train it is all tied up with what you want to do with the model. Like, if you want a model to be good in Claude Code why not train in Claude Code? That’s what Alibaba do, in part because proprietary harnesses…

are closed and evolve rapidly; reimplementing their control flow inside an RL framework is impractical

But! Then its black box. You want to keep your GPUs busy by flexing which ones are training and which are doing rollouts, but if you stall your harness you start getting different outputs for reasons which are nothing to do with the model itself. Alibaba keep their sandbox containers running and route inference through a proxy. They cluster their GPUs into cells which flip rapidly between rollouts and training based on demand. The proxy reroutes in-flight requests and KV cache gets shuttled between cells.

DeepSeek, on the other hand, use their own harness so they can just pause everything. They are also trying to optimize overall throughput, so will routinely preempt long-running rollouts. Before v4.1 the agent loop lived in the (preemptible) GPU job, and had to be reconstructed via a reply when the job came back. With 4.1 they moved the harness inside the sandbox itself and just froze the whole state: DRAM and the writable layer on the file system.

Even when the harness is consistent, you can still go off-policy with the trajectories themselves. In simpler RL-times rollouts were single streams of generation, but tool calls introduced extra tokens that needed to be masked out lest the model train on things it didn’t produce. Even then, if you let the inference server *parse* the models output text you can introduce variations: a tool argument of *false* might become *False*. As we’ve talked about before, the Prime Intellect team spent a lot of time on this in their [Renderers](https://www.primeintellect.ai/blog/renderers) library. QwenGyre’s proxy even caches the original tool-call payloads because the harness reformats them between turns.

Even with that, you increasingly aren’t dealing with one rollout per… rollout. Agents spawn subagents, make parallel tool calls, and compact long sessions.<sup>[1](#fn-300c6e05-18a9-42db-81aa-1457274c0f9f)</sup> What comes back from a long session is a tree, which isn’t all that easy to represent to the trainer.

[Tree Training](https://arxiv.org/abs/2511.00413) serializes the tree into one (or more) streams and weights each token by the percentage of branches that have it in history. QwenGyre caps the number of paths that can be trained on<sup>[2](#fn-e689d182-81c5-4c4d-853e-286a367d470f)</sup>, and averages token weights across rollouts so ones with that spawn a lot of subagents don’t automatically get more gradient.

Finally, even if you have worked hard to get reliable tokens, you have to worry about the operations on those tokens. Because inference is (often!) latency oriented with small batches, and training is throughput oriented with larger batches, you can end up with different kernels for the same jobs, running in different engines: SGLang or vLLM for the rollouts, Megatron or TorchTitan for the training. We’ve talked about the pain of [training-inference mismatch](https://ianbarber.blog/2025/11/03/lets-all-switch-to-fp16/) before, but it gets worse as runs get longer.

One simple fix is to recompute the old log-probs on the trainer. This is brute-forcing the ratio to 1 by going off-policy to the sampler, and it means you have to do an extra forward pass over every rollout. Another approach other is to correct for it: QwenGyre’s trainer clips the ratio per rollout, and caps the weight for each token. This is a bit of a fudge, which, for example, the Megatron [folks](https://github.com/NVIDIA/Megatron-LM/issues/7831) are pointedly unenthusiastic about, as it “hides the mismatch rather than removing it.”

If you can make computation completely bitwise-equivalent, you don’t need to correct or to recompute. A [paper from colleagues of mine at Meta](https://arxiv.org/abs/2609.11356) dug into where bitwise differences come from for floats in a number of kernels: reduction ordering, in large part, but also partial-sum precision, FMAs, and where you round. Fixing the order isn’t free: it can cost 10-20% of performance: Colfax’s recent [FlashAttention-4 work](https://research.colfax-intl.com/optimization-diaries-improving-flashattention-4-backward-for-head-dimension-64/) shows the same on the training side: 1.13x on the head-dim-64 backward pass, but only 1.05x with deterministic accumulation. There are some efforts to build in bitwise-consistency earlier: [AReaL-TIK](https://arxiv.org/abs/2609.35140) is an agentic loop which starts from a single bitwise-consistent kernel shared by inference and training, and then has agents make each faster while maintaining parity.

I do appreciate that agentic post-training is responsible for billions of dollars of revenue, the field being renamed superintelligence, etc., but it probably helps that it is a fun ball of mud to work through too.

1. You want to train the summarization too  so the model learns to compact well. [CompactionRL](https://arxiv.org/abs/2607.05378) has a good recipe.[↩︎](#fn-300c6e05-18a9-42db-81aa-1457274c0f9f-link)
2. A max of 5, ordered by main agent, its summaries, subagents, their summaries. [↩︎](#fn-e689d182-81c5-4c4d-853e-286a367d470f-link)