Harness Training A developer known as workofart has created a PyTorch-like framework for training AI agent harnesses, achieving a 45-minute experiment cycle on terminal bench tasks by enforcing deterministic inference via SGLang on NVIDIA 5090 GPUs. The framework treats the harness as trainable weights, using an improvement agent (Codex or Claude Code CLI) as a gradient estimator and a learning memo as optimizer state, enabling recursive self-improvement of the system that wraps around a frozen LLM backbone (Qwen 3.6 35B A3B FP4). Project Repository: https://github.com/workofart/harness-experiment https://github.com/workofart/harness-experiment So I recently wanted to see whether an AI agent could self-improve a harness to solve terminal bench https://www.tbench.ai/ tasks. To align on the definitions, “harness” means the system e.g. Claude Code, Codex, ChatGPT web interface etc… wrapping around the model e.g. GPT-5.5, Claude Opus 4.7 etc… that interacts with a specific environment. The harness controls what the model sees, what tools the model can use, and how environment responses are fed back to the model etc… My initial attempt was mostly treating the self-improvement loop as an experiment design problem. I wrote about the learnings in my previous blog post What 1,000+ Harness Experiments Taught Me About Self-Improving Agents https://www.henrypan.com/blog/2026-05-25-self-improvement-harness/ . Since then I’ve completely revamped the way I run experiments hint: determinism . This allowed me to reduce the experiment time, to extract more signal per experiment and to build a general PyTorch-like framework general-framework for “training harnesses”. In this blog post, I’m going to talk about how this is done and why it matters in the context of recursive self-improvement. Here are some results from evaluating the same trained harness + model X: Training the Harness For a precise definition, this problem is framed as an agent-guided discrete program search with terminal reward and persistent search memory. Here’s a map of the recurring concepts and the PyTorch analogy that inspired the framework introduced later general-framework : | Concept | PyTorch/Machine Learning analogy | |---|---| Harness core.py : The program being trained. The current best version is the baseline; a proposed edit to the python file is a candidate | Model weights | Improvement agent Codex CLI or Claude Code CLI : Reads the experiment evidence, proposes one bounded harness change per epoch iteration AgenticEstimator in the framework | Gradient estimator | Task LLM: Executes the training tasks through the harness; never changes during training | Frozen backbone | Criterion: Compares candidate vs. baseline task outcomes to determine whether the candidate was good | Loss function | Optimizer: Promotes the candidate to new baseline git fast-forward or rejects it preserved as a git ref | optimizer.step | Experiment run: Full execution of the training task set under one candidate | Training epoch | Learning memo learning.md : Persistent memory across epochs: what worked, what didn’t, what to try next | Optimizer state | Setup My setup is pretty basic, mostly to control costs and reach a decent iteration speed of one experiment 30 - 40 tasks within ~45 minutes: - Rented 5090 GPUs in the cloud 1 - 4 depending on how many concurrent tasks I’m running . The LLM inference providers don’t guarantee determinism, which I will explain later determinism . : The model weights occupy ~24 GB, while the KV cache with radix/prefix caching disabled uses ~4 GB to support up to 10 concurrent requests, a limit imposed by the deterministic kernel discussed later . The remaining memory is consumed by activations and CUDA overhead. Qwen 3.6 35B A3B FP4 quantized model https://huggingface.co/nvidia/Qwen3.6-35B-A3B-NVFP4 LLM inference configurations: seed 12345, temp 1.0, top-p 0.95, 32k context length, max token count 8k, reasoning token count 6k. These were mostly constrained by GPU memory and aiming decoding time of below 5 minutes for the worst-case max token count, and I didn’t want to split the model across GPUs SGLang inference engine: it supports deterministic batch inference, which I will explain later determinism why it’s a prerequisite for training the harness bootstrap script https://gist.github.com/workofart/4787e28dd384d3c75b17f1ee62e611bb Next, I will dive into the two types of tasks where I trained my harness: SWE bench tasks and Terminal Bench tasks. Baseline Harness It’s important to mention how I define my baseline harness because this is kind of like how one would initialize a machine learning model during model training. And it is very important to initialize your harness properly because otherwise it’s going to be very hard for the harness training to find good learning signals gradient . For example, I’ve seen that if the baseline harness is too thin, without the LLM response handling logic, most of the training will be spent on creating error handling for the LLM calls. The other extreme is when the baseline harness is overloaded with 10+ tool definitions, at that point there’s too much bias. Maybe certain models are trained with that, but it’s definitely not a generic starting point. In my baseline harness https://github.com/workofart/harness-training/blob/0900e8acc124caa8b4c7354cfeeb6750873d395e/src/policy/core.py L97 , I defined two actions in total for tool use: Run for running any shell command and Submit for letting the harness know it’s ready to call the task verifier to check the results. By no means is this a good baseline, but it’s a clean yet learnable state. Training on SWE Bench Tasks Training Setup: - Trained on 39 SWE-bench tasks django/pylint/pytest/sphinx/sympy 1 fn:training-SWE-Bench-tasks - Capped at max 110 steps per task or 75 mins timeout per task - Frozen Task LLM: Qwen 3.6 35B A3B FP4 quantized model https://huggingface.co/nvidia/Qwen3.6-35B-A3B-NVFP4 - Agent used as src/trainer/estimator.py:AgenticEstimator for training the harness: GPT-5.5 high via Codex CLI Git branch https://github.com/workofart/harness-training/tree/v2-exp-swe-0706 for this experiment with all the candidate commits from training Result: 8/39 - 14/39 solved over 29 experiment runs: 8 promoted, 20 rejected, 1 baseline. All 8 promoted harness changes The secondary reward is a tie-breaker used only when a candidate and its baseline solve exactly the same tasks. | | Commit | Mechanism | New tasks solved | |---|---|---|---| | 1 | 84241c31 | Reminder at step = 100: “only files on disk at submit are graded. If your fix is written, submit now” | 8 to 10: pytest-7490, sphinx-10673 | | 2 | ca412ea7 | Action guard at step = 109: force a bare submit if edits exist on disk and none was submitted | 11: sympy-12419 | | 3 | eb658d1a | Reasoning-runaway breaker changed from abort to degrade: drop thinking instead of raising RepeatedLengthCutoffError . A discarded trial can’t be graded, a degraded one can | 12: pylint-6528 | | 4 | 92e6c60d | Reminder at steps 70-99 if on-disk diff is empty: stop exploring, write the fix | 12 secondary reward of 53 verifier unit test improvements across 2 tasks | | 5 | 0096fce8 | forced git diff --stat / --check self-review “keep the production diff minimal” when agent’s first submit is between steps 60-99 and on-disk diff touches at least two files | 12 secondary reward of improving valid tool calls on first attempt from 99.32% to 99.36% | | 6 | 527712d4 | Git-state guard: once an edit is on disk, git stash / checkout -- / restore / reset --hard / clean are swapped for a safe diff-print – agents were wiping their own fix while “cleaning up” | 13: pytest-7205 | | 7 | 2948bdd2 | Pre-submit cleanup of generated artifacts repro scripts, build junk so they don’t pollute the graded diff | 13 secondary | | 8 | 292e6ff6 | Rewrite pytest ... \| tail as bash -o pipefail -c '...' – tail’s exit code 0 was masking failing test runs, so agents believed broken patches passed | 14: sphinx-10323 | There were 21 tasks out of 39 tasks that were never solved in any of the 29 runs. Based on experiment artifacts, a third of them hit the 110 step-cap, which I enforced to speed up the training iterations. But there’s definitely a trade-off here where if I extend the step cap, we can collect more training signals, but each epoch will run longer. Or maybe there’s some model capability ceiling or my LLM inference resources were too limited 2. Training on Terminal Bench Tasks Training Setup: - Trained on 38 terminal bench 2.0 tasks . The tasks were selected to remove cyber/cryptography tasks since certain models refuse to complete it. And it was narrowed down to a subset to maximize both variety/difficulty and minimize task completion time. 3 fn:training-terminal-bench-tasks - capped at max 110 steps per task or 30 mins timeout per task - Frozen Task LLM: Qwen 3.6 35B A3B FP4 quantized model https://huggingface.co/nvidia/Qwen3.6-35B-A3B-NVFP4 - Agent used as src/trainer/estimator.py:AgenticEstimator for training the harness: GPT-5.5 high via Codex CLI Git branch https://github.com/workofart/harness-training/tree/v2-exp-tb-0706 for this experiment with all the candidate commits from training Two-stage experiment runs show non-deterministic factors hinder the self-improving capability You can see in the plot below that I reset the baseline and dropped all the promoted harness changes around the 26th experiment due to identifying some non-deterministic environment factors deterministic-environment that polluted the experiment results. You can even see that the rejected candidates before the reset had a very narrow but steady variance. After the non-determinism fix deterministic-environment and restart, the harness was able to quickly find good changes and beat the previous best score in less than half of the experiment time. All 9 promoted harness changes | | Commit | Mechanism | New tasks solved | |---|---|---|---| | – | pre-determinism baseline 18/38 | || | 1 | 927405ea | replayed write content=... args capped at 250 tokens file is on disk anyway | 19 total +1 : sparql-university | | 2 | b5e204fa | Drop empty junk args from tool calls before validation | 19 total: nginx-request-logging net 0 against the previous baseline | | 3 | cd2e2ad0 | If at step =107 and last 8 actions are all read-only probes, nudge with prompt “convert hypothesis into on-disk edits and submit” | 19 total same solved set; secondary tie-break improved | | – | determinism-hardening reset; re-baseline 16/38 | || | 4 | a70ec204 | Force bare submit at step =108 if edits exist on disk | 17 total +1 : regex-log | | 5 | a0dccbb4 | On llm request validation error, strip only extra forbidden fields and re-validate once | 17 total same solved set; first-try valid tool calls improved from 97.78% to 97.95% | | 6 | 925f790a | Steer oversized-call cutoff from the 2nd occurrence on 1st abort stays byte-identical | 17 total same solved set; first-try valid tool calls improved from 97.95% to 98.29% | | 7 | 14ef7381 | if llm response had a length cutoff with no tool call inside, disable thinking and steer with prompt | 22 total +5 : code-from-image, distribution-search, extract-elf, llm-inference-batching-scheduler, overfull-hbox | | 8 | 375c3099 | LENGTH CUTOFF ABORT THRESHOLD from 2 to 3 | 23 total +1 : custom-memory-heap-crash | | 9 | ca49ae96 | if llm response cutoff with partial tool call, treat as oversized reuse split-it repair , not malformed | 23 total same solved set; first-try valid tool calls improved from 97.62% to 97.69% | Note on the tie-breaker promotion: the harness training framework general-framework allows users to define secondary reward https://github.com/workofart/harness-training/blob/0900e8acc124caa8b4c7354cfeeb6750873d395e/src/rollout/metrics.py L18 to break primary loss function ties. In the above training run, the secondary reward https://github.com/workofart/harness-training/blob/0900e8acc124caa8b4c7354cfeeb6750873d395e/src/rollout/metrics.py L18 is defined as the following, evaluated in order: - % of tool calls that are valid on the first attempt - the number of steps used to solve the task Biggest jump 14ef7381, 17 to 22 solves : entire LLM output response budget burned on reasoning with no tool call emitted, which previously killed the trial. The harness change proposed was to disable thinking and inject the repair prompt below. The 5 converted solved tasks were all long-horizon solved in 67-90 steps . reasoning runaway repair prompt = "Your previous response reached the output length limit while still " "reasoning, before it emitted any tool call. Stop reasoning now and reply " "with a single, concise tool call and no other text." Evaluation Results Is training the harness actually worthwhile to do? In this section, I’m going to go through some of the evaluation runs by holding the trained harness fixed and swapping out the underlying task LLM model. Here are the evaluation configurations. Trained training-on-terminal-bench-tasks on Terminal Bench 2.0 38 tasksout of 89 tasks. 3 fn:training-terminal-bench-tasks Git branch https://github.com/workofart/harness-training/tree/v2-exp-tb-0706 with all the trained harness commits.- Evaluated on Terminal Bench 2.0 all 89 tasks. It’s obvious that there is overlap between the training set and the evaluation set. 23 out of 38 tasks were solved by Qwen 3.6 35B A3B FP4 quantized model https://huggingface.co/nvidia/Qwen3.6-35B-A3B-NVFP4 with the trained harness, and it didn’t include task-specific mechanisms to “cheat”. We will also show that evaluating the same Terminal Bench-trained harness against SWE-bench, there are transfer learning gains too. | Config field | DeepSeek V3.2 | MiniMax M2.5 | GPT-OSS 120B | GPT-OSS 20B | MiMo V2.5 | GPT-5.5 | |---|---|---|---|---|---|---| | Terminal Bench Settings | Same as Official Terminal Bench 2.0 settings | Same as Official Terminal Bench 2.0 settings | Same as Official Terminal Bench 2.0 settings | Same as Official Terminal Bench 2.0 settings | Same as Official Terminal Bench 2.0 settings | Same settings; 76-task panel, the remaining 13 the API rejects | | max context length | 128K | 205K | 128K | 128K | 1M | 256K | | max tokens | 30K | 30K | 30K | 30K | 30K | N/A using Codex Backend API | | reasoning effort | high | high | high | high | high | xhigh | | temperature | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | N/A using Codex Backend API | | top p | 1.0 | 1.0 | 1.0 | 1.0 | 0.95 recommended by model card | N/A using Codex Backend API | Generalized and transfer learning capability This shows that for a frozen harness that was trained once using one model Qwen 3.6 35B A3B FP4 quantized model https://huggingface.co/nvidia/Qwen3.6-35B-A3B-NVFP4 , it can lift up the capability of a wide variety of models. It also shows that if we train a harness on one task set SWE-Bench 1, that same harness can improve task-solving capability towards a different task set Terminal Bench . 4 fn:evaluation-terminal-bench-tasks New capability There are a couple of mechanisms in the trained harness that contribute to better task-solving abilities In this plot below, you can see: - “failed verification” converted to “solved” +23 because the trained system prompt added “run output is scratch: editing a file inside a python snippet or a here-doc that only prints does NOT persist your fix” and “the on-disk state at submit is the only thing graded” This targets a baseline failure mode where a model wrote the code in some ephemeral output and submitted an unchanged disk state. Inside the 23 converted “solved” tasks, they used the new file tools ~300 of ~1200 calls . DeepSeek 3.2 is the biggest beneficiary +7 . - “ran out of time” converted to “solved” +24 from the trained harness spending half or less of the baseline’s LLM latency. Specifically, GPT-5.5 headless-terminal task reduced from 1755s to 221s, DeepSeek compile-compcert reduced from 1538s to 165s. The trained harness mechanisms that contributed were:- read clipped to 250 lines to cap environment stdout/stderr bloat - shorter completions for several models output tokens per call: GPT-5.5 −19%, MiniMax2.5 −22%, GPT-OSS 20B −11% - if the “Run” command/tool call times out, prompt the LLM with a nudge “change your approach rather than rerunning”, which avoids repeated 300s stalls. - “no valid tool call repeatedly” converted to “solved” +6 , because in the trained harness, more file tools were introduced added “read”, “write”, “replace” to the existing “read”, “submit” . More specifically, the “write” and “replace” that persist edits quote-safely via base64, which improved “missing tool call” recovery from 69% to 94%, despite a byte-identical repair prompt. For invalid calls 5 fn:missing-tool-call , the trained harness repair echoes the failed call back as something like “You sent: {calls}”, which improves a ValidationError 6 fn:validation-error recovery from 51% to 83%. 6 fn:validation-error If we combine the above figure with the below figure, the “hit timeout” to “solved” conversion would organically cause high step counts, since the same tasks that previously ran out of wall-clock are now being solved, which is driven by efficiency. MiniMax 2.5 has near-zero parse failures in both the trained and baseline harnesses, yet still gains +7 solves beyond 50 steps. Five are timeout/verification flips, suggesting the gains come from “persisting fixes” 7 and write/replace , not parsing. For GPT-OSS 20B, the fatal step rate fell from 9.4% to 1.9%, allowing trajectories to run longer attempted steps increased from 5.6k to 9.5k and enabling some runs to convert late. Tool call failures per step remained essentially unchanged at ~31%. This means that failures survive in the trained harness, and the surviving steps bill 43% more output tokens. The other model family’s long-trajectory wins come from the “timeout” and “failed verification” to “solve” mechanisms. GPT-OSS 120B gains solves while its trajectories shrink. This again shows that each model really has a different failure model or task-solving strategy baked into the weights. 8 fn:write-replace If we decompose “Task Solve” mathematically as \ \text{Submitted} \wedge \text{Passed Verification}\ , the trained harness could only achieve its goal through those two lenses, and that’s what it did. “Submit better” comes from persisting fixes 7, plus the trained harness’s longer, more careful trajectories filtering weak submissions out of the pool. For DeepSeek 3.2, pass rate also rose as failing submissions stopped reaching submit at all. Only MiniMax 2.5 netted solves from submitting more, at the cost of a lower pass rate. Token efficiency For the genuine token efficiency wins, since the trained harness introduced more file tools, it shifted the tool call mix toward cheap calls. In GPT-5.5’s trained harness evaluation, read averages 433 output tokens vs 1,013 for the untrained baseline harness evaluation, and 374 of ~2,200 calls moved to read/replace tokens per call 1,422 decreased to 1,151 . There are essentially two camps here: - With GPT-OSS 20B as the representative, its context blowup is rescued by the trained harness survival mechanism, so the task can still be solved at longer steps, but they pay for it using more input tokens baseline steps ▓▓▓▓▓ breaker trips ~step 5, low recovery — dies input ██ context never gets big trained steps ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ recovers 94%, runs on input ████████████████████████████ each step re-reads more attempted steps 5618 grows to 9472 total prompt +110% input/solve +73% output/solve +18% total output +43% - With GPT-OSS 120B as the representative, its trajectories shrank while solving more, which is shown by the cost savings on the output tokens side. baseline steps ▓▓▓▓▓▓▓▓▓▓▓▓▓▓ wanders longer before solving input ██████████████████ trained steps ▓▓▓▓▓▓▓▓ tighter path to submit input ███████ attempted steps 5834 decreases to 4290 input/solve −55% output/solve −35% Tokens-per-solve falls because trained harness’s first-order effect was to increase solve, which increased the denominator, which can be seen for most cases in the below figure. We see divergence baked into the weights for token efficiency as well. In terms of the cost of solving a task, the two GPT-OSS models from the same family sit at opposite extremes. 120B reduced calls per solve by 36% through fewer model calls, while output tokens per call remained nearly flat +1% . By contrast, 20B increased calls per solve by 32%: longer survival task solves outweighed an 11% reduction in output tokens per call. Note that “model call retry %” didn’t change much: 20B 31% to 27%, 120B 9% to 12%, everyone else shows less than 1% gap between the baseline and trained harnesses. So \ \frac{\text{model calls}}{\text{solve count}}\ are driven mostly by \ \frac{\text{trajectory length}}{\text{solve count}}\ , not repair traffic. MiMo 2.5 behaves the opposite from GPT-5.5. MiMo 2.5 is the only model whose calls got longer +19% per call while getting fewer −8% calls/solve . This means the model does more deliberation per step at fewer steps, net +9% pricier per solve. GPT-5.5 did the reverse −19% per call, +2% calls . A rising useful-share can mean opposite things per model: - MiniMax 2.5’s biggest jump, where token shares tripled from 13% to 41% but only on +2 solves, is driven by tokens spent on solved tasks exploding from 12M to 56M, where 36.5M of that is just two tasks winning-avg-corewars 21M, fix-ocaml-gc 15.5M, the 242-step timeout to solved flip . Its failed-spend barely moved. Maybe this harness is not suitable for MiniMax 2.5 on those tasks? - For GPT-OSS 120B, the “failed-spend” fell from 221M to 98M −56% , it’s mostly because failure itself got cheaper shorter trajectories before dying or timing out . This is because of the same trajectory-shrinking mechanism in the trained harness. 9 fn:trajectory-shrink-mechanism - For GPT-OSS 20B, it’s an interesting one. Keeping 20B alive poured an extra 192M tokens into tasks that still failed versus extra 16M onto solved tasks. When combined with the above analysis for this model, there’s definitely some capability issue on the model side that’s making it inefficient to rescue. Or if there was a better way to detect this early on, we can let the harness “give up early” when running with less capable models. Note on interpreting the numbers: “Figure: Efficiency Levers” and “Figure: Useful Share” denominators mix failed-task spend into per-solve/share metrics, so every number inherits the composition effects we talked about earlier which tasks die, time out, or reach submit . “Figure: Efficiency Levers”’s identity per-solve = per-call calls/solve is exact, but neither panel isolates “the same task got cheaper”. Determinism In order to achieve the evaluation results above with the trained harness, we need to understand the core prerequisite of training a harness. Remember that I initially ran a lot of experiments without considering this factor and used multiple trials to “gain statistical power”? Well, that’s not scalable for this project where I don’t have enough compute power to speed up the experiment cycles. So I changed the way I run experiments by guaranteeing determinism. What do I mean by determinism ? Given the same setup, configs and harness etc…, I expect the exact same logits returned from the task LLM, same output tokens, same reasoning tokens and executing the exact same tool calls against the environment, returning the exact same results from the environment and reaching the exact same conclusions. To motivate this path, let me share my initial attempt without determinism. Motivation in the self-improvement loop context Before I worked on guaranteeing determinism, I ran over 1000 experiments. During those runs, I saw a lot of experiment noise that caused tasks to flip from “solved” to “unsolved” even though the experiment treatment harness didn’t change the critical path for that particular task. I tracked every mechanism firing with a 1-1 mapped metric to prove this. As a result, I spent a lot of time in designing a “good” candidate promotion criteria https://www.henrypan.com/blog/2026-05-25-self-improvement-harness/ 3-how-to-judge-progress-candidate-promotion . This example below is the LLM responses being non-deterministic: - task: swebench django-14534 - model: gpt-5.5 - input: identical 751-token prompt, identical tool-schema, fingerprint 1000b896960b , request key 0b4c96fa…3cad, sent 39 times - output: run A exp-20260614-221243, step 4 : list dir {"path": "/testbed"} run B exp-20260615-022510, step 4 : search text {"query": "class BoundWidget", "root": "/testbed"} At the experiment trial-level, trials are not deterministic, so experiment outcome depended entirely on coin flips: Ok, so we saw the initial experiment setup was non-deterministic, but why do we need determinism here? Imagine this: if a harness change proposed by the harness training loop had a very small effect towards a very small number of tasks and behavior change, from a statistics perspective, many repeated trials are needed to determine whether the “solve” was noise or a true effect of the treatment. But even then, that only provided confidence intervals, not a guarantee of a “good” harness change. So is non-determinism bad? Not necessarily. Another way to think about this problem is: determinism is useful for credit assignment but once our policy harness is frozen, non-determinism can help with real-world performance by avoiding getting stuck in a local optimum. I came to this realization through a random thought I had one day: Why do Reinforcement Learning RL environments like https://github.com/openai/gym https://github.com/openai/gym iterate through each experiment so fast and allow for stable learning signals from each iteration? I realized the key missing component was determinism. If we can ensure determinism, then every experiment outcome can be completely traced back to harness change solely. In the RL framing, this is like making credit assignment cleaner. N=1 You might have seen that standard agentic evaluations need pass@k over many seeds to fight noise. That’s because the evaluations need to reflect real-world use-cases, and non-determinism is present in nearly all of those use-cases. But in our harness training case, our goal is to use determinism to find a good harness without the noise. Once we commit to a finalized harness frozen , we can use it with non-deterministic environments and LLMs. Now that we’ve motivated the determinism efforts, let’s talk about the two factors that we need to take into account: LLM inference determinism and environment determinism. I will go over each one. Deterministic LLM Inference When we talk about LLM inference determinism, we’re usually talking about the deterministic logits output for the same model serving settings e.g. LLM checkpoint, inference server version etc… . Note that “temperature” and other sampling args like “top k” affect how we sample the tokens from a given logits/softmax distribution, but they don’t affect how those logits are generated by the model in the first place. The root cause of non-determinism There is a great blog post Defeating Nondeterminism in LLM Inference https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/ 11 on this exact topic which I’ve learned a lot from, so I won’t go into details. An over-simplified explanation to motivate the next section of the blog Recall that LLM inference forward pass is a bunch of matrix multiplication to produce a set of logits, which then gets converted into a probability distribution over the vocabulary space, then sampled to get a specific token output. Well, the matrix multiplication part doesn’t always produce the same set of logits, given the exact same input tokens. The reason is because floating point number addition is non-associative dot products involve additions . This problem can be amplified by factors like batch shapes, which in turn affects how LLM kernels schedule these matrix operations. Different order - different rounding due to precision - different logits - different probability distribution - different final token even with temperature = 0 . \ a + b + c \ne a + b + c \ For example, in float32 , let Then \ a+b +c = 0+1 = 1 \\ a+ b+c = 10^8 + -10^8+1 \ Since 1 is too small to change \ -10^8\ in float32 , Therefore: \ a+b +c = 1 \\ a+ b+c = 0\ The reason why non-determinism is the default for many inference engines is because it’s often more efficient to avoid the special handling required to achieve determinism. Oftentimes it requires special kernel implementations https://github.com/flashinfer-ai/flashinfer/pull/1675 for determinism guarantees. LLM inference providers don’t guarantee determinism Initially, I used different inference providers to train my harness for the task LLM. But after trying out many providers, I realized determinism was only best-effort, never guaranteed. There are a couple of reasons that I found: Multi-tenant inference. We’ve talked about how different batch shapes would cause the matrices to be multiplied in different order, which causes non-determinism. This is amplified by the fact that LLM providers need to batch many requests from many users together to save cost fully utilize GPU memory .- . Some providers support it, but some don’t. But even if they do, they never guarantee it seed parameter for inference is not guaranteed for example https://console.groq.com/docs/prompting deterministic-outputs-with-seed .Determinism is best-effort and is not guaranteed across model versions Rate limits / infra issues can cause the task-solving trajectory to diverge or even crash the run due to timeouts What I used instead For LLM inference options, in a naive case, if you just call the model’s forward pass via PyTorch, you would get deterministic results, assuming you have set all the seed values correctly. But if you start to use these inference engines like vLLM or SGLang, they have a lot more optimizations done to improve the LLM inference throughput and memory footprint that break determinism. I eventually decided to go with SGLang as the inference engine, as it’s the one that explicitly said deterministic inference https://docs.sglang.io/docs/advanced features/deterministic inference sglang%E2%80%99s-solution is supported by a couple of attention backends. I’ll leave out the experiment notes for tuning the inference engine since it’s too distant from this “harness training” project. The main takeaways are: - Setting --enable-deterministic-inference and setting a fixed seed is the first thing that I did. - mixed-length requests by themselves don’t break determinism - client concurrency itself doesn’t break determinism, it’s more about how the requests are batched - even with radix caching disabled, the max number of active request slots Mamba-slot cap for Qwen3.6-35B-A3B-NVFP4 model’s State Space Model still has a ceiling before non-determinism slips in. In this case, active request slots had to be <= 10 to guarantee determinism for my setup 12 fn:setup bootstrap script https://gist.github.com/workofart/4787e28dd384d3c75b17f1ee62e611bb LLM Determinism costs throughput degradation In the below plot, you can see that we can guarantee determinism up to 10 concurrent requests on this diagnostic box, but after that, we observe drift. This also means that we lose out on higher throughput. Some throughput levers I tested that failed the determinism gate: torch.compile incompatible with batch-invariant ops- bf16 mamba dtype improves throughput by +12.7% but breaks determinism - radix cache fails mixed-prefix gate - NGRAM spec decoding wildly nondeterministic Deterministic environment Another factor that we’ve taken for granted is that the environment the LLM is interacting with input to the LLM is deterministic, which doesn’t always hold. We can see this in the example below. Setup: temp=1.0, top p=1.0, seed=12345, sampling from the exact same deterministic logits from the deterministic LLM inference engine Both runs made the same tool call at step 15. Everything before step 15 was completely identical on both the LLM and environment side. e = Exists Product.objects.all q = Q result = e & q print "Exists & Q:", result But the environment returned different stdout back to the LLM, due to memory addresses being random. Run A env stdout : Exists & Q: AND: