Harness Training

Project Repository: https://github.com/workofart/harness-experiment

So I recently wanted to see whether an AI agent could self-improve a harness to solve terminal bench tasks. To align on the definitions, “harness” means the system (e.g. Claude Code, Codex, ChatGPT web interface etc…) wrapping around the model (e.g. GPT-5.5, Claude Opus 4.7 etc…) that interacts with a specific environment. The harness controls what the model sees, what tools the model can use, and how environment responses are fed back to the model etc…

My initial attempt was mostly treating the self-improvement loop as an experiment design problem. I wrote about the learnings in my previous blog post What 1,000+ Harness Experiments Taught Me About Self-Improving Agents.

Since then I’ve completely revamped the way I run experiments (hint: determinism). This allowed me to reduce the experiment time, to extract more signal per experiment and to build a general PyTorch-like framework for “training harnesses”. In this blog post, I’m going to talk about how this is done and why it matters in the context of recursive self-improvement.

Here are some results from evaluating the same trained harness + model X:

Training the Harness #

For a precise definition, this problem is framed as an agent-guided discrete program search with terminal reward and persistent search memory.

Here’s a map of the recurring concepts and the PyTorch analogy that inspired the framework (introduced later):

Concept	PyTorch/Machine Learning analogy
Harness (core.py ): The program being trained. The current best version is the baseline; a proposed edit to the python file is a candidate
Model weights
Improvement agent (Codex CLI or Claude Code CLI): Reads the experiment evidence, proposes one bounded harness change per epoch (iteration) (AgenticEstimator in the framework)
Gradient estimator
Task LLM: Executes the training tasks through the harness; never changes during training
Frozen backbone
Criterion: Compares candidate vs. baseline task outcomes to determine whether the candidate was good
Loss function
Optimizer: Promotes the candidate to new baseline (git fast-forward) or rejects it (preserved as a git ref)
optimizer.step()
Experiment run: Full execution of the training task set under one candidate
Training epoch
Learning memo (learning.md ): Persistent memory across epochs: what worked, what didn’t, what to try next
Optimizer state

Setup

My setup is pretty basic, mostly to control costs and reach a decent iteration speed of one experiment (30 - 40 tasks) within ~45 minutes:

• Rented 5090 GPUs in the cloud (1 - 4 depending on how many concurrent tasks I’m running). The LLM inference providers don’t guarantee determinism, which I will explain later. : The model weights occupy ~24 GB, while the KV cache (with radix/prefix caching disabled) uses ~4 GB to support up to 10 concurrent requests, a limit imposed by the deterministic kernel(discussed later). The remaining memory is consumed by activations and CUDA overhead.Qwen 3.6 35B A3B FP4 quantized modelLLM inference configurations: seed 12345, temp 1.0, top-p 0.95, 32k context length, max token count 8k, reasoning token count 6k. These were mostly constrained by GPU memory and aiming decoding time of below 5 minutes for the worst-case max token count, and I didn’t want to split the model across GPUsSGLang inference engine: it supports deterministic batch inference, which I will explainlaterwhy it’s a prerequisite for training the harness (bootstrap script)

Next, I will dive into the two types of tasks where I trained my harness: SWE bench tasks and Terminal Bench tasks.

Baseline Harness

It’s important to mention how I define my baseline harness because this is kind of like how one would initialize a machine learning model during model training. And it is very important to initialize your harness properly because otherwise it’s going to be very hard for the harness training to find good learning signals (gradient). For example, I’ve seen that if the baseline harness is too thin, without the LLM response handling logic, most of the training will be spent on creating error handling for the LLM calls. The other extreme is when the baseline harness is overloaded with 10+ tool definitions, at that point there’s too much bias. Maybe certain models are trained with that, but it’s definitely not a generic starting point.

In my baseline harness, I defined two actions in total for tool use: Run

for running any shell command and Submit

for letting the harness know it’s ready to call the task verifier to check the results. By no means is this a good baseline, but it’s a clean yet learnable state.

Training on SWE Bench Tasks

Training Setup:

• Trained on 39 SWE-bench tasks (django/pylint/pytest/sphinx/sympy) 1 - Capped at max 110 steps per task or 75 mins timeout per task
• Frozen Task LLM: Qwen 3.6 35B A3B FP4 quantized model - Agent used as src/trainer/estimator.py:AgenticEstimator

for training the harness: GPT-5.5 high via Codex CLI Git branchfor this experiment with all the candidate commits from training

Result: 8/39 -> 14/39 solved over 29 experiment runs: 8 promoted, 20 rejected, 1 baseline.

All 8 promoted harness changes

The secondary reward is a tie-breaker used only when a candidate and its baseline solve exactly the same tasks.

#	Commit	Mechanism	New tasks solved
1	84241c31
Reminder at step >= 100: “only files on disk at submit are graded. If your fix is written, submit now”	8 to 10: pytest-7490, sphinx-10673
2	ca412ea7
Action guard at step >= 109: force a bare submit if edits exist on disk and none was submitted
11: sympy-12419
3	eb658d1a
Reasoning-runaway breaker changed from abort to degrade: drop thinking instead of raising RepeatedLengthCutoffError . A discarded trial can’t be graded, a degraded one can
12: pylint-6528
4	92e6c60d
Reminder at steps 70-99 if on-disk diff is empty: stop exploring, write the fix	12 (secondary reward of 53 verifier unit test improvements across 2 tasks)
5	0096fce8
forced git diff --stat / --check self-review “keep the production diff minimal” when agent’s first submit is between steps 60-99 and on-disk diff touches at least two files
12 (secondary reward of improving valid tool calls on first attempt from 99.32% to 99.36%)
6	527712d4
Git-state guard: once an edit is on disk, git stash /checkout -- /restore /reset --hard /clean are swapped for a safe diff-print – agents were wiping their own fix while “cleaning up”
13: pytest-7205
7	2948bdd2
Pre-submit cleanup of generated artifacts (repro scripts, build junk) so they don’t pollute the graded diff	13 (secondary)
8	292e6ff6
Rewrite pytest ... | tail as bash -o pipefail -c '...' – tail’s exit code 0 was masking failing test runs, so agents believed broken patches passed
14: sphinx-10323

There were 21 tasks out of 39 tasks that were never solved in any of the 29 runs. Based on experiment artifacts, a third of them hit the 110 step-cap, which I enforced to speed up the training iterations. But there’s definitely a trade-off here where if I extend the step cap, we can collect more training signals, but each epoch will run longer. Or maybe there’s some model capability ceiling or my LLM inference resources were too limited 2.

Training on Terminal Bench Tasks

Training Setup:

• Trained on 38 terminal bench 2.0 tasks . The tasks were selected to remove cyber/cryptography tasks since certain models refuse to complete it. And it was narrowed down to a subset to maximize both variety/difficulty and minimize task completion time.3 - capped at max 110 steps per task or 30 mins timeout per task
• Frozen Task LLM: Qwen 3.6 35B A3B FP4 quantized model - Agent used as src/trainer/estimator.py:AgenticEstimator

for training the harness: GPT-5.5 high via Codex CLI Git branchfor this experiment with all the candidate commits from training

Two-stage experiment runs show non-deterministic factors hinder the self-improving capability

You can see in the plot below that I reset the baseline and dropped all the promoted harness changes around the 26th experiment due to identifying some non-deterministic environment factors that polluted the experiment results. You can even see that the rejected candidates before the reset had a very narrow but steady variance.

After the non-determinism fix and restart, the harness was able to quickly find good changes and beat the previous best score in less than half of the experiment time.

All 9 promoted harness changes

#	Commit	Mechanism	New tasks solved
–	pre-determinism baseline 18/38
1	927405ea
replayed write(content=...) args capped at 250 tokens (file is on disk anyway)
19 total (+1): sparql-university
2	b5e204fa
Drop empty junk args from tool calls before validation	19 total: nginx-request-logging (net 0 against the previous baseline)
3	cd2e2ad0
If at step >=107 and last 8 actions are all read-only probes, nudge with prompt “convert hypothesis into on-disk edits and submit”	19 total (same solved set; secondary tie-break improved)
–	determinism-hardening reset; re-baseline 16/38
4	a70ec204
Force bare submit at step >=108 if edits exist on disk
17 total (+1): regex-log
5	a0dccbb4
On llm request validation error, strip only extra_forbidden fields and re-validate once
17 total (same solved set; first-try valid tool calls improved from 97.78% to 97.95%)
6	925f790a
Steer oversized-call cutoff from the 2nd occurrence on (1st abort stays byte-identical)	17 total (same solved set; first-try valid tool calls improved from 97.95% to 98.29%)
7	14ef7381
if llm response had a length cutoff with no tool call inside, disable thinking and steer with prompt	22 total (+5): code-from-image, distribution-search, extract-elf, llm-inference-batching-scheduler, overfull-hbox
8	375c3099
LENGTH_CUTOFF_ABORT_THRESHOLD from 2 to 3
23 total (+1): custom-memory-heap-crash
9	ca49ae96
if llm response cutoff with partial tool call, treat as oversized (reuse split-it repair), not malformed
23 total (same solved set; first-try valid tool calls improved from 97.62% to 97.69%)

Note on the tie-breaker promotion: the harness training framework allows users to define secondary reward to break primary loss function ties. In the above training run, the secondary reward is defined as the following, evaluated in order:

• % of tool calls that are valid on the first attempt
• the number of steps used to solve the task

Biggest jump ( 14ef7381, 17 to 22 solves): entire LLM output response budget burned on reasoning with no tool call emitted, which previously killed the trial. The harness change proposed was to disable thinking and inject the repair prompt below. The 5 converted solved tasks were all long-horizon (solved in 67-90 steps).

reasoning_runaway_repair_prompt = (
    "Your previous response reached the output length limit while still "
    "reasoning, before it emitted any tool call. Stop reasoning now and reply "
    "with a single, concise tool call and no other text."
)

Evaluation Results #

Is training the harness actually worthwhile to do? In this section, I’m going to go through some of the evaluation runs by holding the trained harness fixed and swapping out the underlying task LLM model. Here are the evaluation configurations.

Trainedon Terminal Bench 2.0 38 tasksout of 89 tasks.3Git branchwith all the trained harness commits.- Evaluated on Terminal Bench 2.0 all 89 tasks. It’s obvious that there is overlap between the training set and the evaluation set. 23 out of 38 tasks were solved by Qwen 3.6 35B A3B FP4 quantized modelwith the trained harness, and it didn’t include task-specific mechanisms to “cheat”. We will also show that evaluating the same Terminal Bench-trained harness against SWE-bench, there are transfer learning gains too.

Config field	DeepSeek V3.2	MiniMax M2.5	GPT-OSS 120B	GPT-OSS 20B	MiMo V2.5	GPT-5.5
Terminal Bench Settings	Same as Official Terminal Bench 2.0 settings	Same as Official Terminal Bench 2.0 settings	Same as Official Terminal Bench 2.0 settings	Same as Official Terminal Bench 2.0 settings	Same as Official Terminal Bench 2.0 settings	Same settings; 76-task panel, the remaining 13 the API rejects
max_context_length	128K	205K	128K	128K	1M	256K
max_tokens	30K	30K	30K	30K	30K	N/A (using Codex Backend API)
reasoning_effort	high	high	high	high	high	xhigh
temperature	1.0	1.0	1.0	1.0	1.0	N/A (using Codex Backend API)
top_p	1.0	1.0	1.0	1.0	0.95 (recommended by model card)	N/A (using Codex Backend API)

Generalized and transfer learning capability

This shows that for a frozen harness that was trained once using one model (Qwen 3.6 35B A3B FP4 quantized model), it can lift up the capability of a wide variety of models.

It also shows that if we train a harness on one task set (SWE-Bench) 1, that same harness can improve task-solving capability towards a different task set (Terminal Bench)

.

4### New capability

There are a couple of mechanisms in the trained harness that contribute to better task-solving abilities

In this plot below, you can see:

“failed verification” converted to “solved” (+23) because the trained system prompt added

“run output is scratch: editing a file inside a python snippet or a here-doc that only prints does NOT persist your fix”

and

“the on-disk state at submit is the only thing graded”

This targets a baseline failure mode where a model wrote the code in some ephemeral output and submitted an unchanged disk state. Inside the 23 converted “solved” tasks, they used the new file tools (~300 of ~1200 calls). DeepSeek 3.2 is the biggest beneficiary (+7).

• “ran out of time” converted to “solved” (+24) from the trained harness spending half or less of the baseline’s LLM latency. Specifically, GPT-5.5 headless-terminal

task reduced from 1755s to 221s, DeepSeekcompile-compcert

reduced from 1538s to 165s. The trained harness mechanisms that contributed were:- read clipped to 250 lines to cap environment (stdout/stderr) bloat

• shorter completions for several models (output tokens per call: GPT-5.5 −19%, MiniMax2.5 −22%, GPT-OSS 20B −11%)

• if the “Run” command/tool call times out, prompt the LLM with a nudge “change your approach rather than rerunning”, which avoids repeated 300s stalls.

• “no valid tool call repeatedly” converted to “solved” (+6), because in the trained harness, more file tools were introduced (added “read”, “write”, “replace” to the existing “read”, “submit”). More specifically, the “write” and “replace” that persist edits quote-safely via base64, which improved “missing tool call” recovery from 69% to 94%, despite a byte-identical repair prompt. For invalid calls5, the trained harness repair echoes the failed call back as something like “You sent: {calls}”, which improves a ValidationError6recovery from 51% to 83%.6

If we combine the above figure with the below figure, the “hit_timeout” to “solved” conversion would organically cause high step counts, since the same tasks that previously ran out of wall-clock are now being solved, which is driven by efficiency. MiniMax 2.5 has near-zero parse failures in both the trained and baseline harnesses, yet still gains +7 solves beyond 50 steps. Five are timeout/verification flips, suggesting the gains come from “persisting fixes” 7 and write/replace

, not parsing. For GPT-OSS 20B, the fatal step rate fell from 9.4% to 1.9%, allowing trajectories to run longer (attempted steps increased from 5.6k to 9.5k) and enabling some runs to convert late. Tool call failures per step remained essentially unchanged at ~31%. This means that failures survive in the trained harness, and the surviving steps bill 43% more output tokens. The other model family’s long-trajectory wins come from the “timeout” and “failed verification” to “solve” mechanisms. GPT-OSS 120B gains solves while its trajectories shrink. This again shows that each model really has a different failure model or task-solving strategy baked into the weights.

8If we decompose “Task Solve” mathematically as (\text{Submitted} \wedge \text{Passed Verification}), the trained harness could only achieve its goal through those two lenses, and that’s what it did. “Submit better” comes from persisting fixes 7, plus the trained harness’s longer, more careful trajectories filtering weak submissions out of the pool. For DeepSeek 3.2, pass rate also rose as failing submissions stopped reaching submit at all. Only MiniMax 2.5 netted solves from submitting more, at the cost of a lower pass rate.

Token efficiency

For the genuine token efficiency wins, since the trained harness introduced more file tools, it shifted the tool call mix toward cheap calls. In GPT-5.5’s trained harness evaluation, read averages 433 output tokens vs 1,013 for the untrained baseline harness evaluation, and 374 of ~2,200 calls moved to read/replace (tokens per call 1,422 decreased to 1,151).

There are essentially two camps here:

With GPT-OSS 20B as the representative, its context blowup is rescued by the trained harness survival mechanism, so the task can still be solved at longer steps, but they pay for it using more input tokens

 baseline   steps  ▓▓▓▓▓  (breaker trips ~step 5, low recovery — dies)
            input  ██                          context never gets big

 trained    steps  ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓  (recovers 94%, runs on)
            input  ████████████████████████████ each step re-reads more

 attempted steps 5618 grows to 9472   total prompt +110%
 input/solve +73% output/solve +18%   total output +43%

With GPT-OSS 120B as the representative, its trajectories shrank while solving more, which is shown by the cost savings on the output tokens side.

 baseline   steps  ▓▓▓▓▓▓▓▓▓▓▓▓▓▓  (wanders longer before solving)
            input  ██████████████████

 trained    steps  ▓▓▓▓▓▓▓▓  (tighter path to submit)
            input  ███████                     

 attempted steps 5834 decreases to 4290   input/solve −55%   output/solve −35%

Tokens-per-solve falls because trained harness’s first-order effect was to increase solve, which increased the denominator, which can be seen for most cases in the below figure.

We see divergence (baked into the weights) for token efficiency as well. In terms of the cost of solving a task, the two GPT-OSS models from the same family sit at opposite extremes. 120B reduced calls per solve by 36% through fewer model calls, while output tokens per call remained nearly flat (+1%). By contrast, 20B increased calls per solve by 32%: longer survival task solves outweighed an 11% reduction in output tokens per call.

Note that “model call retry %” didn’t change much: 20B 31% to 27%, 120B 9% to 12%, everyone else shows less than 1% gap between the baseline and trained harnesses. So (\frac{\text{model calls}}{\text{solve count}}) are driven mostly by (\frac{\text{trajectory length}}{\text{solve count}}), not repair traffic.

MiMo 2.5 behaves the opposite from GPT-5.5. MiMo 2.5 is the only model whose calls got longer (+19% per call) while getting fewer (−8% calls/solve). This means the model does more deliberation per step at fewer steps, net +9% pricier per solve. GPT-5.5 did the reverse (−19% per call, +2% calls).

A rising useful-share can mean opposite things per model:

• MiniMax 2.5’s biggest jump, where token shares tripled (from 13% to 41%) but only on +2 solves, is driven by tokens spent on solved tasks exploding from 12M to 56M, where 36.5M of that is just two tasks (winning-avg-corewars 21M, fix-ocaml-gc 15.5M, the 242-step timeout to solved flip). Its failed-spend barely moved. Maybe this harness is not suitable for MiniMax 2.5 on those tasks?
• For GPT-OSS 120B, the “failed-spend” fell from 221M to 98M (−56%), it’s mostly because failure itself got cheaper (shorter trajectories before dying or timing out). This is because of the same trajectory-shrinking mechanism in the trained harness.9 - For GPT-OSS 20B, it’s an interesting one. Keeping 20B alive poured an extra 192M tokens into tasks that still failed versus extra 16M onto solved tasks. When combined with the above analysis for this model, there’s definitely some capability issue on the model side that’s making it inefficient to rescue. Or if there was a better way to detect this early on, we can let the harness “give up early” when running with less capable models.

Note on interpreting the numbers: “Figure: Efficiency Levers” and “Figure: Useful Share” denominators mix failed-task spend into per-solve/share metrics, so every number inherits the composition effects we talked about earlier (which tasks die, time out, or reach submit). “Figure: Efficiency Levers”’s identity (per-solve = per-call * calls/solve) is exact, but neither panel isolates “the same task got cheaper”.

Determinism #

In order to achieve the evaluation results above with the trained harness, we need to understand the core prerequisite of training a harness.

Remember that I initially ran a lot of experiments without considering this factor and used multiple trials to “gain statistical power”? Well, that’s not scalable for this project where I don’t have enough compute power to speed up the experiment cycles. So I changed the way I run experiments by guaranteeing determinism. What do I mean by determinism?

Given the same setup, configs and harness etc…, I expect the exact same logits returned from the task LLM, same output tokens, same reasoning tokens and executing the exact same tool calls against the environment, returning the exact same results from the environment and reaching the exact same conclusions. To motivate this path, let me share my initial attempt without determinism.

Motivation in the self-improvement loop context

Before I worked on guaranteeing determinism, I ran over 1000 experiments. During those runs, I saw a lot of experiment noise that caused tasks to flip from “solved” to “unsolved” even though the experiment treatment (harness) didn’t change the critical path for that particular task. I tracked every mechanism firing with a 1-1 mapped metric to prove this. As a result, I spent a lot of time in designing a “good” candidate promotion criteria.

This example below is the LLM responses being non-deterministic:

• task: swebench django-14534
• model: gpt-5.5
• input: identical 751-token prompt, identical tool-schema, fingerprint (1000b896960b), request key 0b4c96fa…3cad, sent 39 times

output:

run A (exp-20260614-221243, step 4): list_dir    {"path": "/testbed"}
run B (exp-20260615-022510, step 4): search_text {"query": "class BoundWidget", "root": "/testbed"}

At the experiment trial-level, trials are not deterministic, so experiment outcome depended entirely on coin flips:

Ok, so we saw the initial experiment setup was non-deterministic, but why do we need determinism here?

Imagine this: if a harness change (proposed by the harness training loop) had a very small effect towards a very small number of tasks and behavior change, from a statistics perspective, many repeated trials are needed to determine whether the “solve” was noise or a true effect of the treatment. But even then, that only provided confidence intervals, not a guarantee of a “good” harness change.

So is non-determinism bad? Not necessarily. Another way to think about this problem is: determinism is useful for credit assignment but once our policy (harness) is frozen, non-determinism can help with real-world performance by avoiding getting stuck in a local optimum.

I came to this realization through a random thought I had one day: Why do Reinforcement Learning (RL) environments like https://github.com/openai/gym iterate through each experiment so fast and allow for stable learning signals from each iteration? I realized the key missing component was determinism.

If we can ensure determinism, then every experiment outcome can be completely traced back to harness change solely. In the RL framing, this is like making credit assignment cleaner.

N=1

You might have seen that standard agentic evaluations need pass@k over many seeds to fight noise. That’s because the evaluations need to reflect real-world use-cases, and non-determinism is present in nearly all of those use-cases.

But in our harness training case, our goal is to use determinism to find a good harness without the noise. Once we commit to a finalized harness (frozen), we can use it with non-deterministic environments and LLMs.

Now that we’ve motivated the determinism efforts, let’s talk about the two factors that we need to take into account: LLM inference determinism and environment determinism. I will go over each one.

Deterministic LLM Inference

When we talk about LLM inference determinism, we’re usually talking about the deterministic logits output for the same model serving settings (e.g. LLM checkpoint, inference server version etc…). Note that “temperature” and other sampling args like “top_k” affect how we sample the tokens from a given logits/softmax distribution, but they don’t affect how those logits are generated by the model in the first place.

The root cause of non-determinism

There is a great blog post Defeating Nondeterminism in LLM Inference 11 on this exact topic which I’ve learned a lot from, so I won’t go into details.

An over-simplified explanation to motivate the next section of the blog

Recall that LLM inference (forward pass) is a bunch of matrix multiplication to produce a set of logits, which then gets converted into a probability distribution over the vocabulary space, then sampled to get a specific token output. Well, the matrix multiplication part doesn’t always produce the same set of logits, given the exact same input tokens. The reason is because floating point number addition is non-associative (dot products involve additions). This problem can be amplified by factors like batch shapes, which in turn affects how LLM kernels schedule these matrix operations. Different order -> different rounding due to precision -> different logits -> different probability distribution -> different final token (even with temperature = 0).

[(a + b) + c \ne a + (b + c)]For example, in float32

, let

Then

[(a+b)+c = (0+1) = 1 \ a+(b+c) = 10^8 + (-10^8+1)]Since 1 is too small to change (-10^8) in float32

,

Therefore:

[(a+b)+c = 1 \ a+(b+c) = 0]The reason why non-determinism is the default for many inference engines is because it’s often more efficient to avoid the special handling required to achieve determinism. Oftentimes it requires special kernel implementations for determinism guarantees.

LLM inference providers don’t guarantee determinism

Initially, I used different inference providers to train my harness for the task LLM. But after trying out many providers, I realized determinism was only best-effort, never guaranteed. There are a couple of reasons that I found:

Multi-tenant inference. We’ve talked about how different batch shapes would cause the matrices to be multiplied in different order, which causes non-determinism. This is amplified by the fact that LLM providers need to batch many requests from many users together to save cost (fully utilize GPU memory).- . Some providers support it, but some don’t. But even if they do, they never guarantee it (seed

parameter for inference is not guaranteedfor example).Determinism is best-effort and is not guaranteed across model versions

Rate limits / infra issues can cause the task-solving trajectory to diverge or even crash the run due to timeouts

What I used instead

For LLM inference options, in a naive case, if you just call the model’s forward pass via PyTorch, you would get deterministic results, assuming you have set all the seed

values correctly. But if you start to use these inference engines like vLLM or SGLang, they have a lot more optimizations done to improve the LLM inference throughput and memory footprint that break determinism.

I eventually decided to go with SGLang as the inference engine, as it’s the one that explicitly said deterministic inference is supported by a couple of attention backends.

I’ll leave out the experiment notes for tuning the inference engine since it’s too distant from this “harness training” project. The main takeaways are:

• Setting --enable-deterministic-inference

and setting a fixedseed

is the first thing that I did. - mixed-length requests by themselves don’t break determinism

• client concurrency itself doesn’t break determinism, it’s more about how the requests are batched
• even with radix caching disabled, the max number of active request slots (Mamba-slot cap for Qwen3.6-35B-A3B-NVFP4 model’s State Space Model) still has a ceiling before non-determinism slips in. In this case, active request slots had to be <= 10 to guarantee determinism for my setup (12bootstrap script)

LLM Determinism costs throughput degradation

In the below plot, you can see that we can guarantee determinism up to 10 concurrent requests on this diagnostic box, but after that, we observe drift. This also means that we lose out on higher throughput.

Some throughput levers I tested that failed the determinism gate:

torch.compile

incompatible with batch-invariant ops- bf16 mamba dtype improves throughput by +12.7% but breaks determinism

• radix cache fails mixed-prefix gate
• NGRAM spec decoding wildly nondeterministic

Deterministic environment

Another factor that we’ve taken for granted is that the environment the LLM is interacting with (input to the LLM) is deterministic, which doesn’t always hold. We can see this in the example below.

Setup: temp=1.0, top_p=1.0, seed=12345, sampling from the exact same deterministic logits from the deterministic LLM inference engine

Both runs made the same tool call at step 15. Everything before step 15 was completely identical on both the LLM and environment side.

e = Exists(Product.objects.all())
q = Q()
result = e & q
print("Exists & Q:", result)

But the environment returned different stdout back to the LLM, due to memory addresses being random.

Run A (env stdout): Exists & Q: (AND: <django.db.models.expressions.Exists object at 0x7ffffd51dd30>, (AND: ))
Run B (env stdout): Exists & Q: (AND: <django.db.models.expressions.Exists object at 0x7ffffd544d30>, (AND: ))

After that, the LLM output diverged starting from Step 16:

Run A (LLM output tokens): <emitted a valid structured tool call>
Run B (LLM output tokens): <emitted the intended tool call as text inside reasoning_content, so the harness rejected it>

More examples of environment randomness

Task that forked	Random element	Fix
django-14017	Python object repr embeds memory address at 0x7ffffd...
at <ADDR> scrub (src/determinism.py:97 )
sphinx-10673	tempfile suffix: /tmp/sphinx-err-zukakzni.log vs -m0nj2_k1.log – forked every trajectory after step 85
<RAND> scrub
pytest-5840	git log --after=2019-08-26 returned different commits in UTC vs America/Los_Angeles
TZ=UTC pinned in solve env
sympy-23950	test runner prints per-process random seed: 73158167
<SEED> scrub
pypi-server, build-cython-ext	pip’s random build paths: pip-ephem-wheel-cache-xg6l8n1u , wheel size=... sha256=...
3 scrub rules
kv-store-grpc	/proc/net/tcp socket inode + kernel socket cookie vary per boot
<INODE> /<SK> scrub
distribution-search	/dev/urandom differs per run
vendored libfaketimeMT.so.1 LD_PRELOAD, FAKERANDOM_SEED=0x12345678DEADBEEF
any task running apt-get update
live round-robin mirrors at differing sync states	apt-cacher-ng
(all tasks)	hash seeds, git dates, hostname, mtimes, GDB PIDs	PYTHONHASHSEED=0 , PERL_HASH_SEED=0 , git dates @0 +0000 , SOURCE_DATE_EPOCH=0 , fixed hostname, mtimes reset to 1980 via git hook, GDB inferior-events off

Determinism needs to be an end-to-end property of the loop, not just an LLM policy property.

Doing this in practice is not easy. We can set some seed or turn off some verbose logging, but oftentimes since we don’t own the environment, it’s a game of whack-a-mole. As you can see, I’ve constrained many non-determinism factors at the source (environment), but there are still many that leak out that cannot be easily controlled or may affect task solving setup.

In the training framework (in the next section), before comparing a candidate against a baseline, the framework “certifies” each task by re-executing the baseline’s recorded action chain in a fresh environment and checks that every step reproduces the same output. Any tasks that fork are excluded because we want to only capture candidate effects caused by the candidate’s change, not environment noise.

Comparing pre/post-determinism

Holistically, we can see that with the same candidate promotion criteria, the candidate promotion rate increased

General Framework #

As part of this project, I was able to create a general framework that allows any harness to self-improve to a state that is model-agnostic and task-environment-agnostic. I call this “Harness Training”. It’s a discrete program optimization behind a PyTorch-like API. An estimator edits the harness policy, the framework evaluates the candidate against the baseline, and a criterion decides whether the candidate becomes the next baseline.

criterion = StrictPareto()
optimizer = GreedyMonotonic()
trainer = Trainer(
    config_path="config/train_harness.yaml",
    estimator=AgenticEstimator(
        backend=CodexAgentBackend(trace_dir=Path("experiments/codex-traces"))
    ),
    criterion=criterion,
    optimizer=optimizer,
)

for loss in trainer.epochs(30):
    loss.backward()
    optimizer.step()

That’s it. The inner optimizer.step()

persists the candidate harness change if it was proven to be “good”. The outer for-loop is like program search or hill climbing.

The inputs for a training run are:

• task environment (e.g. SWEBench, terminal bench tasks)
• deterministic LLM inference endpoint with a frozen LLM checkpoint
• loss function (criterion on what’s good and bad, e.g. more task solves is good, more LLM tokens consumed is bad)
• optimizer (how to read loss and decide when to persist the harness changes)

Output: full experiment traces of how the harness performed over time to solve more and more tasks, including the exact raw input and parsed tool calls and raw reasoning traces of the LLM calls and telemetry of the experiment. Example

Core Capabilities

New environment to train: implement TaskEnv

_ENV_FINGERPRINT = ...

@dataclass(frozen=True, slots=True)
class Task:
    instruction: str
    tests_dir: Path
    agent_timeout_sec: float | None = 600.0
    replay_id: str | None = _ENV_FINGERPRINT

async def load_tasks(
    *, task_ids: Sequence[str], environment: EnvironmentConfig, verify_cache: bool
) -> TaskSet[Task]:
    root = Path("benchmarks/benchmark-name")
    tasks = {
        task_id: Task(
            instruction=(root / task_id / "instruction.md").read_text(),
            tests_dir=root / task_id / "tests",
        )
        for task_id in task_ids
    }
    return TaskSet(
        kind=environment.kind,
        tasks=tasks,
        env_factory=lambda task, rollout_dir: Env(
            task=task, artifacts_dir=rollout_dir, verify_timeout_sec=300.0
        ),
    )

class Env(DockerTaskEnv[Task]):
    _task_workdir = "/work"

    def _build_solve_env(self, task: Task) -> DockerShellSession:
        return DockerShellSession(image="python:3.13-slim")

    async def verify(self) -> VerifyOutcome:
        await self._start()
        await self._solve_env.upload_dir(
            source_dir=self._task.tests_dir, target_dir="/tests"
        )
        result = await self._solve_env.run(
            command="python -m pytest -q /tests",
            cwd=self._task_workdir,
            timeout=self.verify_timeout_sec,
        )
        passed = result.exit_code == 0
        return VerifyOutcome(
            verdict=VerifyVerdict(completed=True, passed=passed, error=None),
            output=RawEnvOutput(
                exit_code=result.exit_code, stdout=result.stdout, stderr=result.stderr
            ),
            reward=1.0 if passed else 0.0,
            info={},
        )

New workflow to propose harness change: implement Estimator

class AgenticEstimator(Estimator):
    """The default estimator: both decisions are delegated to an LLM agent.

    Three open-ended ingredients:
      1. SEARCH SPACE — anything the agent can edit in its worktree
      2. POLICY       — the LLM itself, steered by a prompt
      3. MEMORY       — a free-form memo the agent rewrites each epoch
    """

    def __init__(self, *, backend: AgentBackend) -> None:
        self.backend = backend

    def propose(
        self,
        *,
        repo_root: Path,
        tracker: RunStore,
        target: TrainingTargetConfig,
    ) -> None:
        """One agent turn leaves a candidate patch in the worktree.

        > measure baseline            # harness: 21/40 solved at commit abc123
        estimator.propose(...)        # THIS FUNCTION
        > capture + measure candidate # harness: diffs the worktree -> commit def456,
        """
        staged = self._stage_reads(tracker, worktree=repo_root, experiment_id=None)

        self.backend.run_turn(
            prompt=f"Read program.md; you are the proposer. Diagnose the prior run "
            f"from {staged.root}, then make one bounded improvement to "
            f"{', '.join(target.patch_paths)}.",
            repo_root=repo_root,
        )

    def diagnose(
        self,
        result: ExperimentResult,
        *,
        repo_root: Path,
        tracker: RunStore,
        target: TrainingTargetConfig,
    ) -> None:
        """The agent authors the memo; the harness checks and publishes it.

        > candidate result 23 > 21 -> "promoted", run saved
        estimator.diagnose(...)       # THIS FUNCTION
        > next propose() reads memo   # the feedback loop closes
        """
        staged = self._stage_reads(
            tracker, worktree=repo_root, experiment_id=result.experiment_id
        )
        draft = staged.root / "learning.draft.md"
        self.backend.run_turn(
            prompt=f"Rewrite the learning memo from run {result.experiment_id}: "
            f"bottlenecks, what was tried, what to try next. Write it to {draft}.",
            repo_root=repo_root,
        )
        tracker.publish_learning(draft.read_text())

New Loss function: Implement Criterion

class TotalTaskSolves(Criterion):
    name = "total_task_solves"

    def _compare(
        self, comparison: Comparison
    ) -> tuple[str, tuple[SecondaryRewardComparison, ...]]:
        delta = len(comparison.candidate_solved) - len(comparison.baseline_solved)
        if delta > 0:
            return "higher_total_task_solves", ()
        if delta < 0:
            return "lower_total_task_solves", ()
        return "no_total_task_solve_improvement", ()

    def _loss_value(self, report: Comparison) -> float:
        return float(len(report.baseline_solved) - len(report.candidate_solved))

New task LLM model/provider: Implement the completion-backend contract

class CustomCompletionBackend(CompletionBackend):
    def __init__(self, *, config: LlmProviderConfig) -> None:
        self.config = config
        self._client = AsyncCustomClient(api_key=os.environ[config.api_key_env])

    @classmethod
    def complete_duration_bound_sec(cls, max_tokens: int) -> float:
        return stream_stall_timeout_seconds(max_tokens)

    async def _complete(self, request: CompletionRequest) -> Completion:
        raw = await self._client.complete(
            model=self.config.model_name,
            messages=request.messages,
            tools=request.tools,
        )
        return Completion(
            content=raw.content,
            tool_calls=tuple(
                ToolCall(name=call.name, arguments=call.arguments or "{}")
                for call in raw.tool_calls or ()  # SDKs return None when no tools fire
            ),
            finish_reason=raw.finish_reason,
            usage=Usage(
                prompt_tokens=raw.usage.input_tokens,
                completion_tokens=raw.usage.output_tokens,
                reasoning_tokens=raw.usage.reasoning_tokens,
            ),
            reasoning_content=raw.reasoning_content,
            response={"id": raw.id, "model": raw.model},
        )

    async def close(self) -> None:
        await self._client.close()

Plugins

Mostly for speeding up training, see note below for prerequisites.

Boundary	Plugin	What’s cached
policy <> world	llm_cache
exact rendered request → completion
agent <> environment	step_cache
scrubbed action → step result
solution <> grade	verify_cache
(instance, diff) → verdict
container <> internet	netcache
HTTPS/package fetches

Note:

llm_cache

only works if the thing it’s caching is a pure function of its cache key. (e.g. LLM cache key = exact request payload), but purity of the function (fixed seed + sampling params = same completion) is the model/provider’s job, not the cache’s. If the LLM inference is not deterministic, whenever it hits its first cache miss, the whole suffix of the rollout diverges. In other words, cache can preserve determinism, but not create it.step_cache

for environment steps requires the environment to be deterministic so that training can speed up CPU intensive operationsverify_cache

must be a pure function of the diff, network responses are frozen by replay scope.

Future Research Directions & Discussions #

Determinism and sparse rewards: What does determinism remove from sparse-reward optimization, and what problems remain?** Overfitting to one seed**: multi-seed determinism (K fixed seeds, promote only changes that help across a majority), which recovers variance estimates without giving up reproducibility. This also connects to regularization.Order-dependence: do all the candidate commits still contribute in the final harness? or did later changes make earlier changes obsolete? Would a different candidate promotion order converge to a different (or better) point? I think there are parallels to model initialization and regularization in deep learning training.Harness vs model ceiling: if a failure disappears when the same trained harness drives GPT-5.5, it was a model ceiling. But if it persists across models, it’s harness territory. How can we detect this? Is there a value in model-agnostic harnesses?Harness baseline/priors: Do mechanisms transfer as a “harness prior”? The SWE-bench and terminal-bench runs independently discovered near-identical mechanisms (reasoning-runaway repair, work-exists-but-never-graded fixes). That convergence suggests a task-environment-agnostic core. Can we merge the harness changes from separate runs? Can joint multi-dataset training beat single-dataset training plus transfer? Is there a small universal set of harness invariants that should just be baked into baselines (model initialization)?Harness training as data generation for weight training: Right now, we are always using a frozen LLM model to solve tasks, while we change the harness during training. Do the trajectories generated from the trained harness contribute better quality SFT/RL data than baseline trajectories? Can harness improvements be distilled into the model (e.g. reasoning-runaway harness repair becomes unnecessary if the trajectories are part of the RL dataset for the LLM post-training). What about compute? Should it be better spent on harness search or fine-tuning the model weights directly?Harness training scaling law: solve rate vs number of experiments vs task count

Appendix #

Interesting observations during harness training

Self-diagnosed reward hacking: in thelearning.md

(cross-epoch shared memory), it noted the promoted candidate as “HOLLOW, the borderline taskdjango-11087

’s change was incorrect but dodged the penalty by never submitting”, so the eventual task result was “timed out” instead of “verifier rejected”Harness learned to catch the model lying to itself: pytest exit codes piped through| tail

always return 0, so the task-solving LLM interpreted the test suite as passing and submitted broken patches. The next turn harness change rewrote the pipeline withpipefail

. Convertedsphinx-10323

at 96 steps. The baseline trajectory had submitted a wrong patch at step 108 believing tests passedInvent new tools: Added a dedicatedsearch

tool because the model’s ad hoc search commands, submitted through the generalrun

tool, were inefficient, unreliable, and sometimes timed out

django__django-10973, django__django-11087, django__django-11490, django__django-14017, django__django-14034, django__django-14155, django__django-14170, django__django-14315, django__django-14376, django__django-14534, django__django-14792, pylint-dev__pylint-4970, pylint-dev__pylint-6386, pylint-dev__pylint-7080, pytest-dev__pytest-10051, pytest-dev__pytest-10081, pytest-dev__pytest-5840, pytest-dev__pytest-7205, sphinx-doc__sphinx-10323, sphinx-doc__sphinx-10435, sphinx-doc__sphinx-10673, sphinx-doc__sphinx-7748, sphinx-doc__sphinx-8056, sphinx-doc__sphinx-9673, sympy__sympy-12419, sympy__sympy-13091, sympy__sympy-13798, sympy__sympy-13974, sympy__sympy-23950, sympy__sympy-24443, django__django-10999, django__django-11477, django__django-11728, pylint-dev__pylint-6528, pytest-dev__pytest-7490, sphinx-doc__sphinx-7462, sphinx-doc__sphinx-9281, sympy__sympy-17318, sympy__sympy-18763

↩↩2 - Qwen 3.6 35B A3B FP4 quantized modeltemp 1.0, top-p 0.95, 32k context length, max token count 8k, reasoning token count 6k↩ - adaptive-rejection-sampler, circuit-fibsqrt, write-compressor, chess-best-move, polyglot-c-py, llm-inference-batching-scheduler, sqlite-db-truncate, make-mips-interpreter, torch-tensor-parallelism, custom-memory-heap-crash, bn-fit-modify, cobol-modernization, sam-cell-seg, extract-elf, sparql-university, path-tracing-reverse, constraints-scheduling, overfull-hbox, distribution-search, build-cython-ext, prove-plus-comm, code-from-image, hf-model-inference, large-scale-text-editing, regex-log, nginx-request-logging, sqlite-with-gcov, count-dataset-tokens, pytorch-model-recovery, pypi-server, modernize-scientific-stack, portfolio-optimization, cancel-async-tasks, build-pmars, kv-store-grpc, multi-source-data-merger, fix-git, log-summary-date-ranges

↩↩2 - We evaluate on 76 of Terminal-Bench 2.0’s 89 tasks, excluding the 13 tasks that some model APIs reject, which are security-, cryptanalysis-, or exploitation-themed (8 tagged security outright, plus 5 adjacent: two FEAL cipher-breaking tasks, model-weight extraction, leaked-credential recovery, and encrypted-database recovery).

↩↩2 - When the LLM model didn’t return any tool calls at all, and only returned prose or reasoning.

↩ - When the LLM model returned tool calls, but they were malformed (e.g. missing/extra/wrong-typed fields).

↩↩2 - The trained system prompt adds “run output is scratch: editing a file inside a python snippet or a here-doc that only prints does NOT persist your fix” and “the on-disk state at submit is the only thing graded”

↩↩2 - the “write” and “replace” tools that persist edits quote-safely via base64

↩ - read clipped to 250 lines caps observation bloat

↩ - The figure shows four tasks from an evaluation of one frozen GPT-5.5 harness across 59 tasks, with 3–5 repeated trials per task. These tasks are shown because each produced both solved and failed outcomes. This single baseline demonstrates within-setup variability rather than estimating a general failure rate.

↩ - He, Horace and Thinking Machines Lab, “Defeating Nondeterminism in LLM Inference” Thinking Machines Lab: Connectionism, Sep 2025

↩ - 5090 GPU,

Qwen 3.6 35B A3B FP4 quantized model.--enable-deterministic-inference --random-seed 12345 --attention-backend triton --disable-radix-cache --kv-cache-dtype fp8_e4m3 --max-running-requests 10 --chunked-prefill-size 8192 --mem-fraction-static 0.90

↩ - This diagnostic used Qwen3.6-35B-A3B on one RTX PRO 6000 GPU with SGLang. The measured ceiling is specific to this model and hardware setup; it is not a general limit for SGLang or other models.

↩ - The pre-determinism period covers April 11–May 25 (1,223 decisions); the post-determinism period covers June 25–July 1 (131 decisions). Because the periods differ in duration and experiment volume, this is a descriptive comparison rather than a controlled causal estimate.

↩

source & further reading

henrypan.com — original article What 1k Harness Experiments Taught Me About Self-Improving Agents