This is the follow-up to Coding Agents Evolved. Our Repositories Didn’t..
My first Repository Harness benchmark produced an uncomfortable result.
The Harness completed the task faster than a monolithic AGENTS.md
, but it used more fresh input tokens.
That could have meant the idea was wrong.
It could also have meant that my first implementation of progressive disclosure was not strict enough.
So I changed the routing model, rebuilt the benchmark baseline, and ran the experiment again.
This time, I ran 30 measured executions.
The result was not a dramatic reduction in token usage.
It was something more useful:
The Repository Harness made the agent’s behavior substantially more predictable.
Most repositories expose their knowledge through a mix of:
AGENTS.md
.A monolithic instruction file works, but every task receives the complete repository handbook, even when most of it is irrelevant.
The Repository Harness replaces that model with a small entry point and routed context:
AGENTS.md
↓
.harness/
├── harness.yaml
├── project-specific guidance
├── engineering guidance
└── asset guidance
The root file explains how to use the Harness.
The manifest classifies the task and selects the repository guidance required for that task.
The original hypothesis was:
Selective context routing should preserve implementation quality while reducing unnecessary context.
The first benchmark did not prove that.
The first Harness used broad rules such as:
load_when:
- implementation
- feature
- validation
Those labels were too general.
Almost every coding task is an implementation. Many tasks are features. Every completed task needs validation.
The agent therefore had a reasonable incentive to load most of the documentation.
I had modularized the repository knowledge.
I had not created strict routing.
The second version added:
Before explaining how that routing behaved, it helps to understand the repository used in the experiment.
The benchmark used a real game project built with Godot and C#.
The task was a gameplay-input change involving camera controls.
The agent needed to:
This repository was useful for the experiment because it contained different kinds of knowledge:
Godot and gameplay conventions
general engineering rules
asset rules
validation commands
A camera-input task should need the Godot-specific guidance.
It should not need asset guidance.
It should only need broader engineering guidance if the implementation crosses an architectural boundary.
That made the task a practical test of selective routing.
I created two isolated versions of the same repository.
AGENTS.md
All repository guidance lived in one concise root document.
AGENTS.md
.harness/
The root file acted as an entry point.
The Harness contained:
.harness/
├── harness.yaml
├── GODOT.md
├── ENGINEERING.md
└── ASSETS.md
For this camera-input task, the expected route was:
AGENTS.md
+ .harness/harness.yaml
+ .harness/GODOT.md
Across all 15 measured Harness runs, the agent inspected the manifest and the Godot-specific guidance without the unrelated engineering or asset documents.
That confirmed that the routing mechanism itself was working.
The remaining question was whether it improved the complete execution.
Both variants used the same:
I ran:
3 warm-ups per variant
15 measured runs per variant
30 measured runs total
The order alternated between variants.
The prompt was identical in every run.
Each repository started from a clean, frozen commit.
Every measured execution satisfied the benchmark acceptance criteria.
AGENTS: 15/15 accepted
HARNESS: 15/15 accepted
The Harness did not reduce reliability.
| Metric | AGENTS | Harness | Harness difference |
|---|---|---|---|
| Duration | 188.36 s | 185.61 s | -1.46% |
| Total input tokens | 803,295 | 829,006 | +3.20% |
| Uncached input tokens | 63,147 | 61,859 | -2.04% |
| Output tokens | 5,947 | 5,630 | -5.32% |
| Reasoning tokens | 987 | 952 | -3.63% |
| Commands | 14.40 | 14.53 | +0.93% |
| Failed commands | 6.07 | 7.00 | +15.38% |
| Changed files | 6.20 | 6.00 | -3.23% |
The Harness was slightly faster.
It used slightly fewer uncached input tokens and produced less output.
It also used more total input tokens and executed more failed commands.
None of those mean differences was large enough to justify a strong claim that the Harness was reliably faster or cheaper.
The honest conclusion is:
Average efficiency was approximately tied.
That was not the most important result.
The concise monolithic AGENTS.md
was roughly 14 KB.
The Harness route loaded about:
small AGENTS.md ~0.8 KB
harness.yaml ~3.3 KB
GODOT.md ~7.3 KB
--------------------------------
routed context ~11.5 KB
The Harness successfully avoided unrelated documents, but the direct reduction was only a few kilobytes.
That is small compared with a complete agent execution containing source files, search results, command output, tests, diffs, and repeated tool context.
The experiment averaged roughly 62,000 uncached input tokens per run.
Saving a few kilobytes of instructions was unlikely to transform the total.
This changed the main question.
The Harness may not primarily optimize the size of the initial prompt.
It may optimize how consistently the agent navigates the rest of the task.
| Metric | AGENTS standard deviation | Harness standard deviation | Reduction |
|---|---|---|---|
| Duration | 46.67 s | 25.10 s | 46.2% |
| Total input tokens | 239,991 | 108,745 | 54.7% |
| Uncached input tokens | 19,211 | 9,855 | 48.7% |
| Output tokens | 1,430 | 772 | 46.0% |
| Commands | 3.27 | 1.13 | 65.6% |
In this sample, the Harness produced:
I did not begin this project with variance as the primary hypothesis.
But predictability may matter more than a small improvement in the mean.
Lower variance improves:
The Harness did not make every run dramatically cheaper.
It made the execution path more repeatable.
A monolithic instruction file gives the agent every repository rule at once.
The agent must decide which ones matter while already performing the task.
A routed Harness makes the selection explicit:
task classification
↓
repository route
↓
required context
↓
optional escalation
↓
selected validation
This does more than organize Markdown files.
It constrains the agent’s decision space.
That provides a plausible explanation for the lower variance, although this benchmark cannot prove causation.
The strongest conclusion supported by the data is:
Repository-defined routing preserved implementation success and made coding-agent behavior more consistent.
Not in the simple way I originally imagined.
The experiment does not prove that the Harness always uses fewer tokens or is reliably faster.
It does show that:
So the result is not:
The Harness is cheaper.
It is:
The Harness is more predictable.
That may be the more valuable engineering property.
The current benchmark tested one task inside one Godot repository.
The next step should test whether the routing model generalizes to a repository with clearly different technical domains.
A backend-and-frontend project is a stronger test because it contains knowledge that should not be loaded for every task:
backend
database
API contracts
frontend
accessibility
design system
integration tests
deployment
The next benchmark should include:
A useful Harness should remain narrow for isolated work and expand only when the task crosses a boundary.
A practical setup would be a TypeScript monorepo:
apps/
├── api/
└── web/
packages/
├── contracts/
└── testing/
Validation could be fully automated:
lint
type-check
unit tests
integration tests
production build
headless end-to-end tests
The next benchmark should measure not only tokens and duration, but also:
The central question for Part 3 is:
Can repository-defined routing stay narrow for backend-only and frontend-only work, then expand correctly for a cross-stack task?
Persistent memory is promising.
A Harness could eventually learn architectural decisions, recurring failures, conventions, validation paths, and known issues.
But adding memory now would mix two different hypotheses:
Memory also makes later runs depend on earlier ones, which complicates a controlled comparison.
The cleaner sequence is:
Part 2
Selective routing and predictability
Part 3
Full-stack generalization
Part 4
Lifecycle hooks and persistent memory
If the Harness generalizes in Part 3, memory becomes the next logical layer.
The original promise was token reduction.
The second benchmark did not produce a large or statistically conclusive reduction in average token use.
Instead, it revealed a different benefit:
Repository-defined context routing can make coding-agent behavior more predictable without reducing implementation success.
Reliable agent workflows need more than correct final code.
They need bounded behavior:
That is what the Repository Harness is becoming.
The first article proposed the idea.
The first benchmark exposed a routing flaw.
The second benchmark showed that stricter routing worked and made the agent more consistent.
Part 3 will test whether that result generalizes beyond Godot.
The question is no longer only:
Can a Harness reduce tokens?
It is:
Can a repository define a predictable, auditable, and progressively disclosed operating environment for any coding agent?
The project is available at:
Repository Harness Specification
https://github.com/Repository-Harness-Specification