{"slug": "repository-harness-part-2-30-runs-later-it-wasnt-cheaper-it-was-more-predictable", "title": "Repository Harness, Part 2: 30 Runs Later, It Wasn’t Cheaper. It Was More Predictable.", "summary": "A developer's Repository Harness benchmark, run 30 times, showed that selective context routing did not reduce token usage but made agent behavior substantially more predictable. The harness routed only relevant documentation to the agent, avoiding unnecessary context loading.", "body_md": "This is the follow-up to [ Coding Agents Evolved. Our Repositories Didn’t.](https://dev.to/lepsistemas/coding-agents-evolved-our-repositories-didnt-f4).\n\nMy first Repository Harness benchmark produced an uncomfortable result.\n\nThe Harness completed the task faster than a monolithic `AGENTS.md`\n\n, but it used more fresh input tokens.\n\nThat could have meant the idea was wrong.\n\nIt could also have meant that my first implementation of progressive disclosure was not strict enough.\n\nSo I changed the routing model, rebuilt the benchmark baseline, and ran the experiment again.\n\nThis time, I ran 30 measured executions.\n\nThe result was not a dramatic reduction in token usage.\n\nIt was something more useful:\n\nThe Repository Harness made the agent’s behavior substantially more predictable.\n\nMost repositories expose their knowledge through a mix of:\n\n`AGENTS.md`\n\n.A monolithic instruction file works, but every task receives the complete repository handbook, even when most of it is irrelevant.\n\nThe Repository Harness replaces that model with a small entry point and routed context:\n\n```\nAGENTS.md\n    ↓\n.harness/\n├── harness.yaml\n├── project-specific guidance\n├── engineering guidance\n└── asset guidance\n```\n\nThe root file explains how to use the Harness.\n\nThe manifest classifies the task and selects the repository guidance required for that task.\n\nThe original hypothesis was:\n\nSelective context routing should preserve implementation quality while reducing unnecessary context.\n\nThe first benchmark did not prove that.\n\nThe first Harness used broad loading rules such as:\n\n```\nload_when:\n  - implementation\n  - feature\n  - validation\n```\n\nThose labels were too general.\n\nAlmost every coding task is an implementation. Many tasks are features. Every completed task needs validation.\n\nThe agent therefore had a reasonable incentive to load most of the documentation.\n\nI had modularized the repository knowledge.\n\nI had not created strict routing.\n\nThe second version added:\n\nBefore explaining how that routing behaved, it helps to understand the repository used in the experiment.\n\nThe benchmark used a real game project built with Godot and C#.\n\nThe task was a gameplay-input change involving camera controls.\n\nThe agent needed to:\n\nThis repository was useful for the experiment because it contained different kinds of knowledge:\n\n```\nGodot and gameplay conventions\ngeneral engineering rules\nasset rules\nvalidation commands\n```\n\nA camera-input task should need the Godot-specific guidance.\n\nIt should not need asset guidance.\n\nIt should only need broader engineering guidance if the implementation crosses an architectural boundary.\n\nThat made the task a practical test of selective routing.\n\nI created two isolated versions of the same repository.\n\n```\nAGENTS.md\n```\n\nAll repository guidance lived in one concise root document.\n\n```\nAGENTS.md\n.harness/\n```\n\nThe root file acted as an entry point.\n\nThe Harness contained:\n\n```\n.harness/\n├── harness.yaml\n├── GODOT.md\n├── ENGINEERING.md\n└── ASSETS.md\n```\n\nFor this camera-input task, the expected route was:\n\n```\nAGENTS.md\n+ .harness/harness.yaml\n+ .harness/GODOT.md\n```\n\nAcross all 15 measured Harness runs, the agent inspected the manifest and the Godot-specific guidance without loading the unrelated engineering or asset documents.\n\nThat confirmed that the routing mechanism itself was working.\n\nThe remaining question was whether it improved the complete execution.\n\nBoth variants used the same:\n\nI ran:\n\n```\n3 warm-ups per variant\n15 measured runs per variant\n30 measured runs total\n```\n\nThe order alternated between variants.\n\nThe prompt was identical in every run.\n\nEach repository started from a clean, frozen commit.\n\nEvery measured execution satisfied the benchmark acceptance criteria.\n\n```\nAGENTS:  15/15 accepted\nHARNESS: 15/15 accepted\n```\n\nThe Harness did not reduce reliability.\n\n| Metric | AGENTS | Harness | Harness difference |\n|---|---|---|---|\n| Duration | 188.36 s | 185.61 s | -1.46% |\n| Total input tokens | 803,295 | 829,006 | +3.20% |\n| Uncached input tokens | 63,147 | 61,859 | -2.04% |\n| Output tokens | 5,947 | 5,630 | -5.32% |\n| Reasoning tokens | 987 | 952 | -3.63% |\n| Commands | 14.40 | 14.53 | +0.93% |\n| Failed commands | 6.07 | 7.00 | +15.38% |\n| Changed files | 6.20 | 6.00 | -3.23% |\n\nThe Harness was slightly faster.\n\nIt used slightly fewer uncached input tokens and produced less output.\n\nIt also used more total input tokens and executed more failed commands.\n\nNone of those mean differences was large enough to justify a strong claim that the Harness was reliably faster or cheaper.\n\nThe honest conclusion is:\n\nAverage efficiency was approximately tied.\n\nThat was not the most important result.\n\nThe concise monolithic `AGENTS.md`\n\nwas roughly 14 KB.\n\nThe Harness route loaded about:\n\n```\nsmall AGENTS.md       ~0.8 KB\nharness.yaml          ~3.3 KB\nGODOT.md              ~7.3 KB\n--------------------------------\nrouted context       ~11.5 KB\n```\n\nThe Harness successfully avoided unrelated documents, but the direct reduction was only a few kilobytes.\n\nThat is small compared with a complete agent execution containing source files, search results, command output, tests, diffs, and repeated tool context.\n\nThe experiment averaged roughly 62,000 uncached input tokens per run.\n\nSaving a few kilobytes of instructions was unlikely to transform the total.\n\nThis changed the main question.\n\nThe Harness may not primarily optimize the size of the initial prompt.\n\nIt may optimize how consistently the agent navigates the rest of the task.\n\n| Metric | AGENTS standard deviation | Harness standard deviation | Reduction |\n|---|---|---|---|\n| Duration | 46.67 s | 25.10 s | 46.2% |\n| Total input tokens | 239,991 | 108,745 | 54.7% |\n| Uncached input tokens | 19,211 | 9,855 | 48.7% |\n| Output tokens | 1,430 | 772 | 46.0% |\n| Commands | 3.27 | 1.13 | 65.6% |\n\nIn this sample, the Harness produced:\n\nI did not begin this project with variance as the primary hypothesis.\n\nBut predictability may matter more than a small improvement in the mean.\n\nLower variance improves:\n\nThe Harness did not make every run dramatically cheaper.\n\nIt made the execution path more repeatable.\n\nA monolithic instruction file gives the agent every repository rule at once.\n\nThe agent must decide which ones matter while already performing the task.\n\nA routed Harness makes the selection explicit:\n\n```\ntask classification\n        ↓\nrepository route\n        ↓\nrequired context\n        ↓\noptional escalation\n        ↓\nselected validation\n```\n\nThis does more than organize Markdown files.\n\nIt constrains the agent’s decision space.\n\nThat provides a plausible explanation for the lower variance, although this benchmark cannot prove causation.\n\nThe strongest conclusion supported by the data is:\n\nRepository-defined routing preserved implementation success and made coding-agent behavior more consistent.\n\nNot in the simple way I originally imagined.\n\nThe experiment does not prove that the Harness always uses fewer tokens or is reliably faster.\n\nIt does show that:\n\nSo the result is not:\n\n```\nThe Harness is cheaper.\n```\n\nIt is:\n\n```\nThe Harness is more predictable.\n```\n\nThat may be the more valuable engineering property.\n\nThe current benchmark tested one task inside one Godot repository.\n\nThe next step should test whether the routing model generalizes to a repository with clearly different technical domains.\n\nA backend-and-frontend project is a stronger test because it contains knowledge that should not be loaded for every task:\n\n```\nbackend\ndatabase\nAPI contracts\nfrontend\naccessibility\ndesign system\nintegration tests\ndeployment\n```\n\nThe next benchmark should include:\n\nA useful Harness should remain narrow for isolated work and expand only when the task crosses a boundary.\n\nA practical setup would be a TypeScript monorepo:\n\n```\napps/\n├── api/\n└── web/\n\npackages/\n├── contracts/\n└── testing/\n```\n\nValidation could be fully automated:\n\n```\nlint\ntype-check\nunit tests\nintegration tests\nproduction build\nheadless end-to-end tests\n```\n\nThe next benchmark should measure not only tokens and duration, but also:\n\nThe central question for Part 3 is:\n\nCan repository-defined routing stay narrow for backend-only and frontend-only work, then expand correctly for a cross-stack task?\n\nPersistent memory is promising.\n\nA Harness could eventually learn architectural decisions, recurring failures, conventions, validation paths, and known issues.\n\nBut adding memory now would mix two different hypotheses:\n\nMemory also makes later runs depend on earlier ones, which complicates a controlled comparison.\n\nThe cleaner sequence is:\n\n```\nPart 2\nSelective routing and predictability\n\nPart 3\nFull-stack generalization\n\nPart 4\nLifecycle hooks and persistent memory\n```\n\nIf the Harness generalizes in Part 3, memory becomes the next logical layer.\n\nThe original promise was token reduction.\n\nThe second benchmark did not produce a large or statistically conclusive reduction in average token use.\n\nInstead, it revealed a different benefit:\n\nRepository-defined context routing can make coding-agent behavior more predictable without reducing implementation success.\n\nReliable agent workflows need more than correct final code.\n\nThey need bounded behavior:\n\nThat is what the Repository Harness is becoming.\n\nThe first article proposed the idea.\n\nThe first benchmark exposed a routing flaw.\n\nThe second benchmark showed that stricter routing worked and made the agent more consistent.\n\nPart 3 will test whether that result generalizes beyond Godot.\n\nThe question is no longer only:\n\nCan a Harness reduce tokens?\n\nIt is:\n\nCan a repository define a predictable, auditable, and progressively disclosed operating environment for any coding agent?\n\nThe project is available at:\n\n**Repository Harness Specification**\n\n[https://github.com/Repository-Harness-Specification](https://github.com/Repository-Harness-Specification)", "url": "https://wpnews.pro/news/repository-harness-part-2-30-runs-later-it-wasnt-cheaper-it-was-more-predictable", "canonical_source": "https://dev.to/lepsistemas/repository-harness-part-2-30-runs-later-it-wasnt-cheaper-it-was-more-predictable-1d8p", "published_at": "2026-07-20 19:24:35+00:00", "updated_at": "2026-07-20 19:39:38.603529+00:00", "lang": "en", "topics": ["developer-tools", "artificial-intelligence", "ai-agents"], "entities": ["Repository Harness", "Godot", "C#"], "alternates": {"html": "https://wpnews.pro/news/repository-harness-part-2-30-runs-later-it-wasnt-cheaper-it-was-more-predictable", "markdown": "https://wpnews.pro/news/repository-harness-part-2-30-runs-later-it-wasnt-cheaper-it-was-more-predictable.md", "text": "https://wpnews.pro/news/repository-harness-part-2-30-runs-later-it-wasnt-cheaper-it-was-more-predictable.txt", "jsonld": "https://wpnews.pro/news/repository-harness-part-2-30-runs-later-it-wasnt-cheaper-it-was-more-predictable.jsonld"}}