Harness Handbook: The Missing Layer for Editing AI Agents #
An AI agent’s capability comes from its harness as much as from the foundation model behind it — the code that builds prompts, manages state, invokes tools, and drives the control loop. As models, APIs, environments, and requirements change, that harness has to be edited continually, and before anyone can edit it, human or coding agent, they first have to find every code location that implements the target behavior. The paper names this the behavior localization problem and argues it is the central bottleneck in evolving a harness. Its two contributions are the Harness Handbook, a behavior-centric representation built automatically from a codebase, and Behavior-Guided Progressive Disclosure (BGPD), a workflow that walks a coding agent from a high-level behavior down to the concrete implementation sites and re-verifies them against the current source.
The behavior-to-code gap
A harness here is the agent scaffolding and runtime. Production harnesses are large, tightly coupled, and behaviorally distributed: a single capability like “retry a failed tool call with a summarized context” is spread across prompt templates, state objects, tool wrappers, and control flow that live in different files. Change requests are written in the language of behavior (what the system should do) while the repository is organized by files and functions. That mismatch is what makes localization hard: keyword search fails when the code does not mention the words in the request, and reading a large repo in full is expensive and easy to get wrong. Existing aids like repository maps, code search, repository memory, and long-context editing are discussed as the gap this work targets, but none of them is run as an empirical baseline here.
The Harness Handbook reorganizes a repository around runtime behavior instead of module structure. Two invariants keep it useful. Progressive disclosure means an agent descends into detail only when the task needs it, rather than the whole repo. Behavior-implementation alignment means every active entry must still resolve to the current source; entries whose locators no longer resolve are frozen and excluded until refreshed, so the repository stays the source of truth rather than a stale document drifting away from the code.
The representation
The Handbook is a three-level document tree and a complementary state register.
L1 is the system overview: architecture, execution model, the sequence of stages, and global data flow.L2 is the stage or component overview: responsibilities, inputs and outputs, dependencies, and local state.L3 is the source-grounded “unit deep dive”: the leaf entries that each link to a verified location in the repository.- The state-register view records state that is shared across stage boundaries, which makes the shared state explicit.
Two leaf modes trade granularity against cost. In function-as-leaf mode, an L3 entry is a function or code region, and construction starts from a trusted seed skeleton of the execution stages. In file-as-leaf mode, an L3 entry is a whole file, and the skeleton is inferred from per-file summaries; this mode is used when there is no reliable seed or when function-level detail would exceed the budget on a large repository.
Building and maintaining the Handbook
Construction runs in three phases. The first is static fact extraction: language adapters parse the repo into a program graph of functions, signatures, and call edges, keeping only calls that resolve to a real internal function or a named external boundary and logging the rest. This phase is deterministic and uses no LLM. The second is behavioral organization: source units are assigned to the execution-stage skeleton through an iterative proposer-reviewer loop between LLM calls. The third is hierarchical synthesis: the L1-L3 tree and the state-register view are rendered, and each L3 entry is linked to a verified source location.
At modification time, the agent first navigates the handbook using Behavior-Guided Progressive Disclosure (BGPD). Starting from the high-level architecture (L1), it identifies the relevant execution stages (L2), drills down to the corresponding implementation entries (L3), and finally resolves them to the current source code. Importantly, the search is behavior-driven rather than file-driven. BGPD also follows dependencies such as shared state and call relationships, allowing it to discover implementation sites that are spread across multiple files instead of relying on keyword matching or semantic search alone.
After localization, the workflow follows a usual agent pipeline:
Plan the required edits using the localized evidence.Execute the modifications in the repository.Resynchronize the handbook if the repository has changed.
The resynchronization step is one of the practical contributions of the paper. Rather than rebuilding the handbook from scratch after every edit, the system incrementally updates only the affected sections whenever possible. If a code reference can no longer be matched to the repository, it is marked as stale instead of silently pointing to the wrong implementation.
This keeps the handbook synchronized with an evolving codebase while minimizing the cost of maintaining it.
Experiments
On plan-quality win rate, the Handbook-Assisted arm reaches 45.6% versus 26.7% on Terminus-2 and 38.3% versus 28.3% on Codex. The gains come with lower cost rather than a bigger budget: planner tokens per request drop by 12.7% on Codex and 8.6% on Terminus-2. Localization accuracy shows the widest margins. All 24 recall/precision/F1 comparisons favor the Handbook, with F1 gains from +5.0 to +18.8; on Terminus-2 the weaker DeepSeek-V4-Pro planner, guided by the Handbook, reaches file-level F1 of 84.7 against the Opus 4.8 reference and 89.3 against GPT-5.5 — that is, its predicted edit sites approach the stronger reference models’ own localization. The share of requests with zero overlap against the reference never rises and falls by as much as 25.9 points.
One of the more interesting experiments shows that a relatively weak planning model, when equipped with Harness Handbook, produces localization results that are much closer to those generated by much stronger models such as GPT-5.5 and Claude Opus 4.8. The handbook improves both precision (editing the right places) and recall (finding all required locations), while substantially reducing complete localization failures where the planner edits entirely unrelated code.
Conclusions
Harness Handbook belongs to a growing line of work that treats the agent harness, rather than the language model itself, as the primary object of optimization. Recent systems such as Agentic Harness Engineering (AHE), HarnessX, and other harness-evolution frameworks focus on automatically improving prompts, tools, memory, and control flow. Harness Handbook tackles an earlier stage of the workflow: repository understanding. Instead of evolving the harness, it builds a behavior-centric representation that helps humans and coding agents understand where behaviors are implemented before making changes.
What distinguishes the paper is that it reframes behavior localization as a first-class problem. Most coding-agent systems assume the relevant code can be found through search, embeddings, or long-context reasoning. Harness Handbook argues that the missing abstraction is a behavior-to-code map, allowing agents to navigate from high-level behaviors to concrete implementation sites through progressive disclosure. I find this framing compelling because it matches a common experience when editing large agent systems: writing the change is often easier than discovering every place where the behavior lives.
The main limitation is the evaluation. The paper measures the quality of edit plans, not the success of executed edits, and relies largely on agreement with LLM-generated reference plans rather than execution-based ground truth. The experiments are also relatively small and compare only against repository exploration without the handbook, not against dedicated repository-understanding tools. Even so, I think the core idea is both practical and broadly applicable. As agent harnesses continue to grow in size and complexity, behavior-centric repository representations seem like a natural next step for making them easier to understand, maintain, and evolve.