Harness-of-Harness: Multi-Day Autonomous Software Development

Researchers from Shanghai Artificial Intelligence Laboratory and National University of Singapore introduced Harness-of-Harness (HoH), a framework that enables LLM-based coding agents to autonomously develop software over multiple days with continual improvement. In evaluations on GameCraft-Bench, FrontierSWE, and ProgramBench, HoH outperformed standalone harnesses across three harness-model pairs (Codex with GPT-5.5, OpenCode with DeepSeek-V4-Pro, and Pi with MiniMax-M3), achieving an average relative gain of 52.25% and a maximum gain of 82.86% after three iterations. In a multi-day deployment with over 70 iterations, HoH autonomously developed a complete first-person-shooter game with storyline, mechanics, visuals, and audio.

Submitted 01 Sept 2026 Harness-of-Harness: Multi-Day Autonomous Software Development with Continual Improvement HZHangfan Zhang /@hangfan-zhang Lei Bai /@lei-bai +1 moreShow less Shuyue Hu /@shuyue-hu Abstract This paper studies autonomous software development, in which LLM-based coding agents transform high-level requirements into complete, functional, and usable software systems without human intervention. We introduce Harness-of-Harness HoH , a framework that enables coding agents to continually improve software during autonomous development. HoH operates on existing coding-agent harnesses, and organizes their executions into iterative planning-coding-testing loops. To sustain improvement across loops, HoH balances repair with capability growth, scopes development into small and verifiable increments, separates implementation-time testing from independent evaluation, and constrains verifiable outputs rather than prescribing agent workflows. It progressively exposes deliverables, role-specific tools, and skills, encourages reuse rather than recreation, and maintains versioned project histories. On GameCraft-Bench, FrontierSWE, and ProgramBench, three harness-model pairs Codex with GPT-5.5, OpenCode with DeepSeek-V4-Pro, and Pi with MiniMax-M3 , HoH consistently outperforms the corresponding standalone harnesses, achieving an average relative gain of 52.25 percent and a maximum gain of 82.86 percent after three iterations. In a multi-day deployment with more than 70 iterations, HoH autonomously develops a first-person-shooter game, featuring a coherent storyline, fully implemented core mechanics, human-playable experience, polished visuals and integrated audio. Github: this https URL Project Page: this https URL AI Overview The Transition to Autonomous Software Development The application of Large Language Models LLMs to software engineering has progressed from simple autocomplete features to more sophisticated coding agents capable of resolving repository-level bugs and navigating complex codebases. However, most contemporary coding agents operate within a "human-in-the-loop" paradigm. In this model, human developers remain responsible for overseeing the agent's actions, providing high-level guidance, reviewing code changes, and intervening when the agent deviates from the project's requirements. Figure 1: Traditional human-in-the-loop development vs. the autonomous approach enabled by Harness-of-Harness HoH . Autonomous software development represents a more complex objective: the transformation of high-level requirements into complete, functional systems without human intervention after the initial specification. This task is fundamentally a long-horizon problem. As development cycles extend, agents face significant challenges in maintaining coherence across thousands of lines of code, managing accumulated technical debt, and recovering from regressions. Existing frameworks, or "harnesses," often treat development as a single episode, which limits their ability to sustain progress over several days or multiple iterations. The "Harness-of-Harness" HoH framework addresses these limitations by structuring the development process into a recurring cycle of planning, coding, and testing. By providing a systematic way to preserve project state and validated evidence across iterations, HoH enables coding agents to achieve continual improvement in software artifacts. The Harness-of-Harness Architecture The HoH framework is not a new coding agent itself but rather a meta-framework that manages existing agent harnesses. A harness defines how an LLM interacts with a code environment, executes actions, and receives feedback. HoH wraps these interactions into an iterative, multi-role workflow. The framework decomposes the development process into three distinct roles, each implemented by a separate invocation of a harness–model configuration: the Project Planner, the Developer, and the QA Tester. Figure 2: The architecture of HoH, detailing the planning–coding–testing loop and the management of artifact and evidence states. The core cycle begins with the Project Planner, which sets the objectives for the current iteration. The Developer then implements these objectives within the codebase. Finally, the QA Tester independently verifies the results against the original requirements and the Planner’s objectives. The output of the Tester provides the grounding evidence for the next iteration's planning phase, creating a closed-loop system for improvement. The Three Specialized Agent Roles To maintain focus and prevent the "forgetting" of requirements that often plagues long-horizon tasks, HoH assigns specific responsibilities and information access to each agent role. Project Planner The Project Planner acts as the architect and strategist. It receives the global software specification and the accumulated execution evidence from the previous loop. It also has read-only access to the current state of the software artifact . Its primary task is to generate a development document that scopes the work for the current iteration . The planning step can be represented as: The document identifies unresolved gaps, prioritizes tasks favoring bug fixes and regressions over new features , and specifies preservation constraints—functionalities that must remain intact during new development. Developer The Developer is responsible for the actual coding and implementation. It operates within the project workspace, guided by the global specification and the specific iteration plan . Unlike the Planner, the Developer has write access to the artifact and is empowered to make technical decisions, use debugging tools, and perform "shift-left" testing—verifying code changes immediately within their local context. The Developer produces an updated artifact : QA Tester The QA Tester provides an independent assessment of the Developer's work. It receives the updated artifact as a frozen, read-only candidate and evaluates it against and . The Tester performs both black-box testing observing user-facing behavior and white-box testing inspecting logs, source code, and configurations . The outcome is a new evidence state : This separation of roles ensures that the person or agent claiming the task is "done" is not the same one verifying the completion, which is a standard best practice in software engineering applied here to autonomous agents. Managing Project State Across Time A critical challenge in multi-day development is the accumulation of failures and suboptimal decisions. HoH manages this by maintaining two complementary states across loops: the Artifact State and the Evidence State . Artifact State : This represents the physical project, including source code, configuration files, and assets. By carrying the artifact forward to the next Developer invocation, HoH enables incremental building rather than starting from scratch each time. Evidence State : This represents the "knowledge" of the project. It includes records of what has been verified to work, what is currently broken, and what requirements are still missing. This state provides the necessary context for the Planner to make informed decisions about what to do next. To handle the growing size of project histories and artifacts, HoH uses a mechanism called Progressive Disclosure . Instead of overwhelming the LLM's context window with every detail of every past iteration, HoH persists detailed logs and versioned histories in the file system. Agents are provided with a concise index of these resources and must explicitly use tools to retrieve specific details when they are relevant to the current task. Empirical Evaluation and Benchmarks The researchers evaluated HoH across three distinct benchmarks—GameCraft-Bench, FrontierSWE, and ProgramBench—using multiple model and harness combinations, including GPT-4o and DeepSeek-V3. Performance on GameCraft-Bench GameCraft-Bench requires agents to build complex, interactive games from scratch. In these evaluations, HoH consistently outperformed "Vanilla" single-pass agents. After three iterations referred to as HoH@3 , agents showed significant improvements in four areas: Core Mechanics, Content Depth, Functional Visuals, and Art and Presentation. Figure 3: Qualitative comparison of games developed by Vanilla agents vs. those developed through HoH@3 iterations. For example, using the Codex harness with GPT-4o, the mean overall score increased from 49.58 Vanilla to 71.52 HoH@3 . The games produced by HoH exhibited more refined gameplay loops, such as guided wall-jump routes in platformers or active special castes in strategy simulations, which the Vanilla agents failed to implement correctly. Results on FrontierSWE and ProgramBench On FrontierSWE , which focuses on real-world software engineering tasks, HoH@3 achieved dominance score gains of 19 to 29 percentage points over standalone baselines. A long-term study showed that HoH could continue to improve quality over as many as ten iterations, effectively doubling the dominance score in some configurations. On ProgramBench , the average test pass rate improved significantly. The iterative nature of HoH allowed agents to catch logic errors that were missed during a single development pass, leading to more robust solutions for algorithmic problems. Case Study: Autonomous FPS Development To test the framework's capability in a truly open-ended, multi-day scenario, the researchers deployed HoH to develop a first-person shooter FPS game titled "Fusepoint" using the Godot engine. This development process lasted over six days and spanned more than 70 iterations. Figure 4: The evolution of Fusepoint, showing the tracking of new and closed issues across a 70-iteration autonomous development cycle. The development followed three distinct phases: Initial Construction: Establishing basic movement, weapon handling, and map importing. Capability Expansion: Adding complex features like enemy AI, HUD elements, lighting polish, and narrative elements. Stabilization: Fixing bugs, refining VFX, and ensuring a playable build. Throughout this process, the system tracked 81 unique issues. HoH demonstrated the ability to resolve regressions—instances where a new feature broke an existing one. For instance, 17 issues that had been previously closed were later reopened and re-fixed, illustrating the framework's capacity for explicit issue management and project traceability. Implications for Future AI Agents The Harness-of-Harness framework demonstrates that autonomous software development is achievable by shifting the focus from individual code generation tasks to a structured, iterative management process. The primary contribution of this work lies in its role-based decomposition and its robust management of project state across time. By treating software development as a process of continual improvement rather than a single-shot execution, HoH provides a blueprint for how AI agents can tackle increasingly complex and long-horizon engineering challenges. The success of the Fusepoint case study suggests that with the right structural constraints and specialized tools, agents can move beyond simple scripts and function completions to become builders of entire software ecosystems. The integration of domain-specific tools like the Godot MCP and structured evidence-based planning points toward a future where autonomous agents can operate as independent engineering teams, managing the entire lifecycle of a software project. MetaGPT: Meta programming for a multi-agent collaborative framework https://www.alphaxiv.org/abs/2308.00352 SWE-Agent: Agent-computer interfaces enable automated software engineering https://www.alphaxiv.org/abs/2405.15793 GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine? https://www.alphaxiv.org/abs/2606.17861