{"slug": "harness-of-harness-multi-day-autonomous-software-development", "title": "Harness-of-Harness: Multi-Day Autonomous Software Development", "summary": "Researchers from Shanghai Artificial Intelligence Laboratory and National University of Singapore introduced Harness-of-Harness (HoH), a framework that enables LLM-based coding agents to autonomously develop software over multiple days with continual improvement. In evaluations on GameCraft-Bench, FrontierSWE, and ProgramBench, HoH outperformed standalone harnesses across three harness-model pairs (Codex with GPT-5.5, OpenCode with DeepSeek-V4-Pro, and Pi with MiniMax-M3), achieving an average relative gain of 52.25% and a maximum gain of 82.86% after three iterations. In a multi-day deployment with over 70 iterations, HoH autonomously developed a complete first-person-shooter game with storyline, mechanics, visuals, and audio.", "body_md": "Submitted 01 Sept 2026\n\n# Harness-of-Harness: Multi-Day Autonomous Software Development with Continual Improvement\n\n[HZHangfan Zhang](/@hangfan-zhang)\n\n[Lei Bai](/@lei-bai)\n\n## +1 moreShow less\n\n[Shuyue Hu](/@shuyue-hu)\n\n## Abstract\n\nThis paper studies autonomous software development, in which LLM-based coding agents transform high-level requirements into complete, functional, and usable software systems without human intervention. We introduce Harness-of-Harness (HoH), a framework that enables coding agents to continually improve software during autonomous development. HoH operates on existing coding-agent harnesses, and organizes their executions into iterative planning-coding-testing loops. To sustain improvement across loops, HoH balances repair with capability growth, scopes development into small and verifiable increments, separates implementation-time testing from independent evaluation, and constrains verifiable outputs rather than prescribing agent workflows. It progressively exposes deliverables, role-specific tools, and skills, encourages reuse rather than recreation, and maintains versioned project histories. On GameCraft-Bench, FrontierSWE, and ProgramBench, three harness-model pairs (Codex with GPT-5.5, OpenCode with DeepSeek-V4-Pro, and Pi with MiniMax-M3), HoH consistently outperforms the corresponding standalone harnesses, achieving an average relative gain of 52.25 percent and a maximum gain of 82.86 percent after three iterations. In a multi-day deployment with more than 70 iterations, HoH autonomously develops a first-person-shooter game, featuring a coherent storyline, fully implemented core mechanics, human-playable experience, polished visuals and integrated audio. Github: this https URL Project Page: this https URL\n\n## AI Overview\n\n## The Transition to Autonomous Software Development\n\nThe application of Large Language Models (LLMs) to software engineering has progressed from simple autocomplete features to more sophisticated coding agents capable of resolving repository-level bugs and navigating complex codebases. However, most contemporary coding agents operate within a \"human-in-the-loop\" paradigm. In this model, human developers remain responsible for overseeing the agent's actions, providing high-level guidance, reviewing code changes, and intervening when the agent deviates from the project's requirements.\n\n*Figure 1: Traditional human-in-the-loop development vs. the autonomous approach enabled by Harness-of-Harness (HoH).*\n\nAutonomous software development represents a more complex objective: the transformation of high-level requirements into complete, functional systems without human intervention after the initial specification. This task is fundamentally a long-horizon problem. As development cycles extend, agents face significant challenges in maintaining coherence across thousands of lines of code, managing accumulated technical debt, and recovering from regressions. Existing frameworks, or \"harnesses,\" often treat development as a single episode, which limits their ability to sustain progress over several days or multiple iterations.\n\nThe \"Harness-of-Harness\" (HoH) framework addresses these limitations by structuring the development process into a recurring cycle of planning, coding, and testing. By providing a systematic way to preserve project state and validated evidence across iterations, HoH enables coding agents to achieve continual improvement in software artifacts.\n\n## The Harness-of-Harness Architecture\n\nThe HoH framework is not a new coding agent itself but rather a meta-framework that manages existing agent harnesses. A harness defines how an LLM interacts with a code environment, executes actions, and receives feedback. HoH wraps these interactions into an iterative, multi-role workflow. The framework decomposes the development process into three distinct roles, each implemented by a separate invocation of a harness–model configuration: the Project Planner, the Developer, and the QA Tester.\n\n*Figure 2: The architecture of HoH, detailing the planning–coding–testing loop and the management of artifact and evidence states.*\n\nThe core cycle begins with the Project Planner, which sets the objectives for the current iteration. The Developer then implements these objectives within the codebase. Finally, the QA Tester independently verifies the results against the original requirements and the Planner’s objectives. The output of the Tester provides the grounding evidence for the next iteration's planning phase, creating a closed-loop system for improvement.\n\n## The Three Specialized Agent Roles\n\nTo maintain focus and prevent the \"forgetting\" of requirements that often plagues long-horizon tasks, HoH assigns specific responsibilities and information access to each agent role.\n\n### Project Planner\n\nThe Project Planner acts as the architect and strategist. It receives the global software specification and the accumulated execution evidence from the previous loop. It also has read-only access to the current state of the software artifact . Its primary task is to generate a development document that scopes the work for the current iteration . The planning step can be represented as:\n\nThe document identifies unresolved gaps, prioritizes tasks (favoring bug fixes and regressions over new features), and specifies preservation constraints—functionalities that must remain intact during new development.\n\n### Developer\n\nThe Developer is responsible for the actual coding and implementation. It operates within the project workspace, guided by the global specification and the specific iteration plan . Unlike the Planner, the Developer has write access to the artifact and is empowered to make technical decisions, use debugging tools, and perform \"shift-left\" testing—verifying code changes immediately within their local context. The Developer produces an updated artifact :\n\n### QA Tester\n\nThe QA Tester provides an independent assessment of the Developer's work. It receives the updated artifact as a frozen, read-only candidate and evaluates it against and . The Tester performs both black-box testing (observing user-facing behavior) and white-box testing (inspecting logs, source code, and configurations). The outcome is a new evidence state :\n\nThis separation of roles ensures that the person (or agent) claiming the task is \"done\" is not the same one verifying the completion, which is a standard best practice in software engineering applied here to autonomous agents.\n\n## Managing Project State Across Time\n\nA critical challenge in multi-day development is the accumulation of failures and suboptimal decisions. HoH manages this by maintaining two complementary states across loops: the **Artifact State** and the **Evidence State**.\n\n**Artifact State ():** This represents the physical project, including source code, configuration files, and assets. By carrying the artifact forward to the next Developer invocation, HoH enables incremental building rather than starting from scratch each time.**Evidence State ():** This represents the \"knowledge\" of the project. It includes records of what has been verified to work, what is currently broken, and what requirements are still missing. This state provides the necessary context for the Planner to make informed decisions about what to do next.\n\nTo handle the growing size of project histories and artifacts, HoH uses a mechanism called **Progressive Disclosure**. Instead of overwhelming the LLM's context window with every detail of every past iteration, HoH persists detailed logs and versioned histories in the file system. Agents are provided with a concise index of these resources and must explicitly use tools to retrieve specific details when they are relevant to the current task.\n\n## Empirical Evaluation and Benchmarks\n\nThe researchers evaluated HoH across three distinct benchmarks—GameCraft-Bench, FrontierSWE, and ProgramBench—using multiple model and harness combinations, including GPT-4o and DeepSeek-V3.\n\n### Performance on GameCraft-Bench\n\nGameCraft-Bench requires agents to build complex, interactive games from scratch. In these evaluations, HoH consistently outperformed \"Vanilla\" (single-pass) agents. After three iterations (referred to as HoH@3), agents showed significant improvements in four areas: Core Mechanics, Content Depth, Functional Visuals, and Art and Presentation.\n\n*Figure 3: Qualitative comparison of games developed by Vanilla agents vs. those developed through HoH@3 iterations.*\n\nFor example, using the Codex harness with GPT-4o, the mean overall score increased from 49.58 (Vanilla) to 71.52 (HoH@3). The games produced by HoH exhibited more refined gameplay loops, such as guided wall-jump routes in platformers or active special castes in strategy simulations, which the Vanilla agents failed to implement correctly.\n\n### Results on FrontierSWE and ProgramBench\n\nOn **FrontierSWE**, which focuses on real-world software engineering tasks, HoH@3 achieved dominance score gains of 19 to 29 percentage points over standalone baselines. A long-term study showed that HoH could continue to improve quality over as many as ten iterations, effectively doubling the dominance score in some configurations.\n\nOn **ProgramBench**, the average test pass rate improved significantly. The iterative nature of HoH allowed agents to catch logic errors that were missed during a single development pass, leading to more robust solutions for algorithmic problems.\n\n## Case Study: Autonomous FPS Development\n\nTo test the framework's capability in a truly open-ended, multi-day scenario, the researchers deployed HoH to develop a first-person shooter (FPS) game titled \"Fusepoint\" using the Godot engine. This development process lasted over six days and spanned more than 70 iterations.\n\n*Figure 4: The evolution of Fusepoint, showing the tracking of new and closed issues across a 70-iteration autonomous development cycle.*\n\nThe development followed three distinct phases:\n\n**Initial Construction:** Establishing basic movement, weapon handling, and map importing.**Capability Expansion:** Adding complex features like enemy AI, HUD elements, lighting polish, and narrative elements.**Stabilization:** Fixing bugs, refining VFX, and ensuring a playable build.\n\nThroughout this process, the system tracked 81 unique issues. HoH demonstrated the ability to resolve regressions—instances where a new feature broke an existing one. For instance, 17 issues that had been previously closed were later reopened and re-fixed, illustrating the framework's capacity for explicit issue management and project traceability.\n\n## Implications for Future AI Agents\n\nThe Harness-of-Harness framework demonstrates that autonomous software development is achievable by shifting the focus from individual code generation tasks to a structured, iterative management process. The primary contribution of this work lies in its role-based decomposition and its robust management of project state across time.\n\nBy treating software development as a process of continual improvement rather than a single-shot execution, HoH provides a blueprint for how AI agents can tackle increasingly complex and long-horizon engineering challenges. The success of the Fusepoint case study suggests that with the right structural constraints and specialized tools, agents can move beyond simple scripts and function completions to become builders of entire software ecosystems. The integration of domain-specific tools (like the Godot MCP) and structured evidence-based planning points toward a future where autonomous agents can operate as independent engineering teams, managing the entire lifecycle of a software project.\n\n[MetaGPT: Meta programming for a multi-agent collaborative framework](https://www.alphaxiv.org/abs/2308.00352)\n\n[SWE-Agent: Agent-computer interfaces enable automated software engineering](https://www.alphaxiv.org/abs/2405.15793)\n\n[GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine?](https://www.alphaxiv.org/abs/2606.17861)", "url": "https://wpnews.pro/news/harness-of-harness-multi-day-autonomous-software-development", "canonical_source": "https://www.alphaxiv.org/abs/2609.01481", "published_at": "2026-09-03 12:47:56+00:00", "updated_at": "2026-09-03 12:53:17.356458+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-agents", "ai-research"], "entities": ["Shanghai Artificial Intelligence Laboratory", "National University of Singapore", "Harness-of-Harness", "Codex", "GPT-5.5", "OpenCode", "DeepSeek-V4-Pro", "MiniMax-M3"], "alternates": {"html": "https://wpnews.pro/news/harness-of-harness-multi-day-autonomous-software-development", "markdown": "https://wpnews.pro/news/harness-of-harness-multi-day-autonomous-software-development.md", "text": "https://wpnews.pro/news/harness-of-harness-multi-day-autonomous-software-development.txt", "jsonld": "https://wpnews.pro/news/harness-of-harness-multi-day-autonomous-software-development.jsonld"}}