{"slug": "doc2harness-benchmarking-how-well-ai-builds-agent-harnesses", "title": "Doc2Harness: Benchmarking how well AI builds agent harnesses", "summary": "A paper submitted to arXiv on 14 October 2026 introduces Doc2Harness, a two-stage benchmark that evaluates how well large language models build the harnesses around coding agents rather than how well they perform under a fixed harness. Across nineteen frontier models, each run with three independent builds on 328 test cases, GPT-6 Astra achieved the highest pass rate at 53.05% while the weakest model reached 27.44%, a gap of roughly twenty-six percentage points. Even the strongest model failed nearly half the test cases, with failures clustering around how cross-cutting functionality is split across modules; the authors call Doc2Harness the first deterministic benchmark for long-horizon harness engineering.", "body_md": "# Computer Science > Software Engineering\n\n  [Submitted on 14 Oct 2026]\n\n# Title:Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents\n\n[View PDF](https://arxiv.org/pdf/2610.20344)\n\n[HTML (experimental)](https://arxiv.org/html/2610.20344v1)\n\nAbstract:Agents driven by large language models are increasingly deployed on long-horizon tasks, where they use tools and modify real workspaces. How well they perform depends as much on the harness around the model as on the model itself. Existing benchmarks measure downstream task success under a fixed harness, leaving harness engineering itself unexplored. We propose Doc2Harness, a two-stage benchmark that evaluates long-horizon harness engineering directly. In Stage 1, a build model operating inside a fixed builder harness receives an interface-only contract skeleton and a development document that consolidates behavior across twelve frontier harnesses, and then constructs a complete candidate harness in an isolated container. In Stage 2, the frozen candidate harness runs in containers separate from the construction environment, one per test case, and is driven end to end by a scripted mock model. We score each test case using four observable channels, namely the requests sent to the model, the files on disk, the process exit code, and the command output; positive controls rule out vacuous passes. Across nineteen frontier models, each evaluated with three independent builds on 328 test cases, GPT-6 Astra achieves the highest pass rate at 53.05%, while the weakest model reaches 27.44%, a gap of roughly twenty-six percentage points. Even the strongest model fails nearly half of the test cases, and failures cluster around how cross-cutting functionality is split across modules. To our knowledge, Doc2Harness provides the first deterministic benchmark for measuring long-horizon harness engineering.\n    \n\n### Additional Features\n\n### References & Citations\n\nLoading...\n\n# Bibliographic and Citation Tools\n\nBibliographic Explorer \n\n*(*[What is the Explorer?](https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer))\nConnected Papers \n\n*(*[What is Connected Papers?](https://www.connectedpapers.com/about))\nLitmaps \n\n*(*[What is Litmaps?](https://www.litmaps.co/))\nscite Smart Citations \n\n*(*[What are Smart Citations?](https://www.scite.ai/))\n# Code, Data and Media Associated with this Article\n\nalphaXiv \n\n*(*[What is alphaXiv?](https://alphaxiv.org/))\nCatalyzeX Code Finder for Papers \n\n*(*[What is CatalyzeX?](https://www.catalyzex.com))\nDagsHub \n\n*(*[What is DagsHub?](https://dagshub.com/))\nGotit.pub \n\n*(*[What is GotitPub?](http://gotit.pub/faq))\nHugging Face \n\n*(*[What is Huggingface?](https://huggingface.co/huggingface))\nScienceCast \n\n*(*[What is ScienceCast?](https://sciencecast.org/welcome))\n# Demos\n\n# Recommenders and Search Tools\n\nInfluence Flower \n\n*(*[What are Influence Flowers?](https://influencemap.cmlab.dev/))\nCORE Recommender \n\n*(*[What is CORE?](https://core.ac.uk/services/recommender))\n# arXivLabs: experimental projects with community collaborators\n\narXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.\n\nBoth individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.\n\nHave an idea for a project that will add value for arXiv's community? [**Learn more about arXivLabs**](https://info.arxiv.org/labs/index.html).", "url": "https://wpnews.pro/news/doc2harness-benchmarking-how-well-ai-builds-agent-harnesses", "canonical_source": "https://arxiv.org/abs/2610.20344", "published_at": "2026-11-10 19:31:36+00:00", "updated_at": "2026-11-10 19:52:41.594320+00:00", "lang": "en", "topics": ["ai-agents", "ai-research", "large-language-models", "ai-tools"], "entities": ["arXiv", "Doc2Harness", "GPT-6 Astra"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/doc2harness-benchmarking-how-well-ai-builds-agent-harnesses", "markdown": "https://wpnews.pro/news/doc2harness-benchmarking-how-well-ai-builds-agent-harnesses.md", "text": "https://wpnews.pro/news/doc2harness-benchmarking-how-well-ai-builds-agent-harnesses.txt", "jsonld": "https://wpnews.pro/news/doc2harness-benchmarking-how-well-ai-builds-agent-harnesses.jsonld"}}