{"slug": "benchmarking-coding-agents-under-evolving-requirements", "title": "Benchmarking Coding Agents Under Evolving Requirements", "summary": "A new arXiv paper, submitted 26 October 2026 and revised 29 October 2026, introduces FeatureLoop, a benchmark of 61 human-reviewed tasks drawn from 34 repositories across five primary programming languages and seven software forms, designed to test coding agents under evolving requirements. Evaluating 12 frontier models, the authors found the strongest model completes 72.1% of tasks when given both evolution and completion feedback, while across all models only 15.4% of observed failed stages recover within two corrective revisions. The results point to incomplete repairs, regressions and model-specific differences in feedback use and token efficiency as obstacles to reliable, low-intervention development.", "body_md": "# Computer Science > Software Engineering\n\n  [Submitted on 26 Oct 2026 (\n\n[v1](https://arxiv.org/abs/2610.35488v1)), last revised 29 Oct 2026 (this version, v2)]\n# Title:Benchmarking Coding Agents under Evolving Requirements\n\n[View PDF](https://arxiv.org/pdf/2610.35488)\n\n[HTML (experimental)](https://arxiv.org/html/2610.35488v2)\n\nAbstract:Software development with coding agents often begins with a rough request that users refine after trying the implementation. Final correctness alone does not capture the repeated guidance this process requires. We introduce FeatureLoop, a benchmark of 61 human-reviewed tasks from 34 repositories across five primary programming languages and seven software forms. Each task follows one development goal in a persistent workspace through evolution feedback, which adds or clarifies requirements, and completion feedback, which reports failures of stated requirements. Cumulative executable checks track new behavior, repair, and preservation of earlier functionality. We evaluate 12 frontier models and find that the strongest completes 72.1% of tasks with both feedback types. Across models, however, only 15.4% of observed failed stages recover within two corrective revisions. Further analysis reveals incomplete repairs, regressions, and model-specific differences in feedback use and token efficiency, identifying obstacles to reliable, low-intervention development.\n    \n\n## Submission history\n\nFrom: Jia Feng [\n[view email](https://arxiv.org/show-email/d7b6b379/2610.35488)]\n\n**Mon, 26 Oct 2026 14:15:53 UTC (7,385 KB)**\n\n[\\[v1\\]](https://arxiv.org/abs/2610.35488v1)\n**[v2]** Thu, 29 Oct 2026 03:37:42 UTC (7,385 KB)\n\n### Additional Features\n\n### References & Citations\n\nLoading...\n\n# Bibliographic and Citation Tools\n\nBibliographic Explorer \n\n*(*[What is the Explorer?](https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer))\nConnected Papers \n\n*(*[What is Connected Papers?](https://www.connectedpapers.com/about))\nLitmaps \n\n*(*[What is Litmaps?](https://www.litmaps.co/))\nscite Smart Citations \n\n*(*[What are Smart Citations?](https://www.scite.ai/))\n# Code, Data and Media Associated with this Article\n\nalphaXiv \n\n*(*[What is alphaXiv?](https://alphaxiv.org/))\nCatalyzeX Code Finder for Papers \n\n*(*[What is CatalyzeX?](https://www.catalyzex.com))\nDagsHub \n\n*(*[What is DagsHub?](https://dagshub.com/))\nGotit.pub \n\n*(*[What is GotitPub?](http://gotit.pub/faq))\nHugging Face \n\n*(*[What is Huggingface?](https://huggingface.co/huggingface))\nScienceCast \n\n*(*[What is ScienceCast?](https://sciencecast.org/welcome))\n# Demos\n\n# Recommenders and Search Tools\n\nInfluence Flower \n\n*(*[What are Influence Flowers?](https://influencemap.cmlab.dev/))\nCORE Recommender \n\n*(*[What is CORE?](https://core.ac.uk/services/recommender))\n# arXivLabs: experimental projects with community collaborators\n\narXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.\n\nBoth individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.\n\nHave an idea for a project that will add value for arXiv's community? [**Learn more about arXivLabs**](https://info.arxiv.org/labs/index.html).", "url": "https://wpnews.pro/news/benchmarking-coding-agents-under-evolving-requirements", "canonical_source": "https://arxiv.org/abs/2610.35488", "published_at": "2026-11-16 18:03:34+00:00", "updated_at": "2026-11-16 18:25:22.380288+00:00", "lang": "en", "topics": ["ai-agents", "ai-research", "ai-tools", "large-language-models", "developer-tools"], "entities": ["FeatureLoop", "arXiv", "Jia Feng"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/benchmarking-coding-agents-under-evolving-requirements", "markdown": "https://wpnews.pro/news/benchmarking-coding-agents-under-evolving-requirements.md", "text": "https://wpnews.pro/news/benchmarking-coding-agents-under-evolving-requirements.txt", "jsonld": "https://wpnews.pro/news/benchmarking-coding-agents-under-evolving-requirements.jsonld"}}