{"slug": "codeforge-v2-building-a-coding-agent-harness-where-llms-propose-deterministic", "title": "CodeForge-v2 — Building a Coding Agent Harness Where LLMs Propose, Deterministic Systems Decide", "summary": "Developer Dienvu1008 released CodeForge-v2, an experimental open-source autonomous software-engineering agent built on a deterministic runtime kernel in which the LLM proposes operations but the kernel enforces policy, tool authorization, task-graph state transitions, budgets, checkpoints and completion decisions. CodeForge-v2 is designed to work with local models via Ollama as well as cloud models, and it includes a failure taxonomy that separates model failures from harness, tool and interaction failures; the project is early-stage and under active development on GitHub. The developer plans to evaluate CodeForge-v2 on SWE-bench but argues benchmark scores alone do not show whether an agent can maintain a clean architecture, preserve invariants during refactoring or avoid architectural debt.", "body_md": "Hey everyone!\n\nLike many developers, I’ve become increasingly dependent on AI coding agents.\n\nThey are incredibly productive. A single developer with a capable coding agent can now accomplish tasks that previously required several developers working together.\n\nBut there is a catch: the more I use them, the more I depend on expensive models. And despite their impressive capabilities, I still find myself dealing with unpredictable execution, hallucinations, limited control, and difficulty understanding what actually happened during a long-running task.\n\nI started wondering:\n\n**Could we build a coding agent harness that makes local and affordable LLMs more useful, while making the overall system more reliable, controllable, and efficient?**\n\nThat question led me to build **CodeForge-v2**.\n\nGitHub: [GitHub - Dienvu1008/CodeForge-v2: An experimental autonomous software-engineering agent built around a deterministic, verifiable runtime kernel. The LLM proposes; the runtime decides. Early-stage and under active development. · GitHub](https://github.com/Dienvu1008/CodeForge-v2)\n\n## \n\nThere are already many excellent coding agents and agent frameworks out there. Rather than forking one of them, I decided to experiment with a different architectural approach.\n\nMy core design principle is:\n\n**LLM proposes. Deterministic systems decide authority and enforce execution.**\n\nIn many agent architectures, the LLM plays a central role in deciding what happens next. The harness provides tools and an execution loop, but the reliability of the overall system still depends heavily on the model.\n\nCodeForge takes a different approach by separating intelligence from execution authority.\n\nThe LLM handles:\n\nThe deterministic runtime kernel handles:\n\n- \nPolicy enforcement and tool authorization.\n- \nTask graph management and state transitions.\n- \nApproval and budget management.\n- \nWorkspace revisions and checkpoints.\n- \nVerification and completion decisions.\n- \nExecution history and provenance.\n\nThe goal is to ensure that an LLM can propose an operation without automatically having the authority to execute it.\n\nI believe this separation is particularly important when building autonomous agents that need to work on software projects over extended periods.\n\n## \n\nI don’t want to build just another ReAct loop with tool calling.\n\nMy longer-term vision is an autonomous software engineering assistant that can:\n\n- \nWork with local models through tools such as Ollama, as well as cloud models.\n- \nExecute software engineering tasks with controlled permissions.\n- \nTrack progress and let developers monitor execution through a dashboard.\n- \nAllow developers to inspect, approve, and control important operations.\n- \nLearn from execution failures through structured feedback and evaluation.\n- \nExpand its capabilities through controlled extensions.\n\nUltimately, I want to explore controlled self-evolution.\n\nHowever, I don’t want an agent to freely rewrite its own governance kernel. I’d rather keep the core runtime responsible for authority and safety while allowing new capabilities to evolve independently under explicit controls.\n\nThis is an important distinction in the direction I’m exploring.\n\n## \n\nI also have a slightly different perspective on evaluating coding agents.\n\nSWE-bench is important, and I plan to use it to evaluate CodeForge. But benchmark scores alone don’t tell us whether an agent can consistently produce and maintain good software.\n\nI care about questions such as:\n\n- \nCan it create a clean and extensible architecture?\n- \nCan it preserve architectural invariants during refactoring?\n- \nCan it improve a codebase without breaking existing functionality?\n- \nCan it avoid accumulating architectural debt?\n- \nCan it maintain useful documentation as a project evolves?\n\nCodeForge incorporates a failure taxonomy to distinguish model failures from harness, tool, and interaction failures.\n\nThe idea is to avoid changing the harness to compensate for limitations that actually belong to the model or execution environment.\n\nI also want to use held-out evaluation sets to detect regressions after optimization.\n\nThis evaluation approach is still evolving, but I believe it is worth exploring.\n\n## \n\nThe deterministic runtime kernel is implemented and serves as the foundation of the project.\n\nThere is still considerable work ahead before CodeForge becomes a comfortable, daily-use coding agent.\n\nMy main priorities are:\n\n**1. Developer experience**\n\nImproving the dashboard, VS Code integration, execution monitoring, and approval workflows.\n\n**2. Environment and toolchain intelligence**\n\nMaking the agent better at identifying missing dependencies, requesting permission to install them, retrieving relevant technical information, and using CI/CD feedback to recover from failures.\n\n**3. Benchmarking and optimization**\n\nRunning SWE-bench experiments, analyzing execution data, and systematically improving the harness.\n\nI’m currently constrained by local storage and compute resources, so I cannot run large-scale experiments as quickly as I’d like.\n\n## \n\nI’m sharing CodeForge because I’d like to connect with people interested in autonomous software engineering and agent infrastructure.\n\nI’m particularly interested in feedback from:\n\n- \nDevelopers building agent harnesses and autonomous coding systems.\n- \nResearchers exploring agent reliability, governance, and evaluation.\n- \nPeople experimenting with local LLMs and Ollama.\n- \nContributors interested in architecture, runtime systems, dashboards, and integrations.\n\nIf you have access to sufficient compute resources and are interested in helping run SWE-bench experiments, I’d also be very grateful. I can provide the setup and configuration; the resulting agent database is enough for me to analyze the execution traces and failures.\n\nYou don’t need to contribute code to help. A technical review, a discussion, or simply trying the project with a local model would already be valuable.\n\n**Repository:** [GitHub - Dienvu1008/CodeForge-v2: An experimental autonomous software-engineering agent built around a deterministic, verifiable runtime kernel. The LLM proposes; the runtime decides. Early-stage and under active development. · GitHub](https://github.com/Dienvu1008/CodeForge-v2)\n\nI’m building this as an open-source project, and I’m looking forward to learning from the community.\n\nThanks for reading!", "url": "https://wpnews.pro/news/codeforge-v2-building-a-coding-agent-harness-where-llms-propose-deterministic", "canonical_source": "https://discuss.huggingface.co/t/codeforge-v2-building-a-coding-agent-harness-where-llms-propose-deterministic-systems-decide/191368#post_2", "published_at": "2026-11-03 07:02:26+00:00", "updated_at": "2026-11-03 07:19:33.007907+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "large-language-models", "ai-safety"], "entities": ["CodeForge-v2", "Dienvu1008", "GitHub", "Ollama", "SWE-bench"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/codeforge-v2-building-a-coding-agent-harness-where-llms-propose-deterministic", "markdown": "https://wpnews.pro/news/codeforge-v2-building-a-coding-agent-harness-where-llms-propose-deterministic.md", "text": "https://wpnews.pro/news/codeforge-v2-building-a-coding-agent-harness-where-llms-propose-deterministic.txt", "jsonld": "https://wpnews.pro/news/codeforge-v2-building-a-coding-agent-harness-where-llms-propose-deterministic.jsonld"}}