{"slug": "how-our-open-source-coding-agent-scored-84-5-on-terminal-bench-2-1-for-27", "title": "How our open source coding agent scored 84.5% on Terminal-Bench 2.1 for $27", "summary": "The team behind AgenC Core, an MIT-licensed open source terminal coding agent, reported that its harness paired with DeepSeek V4.1 Flash scored 84.5% on Terminal-Bench 2.1 at a total cost of $27, versus at least $134 for every entry on the official leaderboard. The run is not on the official leaderboard because community submissions for 2.1 are closed, and the team has filed issue #253 asking maintainers to run the agent, with the Harbor adapter published in the repository. The team said transcript review surfaced harness-level problems, not model problems, and that each fix applies to every model the harness runs.", "body_md": "AgenC Core is an open source (MIT) coding agent for the terminal. Our goal is to multiply model capability per dollar of inference: to get more verified work out of a model for every dollar spent on it.\n\nTo test that goal, we ran Terminal-Bench 2.1 with DeepSeek V4.1 Flash. AgenC Core scored 84.5%, and the whole run cost $27. Every entry on the official leaderboard spent at least $134. All 445 trials are public, with full trajectories.\n\nThis post covers how we ran it, what the number does and does not mean, and what the benchmark taught us about our own harness.\n\nTerminal-Bench measures how well an agent completes real tasks in a terminal: building software, repairing a git history, configuring servers, recovering data. Each trial gives the agent one instruction inside a container, and the task's own tests grade the result.\n\nA leaderboard row is always an agent and a model together. The model supplies the reasoning. The harness around it decides how that reasoning turns into finished work: how the task is planned, which tools exist, how progress is kept, and how results are checked before the agent says it is done. With the same model, a better harness finishes more of the work for the same money.\n\nThe run is public on Harbor Hub: [job 2bf2c84d](https://hub.harborframework.com/jobs/2bf2c84d-83c2-47a3-9341-e635c45bd329).\n\n| Agent and model | Score | Cost | \n|---|---|---|\n| **AgenC Core, DeepSeek V4.1 Flash (max)** | **84.5%** | **$27** | \n| Codex, GPT-6 Astra (high) | 87.4% | $774 | \n| Claude Code, Fable 5 (xhigh) | 83.8% | $553 | \n| Cursor CLI, Grok 4.5 (high) | 79.3% | $134 | \n\nThe AgenC Core row is our public run. The other rows are entries on the official [Terminal-Bench 2.1 leaderboard](https://hub.harborframework.com/datasets/terminal-bench/terminal-bench-2-1?tab=leaderboard&leaderboard=main) as of October 31, 2026, and Cursor CLI's is the cheapest of them. The top entry scored 2.9 percentage points more and cost more than 28 times as much.\n\nTwo caveats. Each row pairs an agent with a model, so this compares combinations, not agents alone. And our run is not on the official leaderboard: community submissions for 2.1 are closed, and only runs done by the maintainers are added. We have asked them to run AgenC Core ([issue #253](https://github.com/harbor-framework/terminal-bench-2-1/issues/253)), and the adapter they would need is public.\n\nThe most useful part of running the benchmark was reading the transcripts. Across our runs, they exposed problems that had nothing to do with the model:\n\n`exec_command` takes `detach: true` to start a service that outlives the session.`$(...)`: 51 of 459 shell calls in one run. Under a full bypass with no sandbox those guards are now lifted, and protected paths stay protected.\nNone of these were model problems, and each fix applies to every model the harness runs. That is what \"multiply model capability\" means in practice.\n\nThe Harbor adapter is in the repository at `runtime/eval/harbor/agenc_agent.py`. It installs the pinned release through the public installer and checks its SHA-256, answers the project trust prompt in the real terminal UI, never changes task timeouts, and writes an ATIF trajectory for every trial.\n\n```\nuv tool install harbor\ngit clone https://github.com/tetsuo-ai/agenc-core\ncd agenc-core\nexport PYTHONPATH=$PWD/runtime/eval/harbor\nDEEPSEEK_API_KEY=... harbor run \\\n  -d terminal-bench/terminal-bench-2-1@sha256:7d7bdc1cbedad549fc1140404bd4dc45e5fd0ea7c4186773687d177ad3a0699a \\\n  -a agenc_agent:Agenc -m deepseek/deepseek-flash \\\n  --ak version=0.19.24 --ak reasoning_effort=max \\\n  -k 5 -n 4 -o ./jobs/agenc-tb21\n```\n\nOur published run also had the egress filter and the logging gateway described above, which are not part of the adapter. The full method is in [docs/eval/terminal-bench.md](https://github.com/tetsuo-ai/agenc-core/blob/main/docs/eval/terminal-bench.md).\n\nAgenC Core works with OpenAI, Anthropic, Grok, Gemini, DeepSeek, OpenRouter and other providers, or with local models through Ollama and LM Studio. It runs in the terminal or headless in scripts and CI with `agenc -p`, and its Goal mode takes a project through implementation, testing and an independent review.\n\nmacOS and Linux:\n\n```\ncurl -fsSL https://get.agenc.ag/install.sh | sh\n```\n\nWindows (PowerShell):\n\n```\niwr -useb https://get.agenc.ag/install.ps1 | iex\n```\n\nThen run `agenc onboard` to choose a provider.\n\nIf you run it on another model or another benchmark, we would like to see your numbers.", "url": "https://wpnews.pro/news/how-our-open-source-coding-agent-scored-84-5-on-terminal-bench-2-1-for-27", "canonical_source": "https://dev.to/pavelevich/how-our-open-source-coding-agent-scored-845-on-terminal-bench-21-for-27-3f2f", "published_at": "2026-11-11 19:46:13+00:00", "updated_at": "2026-11-11 19:52:52.855957+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "ai-research", "large-language-models"], "entities": ["AgenC Core", "DeepSeek", "DeepSeek V4.1 Flash", "Terminal-Bench 2.1", "Harbor", "OpenAI", "Anthropic", "tetsuo-ai"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-our-open-source-coding-agent-scored-84-5-on-terminal-bench-2-1-for-27", "markdown": "https://wpnews.pro/news/how-our-open-source-coding-agent-scored-84-5-on-terminal-bench-2-1-for-27.md", "text": "https://wpnews.pro/news/how-our-open-source-coding-agent-scored-84-5-on-terminal-bench-2-1-for-27.txt", "jsonld": "https://wpnews.pro/news/how-our-open-source-coding-agent-scored-84-5-on-terminal-bench-2-1-for-27.jsonld"}}