cd /news/ai-agents/eight-ai-agents-against-the-evidence · home › topics › ai-agents › article
[ARTICLE · art-173657] src=pub.towardsai.net ↗ pub= topic=ai-agents verified=true sentiment=· neutral

Eight AI Agents Against the Evidence

GitHub announced on November 17 that Copilot Fusion Max scored 71.2% on Terminal-Bench, while n8n Labs reported 98% for its Nod agent on assistant tasks a week earlier, two figures the article says measure different things. The comparison of eight agents — including Nod 0.8.4, Hermes Agent 0.23.2 and OpenClaw 2026.11.5 — argues that a benchmark score depends on the model, harness, environment, budget, reasoning effort and grader, so the percentage alone cannot tell a buyer which product fits a given task.

recommendation

Before trusting an agent benchmark, check which harness, environment, budget and grader produced the score; treat vendor-defined task sets like Nod's 98% as measuring only that vendor's simulated tasks.

more in playbook: Evals and testing for LLM apps
by read10 min views7 publishedNov 21, 2026

71.2% for one, 98% for the other. Those two numbers about AI agents came out a week apart, and they do not measure the same thing.

I picked eight agents you keep running into right now, from the terminal to the personal assistant, and went back over what their public tests actually prove. The goal is not to crown a winner. It is to know how to read a percentage before handing a real task to any of them.

On November 17, GitHub announced 71.2% for Copilot Fusion Max on Terminal-Bench. A week earlier, n8n Labs introduced Nod and its 98% on assistant tasks. The two numbers answer different questions. The first measures how many terminal exercises get solved. The second measures how many tasks, defined by Nod’s own authors, get completed in simulated accounts and websites. To choose between these products, you first need to know what you want them to do. GitHub announcement · Nod’s evaluation method

An agent combines at least three parts. The model proposes the next step. The harness presents the tools, manages the session and interprets the calls. The environment decides which files, applications and operations those tools can actually reach. A benchmark score can also depend on budget, reasoning effort and the grader. This selection therefore keeps public results for some agents and, for others, a documented architecture worth examining. How Nod’s harness works · Terminal-Bench 4.1 method

Nod is a standalone app in research preview, available without an n8n account. Version 0.8.4 shipped on November 17. The chat opens a persistent session for substantial work. The server uses Pi as its harness, PostgreSQL for task state, and separate containers for commands and Chromium. Messages, files and results flow back into the conversation, while sessions can progress independently. With a connected n8n instance, Nod can build a new workflow, test it on sample data and leave it as a draft until you ask for it to be published. Version 0.8.4 · Sessions · Building a workflow

Hermes Agent, in stable version 0.23.2 dated November 16, combines a terminal agent, messaging channels, scheduled jobs and persistent memory. What Nous Research calls learning shows up here as notes and procedures the system can keep and reuse. The Curator mechanism can maintain some of the skills the agent produces. The operator can lock a skill and require a review before new skill writes take effect. Version 0.23.2 · Curator · Configuration

For a routine that runs every day, what matters is what persists. The scheduler documentation says durable memory is available during a scheduled run, while the conversational context of the previous run is not carried over automatically. Isolation also depends on the terminal backend you choose. The local and Docker backends give commands different access to the machine. How Cron works · Docker deployment modes Version 2026.11.5, released on November 17, is the reference point here. OpenClaw’s Gateway is a persistent service that links messaging channels, sessions, control clients and paired nodes. This architecture is useful when a task starts in Telegram, continues on a server and has to deliver its result back to a messaging channel. Version 2026.11.5 · Gateway architecture

The setting to watch is effective execution isolation. In the reference configuration, global sandboxing is off by default. A per-agent or per-role policy can enforce another mode. The Gateway stays on the host when tools are sandboxed. The project describes one trust domain per Gateway, which matters if several people with different levels of trust want to share one install. A comparison between two self-hosted agents should state their effective configuration, not just the product names. Sandboxing · Trust boundary

Claude Code reads a repository, applies changes and runs commands. Anthropic separates the permissions that decide which tools are allowed from the sandbox that actually limits what the shell can reach. That sandbox covers shell commands, while file tools, hooks and MCP servers follow their own boundaries. To review a Claude Code run, you need to know which tool produced the observed effect. How Claude Code works · Sandbox scope

Codex offers a CLI that runs on the user’s machine, plus IDE, desktop and cloud surfaces. Its CLI is under the Apache 2.0 license. The local configuration separates write rights, network access and the policy that triggers an approval request. Where the agent runs and where the model computes remain two separate properties. CLI repository · Sandbox and approvals

GitHub Copilot fits into the issue and pull request cycle. Fusion is a configuration that picks and coordinates models, with several effort levels. The Fusion Max row in the table is therefore a routing-and-execution result inside Copilot. It tells you about the whole system. Copilot in repositories · November 17 release

Leaderboard as read on November 18, 2026, limited to the three agents in this section; the full leaderboard includes other harnesses, such as Antigravity CLI. Values after ± are 95% confidence half-widths, in points. Cost is the figure the leaderboard reports for the full suite. The leaderboard measures these complete configurations on suite 4.1. “Max” is each configuration’s own label and does not make their budgets identical. Public leaderboard · 4.1 grader revision

The right-hand column deserves as much attention as the resolution rate. At the same score, the two Claude Code configurations cost $4.7k and $16.5k for the full suite. Codex with GPT-6.2 Sol reaches 61.5% for $0.3k. Four points of resolution can come with a budget more than fifty times larger.

The table shows configuration results. To study the harness more directly, GitHub published a view on November 17 that compares Copilot CLI, Claude Code and Codex CLI when model and effort level match. Fusion configurations are excluded. GitHub authored this benchmark and is one of the vendors compared. The view is useful for its same-model comparison method, alongside the public Terminal-Bench leaderboard. GitHub’s harness comparison

OpenHands offers a coding agent and an SDK to manage sessions, define tools and move execution between local and remote workspaces. Its MLSys 2026 paper gives an instructive case. On SWE-bench Verified, with Claude Sonnet 4.5, the authors report 64.6% with OpenHands V0 and 72.8% with V1. With Sonnet 4, both versions sit at 68.0%. The authors attribute part of the Sonnet 4.5 gain to extended-thinking support in V1. The result documents how OpenHands evolved for these models under this protocol. OpenHands SDK · MLSys 2026 paper, table 4

Earendil announced Pi 1.0 on October 1. Pi works as a terminal agent and embeds into other software through a TypeScript SDK or RPC. You choose the model, tools and extensions, then decide how to handle session events. Its documentation separates the trust given to a project’s resources from the process’s actual permissions. An extension or tool runs with those permissions. For a product that embeds Pi, isolation and approval rules are designed at the level of the product or the environment that launches it. Nod is one example, combining Pi with its containers and its server-side check. Pi 1.0 announcement · Pi and security · SDK · Nod architecture

WildClawBench runs 60 human-written tasks in the real harnesses of OpenClaw, Hermes, Claude Code or Codex. The authors report up to 18 points of difference for the same model when the harness changes. Grading mixes deterministic criteria, state inspection and model judgment. The result covers the protocol and the versions evaluated in May. Paper · Tasks and code

ClawBench, without the s of ClawsBench, evaluates journeys on real websites and intercepts the final request before it is sent. In its V2 snapshot of May 20, Hermes with Claude Opus 4.7 scores 44.6% under the lenient grading, then 24.6% under the strict grading, on the same 130 tasks. The gap shows how much the definition of success matters. An intercepted request judged compliant stands for an action the agent would have submitted. Grading, leaderboards and traces

ClawsBench, with an s, measures completion and dangerous actions separately across five simulated services. Across the complete configurations studied, the researchers report 39 to 64% success and 7 to 33% dangerous actions depending on the configuration. These are lab numbers tied to specific scenarios and permissions. For comparison, n8n Labs reports 98% completion and zero unauthorized actions detected over 678 runs of its own October suite, with GPT-6.1 Sol and simulated accounts and websites. Each rate depends on the task set and on its definition of an unauthorized action. ClawsBench · Nod protocol

A small community test offers another level of evidence. A public repository compares OpenClaw and Hermes on a single multi-source research task in June, with GPT-5.5. Both results have the same number of sources per category, while the logs show different timings and call counts. The traces let you examine both runs. Counting references is enough to check the structure, but their accuracy and the quality of the synthesis need a different kind of reading. Community test protocol and traces

For a first test between personal assistants, start with a simple task. Take five test messages and a fictional calendar. The agent has to find two free slots, explain any conflict and prepare a draft. You can check the slots, the recipient and the absence of any send in the final state. On the code side, start from the same commit, give each agent the same issue and grade the changes with the same tests. In both cases, the useful evidence comes down to four things. A task can be completed thanks to a human intervention, or completed by going beyond the rights granted. Those cases deserve separate rows in the record. To study the harness, keep the same model and equivalent tools when you can. To evaluate the product the way a reader will use it, also measure its usual configuration, and name it. GitHub’s same-model view · Final-state checks in ClawsBench

On my homelab, these agents do not compete. They hold different jobs in the same chain. A pilot AI writes the brief for each step. An agent carries it out with limited rights: Claude Code for most steps, or Pi with a 27-billion-parameter model running on my own GPUs, for tasks that must not leave the machine. A verifier, also Claude Code or Pi, then reviews the result with a model that first had to pass an exam. I give the GO at the end.

That setup forced the four questions above on me long before I found them in public benchmarks. Every step leaves a report, its tests and a way to roll back. On a recent step, verification took 6 minutes for 22 minutes of work. That is the kind of number, measured on my own tasks, that I hold up against the announced percentages.

For a task inside my accounts, I want to see what was done and under which authorization. For a change to a system, I want to be able to open the diff, rerun the tests and find the decision. Both requirements can live together. Above all, they give a concrete way to read public results and to pick the first task to try with each agent. Method and review. This article is a documentary review as of November 18, 2026. No new test of the eight agents was run for it. Team announcements, research benchmarks and the community test are credited to their authors. The versions cited identify each product’s documented state, while each result keeps the date and protocol of its own test. To suggest a correction or a counter-example, the most useful details are the version, the model, the scenario and a link to the result or trace.

Originally published at https://mounir-raji.dev on November 18, 2026.

I write about running AI on my own hardware, with real measurements. Next: why my main local model is busy less than 1% of the time. Get it by email: https://mounir-raji.dev/newsletter/?from=medium

Eight AI Agents Against the Evidence was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.

── more in #ai-agents 4 stories · sorted by recency
── more on @github 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/eight-ai-agents-agai…] indexed:0 read:10min 2026-11-21 · —