Harness Arena: How to Compare AI Coding Agent Harnesses Head-to-Head

Harness Arena, a platform sponsored by On Demand, launched a blind-comparison tool that runs multiple AI coding agent harnesses — including Codex, On Demand, Hermes, OpenCode, OpenClaw, and Claude Code — on the same task with the same Qwen 3.8 model configuration so users can score anonymized outputs from 1 to 10 before harness names are revealed. The platform's leaderboard breaks results down by coding, research, and operations categories with rating, win rate, votes, wins/losses, and median completion time, and its backend is open source on GitHub under an MIT license. Anyone can submit a new benchmark using existing task sets or a custom Excel template, though early leaderboard numbers can be thin on votes.

Harness Arena: How to Compare AI Coding Agent Harnesses Head-to-Head Harness Arena lets you judge Claude Code, Codex, OpenCode, and other agent harnesses blind on real coding tasks. Here's how the comparison works. What is Harness Arena? Harness Arena is a platform that lets you compare AI agent harnesses the software wrapper around a model, including its instructions, tools, and execution environment by looking at the actual work they produce on shared coding tasks. Instead of trusting a benchmark score you can’t inspect, you review anonymized outputs from multiple harnesses running the same task and model configuration, score them yourself, and then see which harness produced which result. The platform is sponsored by On Demand, which also fields a harness in the arena. TL;DR - Harness Arena compares agent harnesses , not just models, by giving multiple tools the same task and model configuration and comparing what they actually produce. - The battle log shows recorded runs you can filter by status, category, or outcome, including rounds still awaiting judgment that you can evaluate right away. - Judging is blind : you score every required output from 1 to 10 before the harness names behind each submission are revealed. - The leaderboard breaks results down by category coding, research, operations and shows rating, win rate, votes, wins/losses, and median completion time. - Early leaderboard numbers can be thin on votes , so a small rating gap between harnesses with only a handful of judgments shouldn’t be read as a real difference. - Anyone can submit a new benchmark , choosing their own tasks, harnesses, and model, either from existing task sets or by uploading a custom Excel template. - The backend is open source on GitHub under an MIT license , though self-hosting still carries model and hosting costs. How does Harness Arena actually compare agent harnesses? The core idea is controlled comparison. Harness Arena takes a single coding task, a fixed model, and runs it through several different harnesses at once, each in its own separate workspace. Because the model is held constant, any difference in output comes down to how each harness handles the work: reading the relevant code, making an edit, and verifying the change actually worked. That distinction matters because a harness is more than a thin wrapper. It includes the instructions given to the model, the tools it can call, and the execution environment it runs in. Two harnesses driving the same underlying model can produce very different results depending on how well they scaffold that process. Harness Arena is built to surface exactly that gap. In the demonstrated example, a task called “3D globe application, targeted overlay removal” was run across six harnesses, including Codex, On Demand, Hermes, OpenCode, OpenClaw, and Claude Code, all using the same Qwen 3.8 model configuration. The comparison view lays these out in a table with one column per harness, showing the model used, your own score, community ratings, and deliverable counts. What does the judging process look like? Judging happens through a workflow the site calls its methodology. You start by reading the task description and its rubric, then inspect the anonymous outputs from each harness side by side. Every required output gets scored from 1 to 10, and only after you’ve submitted scores for all of them do the harness names get revealed. For a task like the globe overlay removal, that means checking the specific requirement in the prompt does the overlay actually disappear and then testing whether the rest of the application still works as expected. If the original app had rotation or zoom controls, part of a fair evaluation is confirming those still function after the edit. A harness that removes the right element but breaks something else shouldn’t score the same as one that does both correctly. The practical suggestion here is to build a short checklist from the rubric before scoring anything, and apply it consistently across every output. This matters most when one submission looks visually polished but quietly fails a requirement, since a checklist keeps that from skewing the score. Reviewing requires an account. If you’re signed out and try to open a task, you get a sign-in prompt before you can start judging. How does the leaderboard work? The leaderboard sits under its own section in the sidebar and can be filtered by category, such as code, research, or operations, depending on the kind of agent work you care about. Within a category, the table shows rating, win rate, number of votes, wins and losses, and median completion time for each harness. Rating reflects the accumulated judgments from people scoring outputs. Completion time tells you how long a harness took to finish its runs. Both numbers matter on their own, but neither should be read in isolation: a harness that finishes fast but leaves broken code isn’t actually faster in any useful sense, and a high rating built on very few votes isn’t necessarily reliable. That second point is worth taking seriously. On early views of the coding leaderboard, several harnesses can show the same displayed rating with only a single vote behind each one. A one-point or one-vote difference at that stage doesn’t tell you much. Checking how much judging volume sits behind a score is a basic sanity check before treating any ranking as settled. Can you run your own benchmark? Yes. The “new benchmark” option in the sidebar also requiring sign-in lets you set up a custom comparison. You choose which tasks to run, which harnesses to test, and which model they’ll use. Task data can come from a few places. There’s a link to Artificial Analysis for building a data set, or you can download an Excel template, put one task per row, and upload your own set of coding tasks. This is useful if you have specific, recurring problems you want to test agents against rather than relying only on generic benchmark tasks. The platform also ships with tasks already loaded, so you can start without preparing anything yourself. For harnesses, options in the demonstrated setup included Claude Code, Codex CLI, and OpenCode, with a minimum of two harnesses required to run any comparison. Including On Demand’s harness requires adding an On Demand API key first, which is also a prerequisite for using the skills section of the platform. Tasks can be filtered by category coding, for instance , and each one includes a “view task” option to read the full prompt and check for any reference files before you commit to running it. On the model side, options seen in the walkthrough included Kimi K2, GLM 4.5 Flash, and Qwen 3.8, with several marked as free to use in the interface at the time. Before submitting, the setup screen shows a running counter of how many tasks and harnesses you’ve selected. Starting small one task, two harnesses keeps the comparison easy to read rather than trying to parse a large grid all at once. Once submitted, a “view progress” screen tracks the run, and completed tasks become available for judging individually through the evaluate screen or the battle log, so you don’t have to wait for an entire batch to finish before reviewing results. Is Harness Arena worth using? For anyone choosing between agent harnesses for real coding work, the value is in the blind, side-by-side inspection rather than a single aggregated number. You get to see actual code and application behavior, not just a score someone else assigned. The battle log and leaderboard are both viewable without an account, which makes it easy to browse existing results before deciding whether to sign in and judge or run your own benchmark. Plans first. Then code. Remy writes the spec, manages the build, and ships the app. The caveats are the ones built into any early-stage benchmarking platform: vote counts are still thin in places, so leaderboard rankings should be treated as directional rather than definitive until more judgments accumulate. The backend is open source under an MIT license on GitHub, so it can be self-hosted, though running your own instance still means covering model and hosting costs. Frequently Asked Questions What is an AI agent harness? A harness is the software layer around an AI model that handles instructions, available tools, and the execution environment the model operates in. Two harnesses can use the identical model and still produce different results because of how each one scaffolds tasks like reading code, editing files, and verifying changes. How is Harness Arena different from a normal model benchmark? Standard benchmarks typically compare models. Harness Arena holds the model constant and compares the harnesses running it, isolating how much the surrounding tooling and instructions affect real coding outcomes. Do I need an account to use Harness Arena? You can browse the battle log and leaderboard without signing in. An account is required to judge tasks or to submit a new benchmark. Which harnesses are included in Harness Arena? The demonstrated round included Codex, On Demand, Hermes, OpenCode, OpenClaw, and Claude Code, tested against a shared model configuration. Available harnesses for custom benchmarks included Claude Code, Codex CLI, OpenCode, and On Demand which requires its own API key . Can I trust the current leaderboard rankings? Treat early rankings cautiously. Some coding-category results have shown identical ratings across harnesses with only one vote recorded each, which isn’t enough judging volume to treat small differences as meaningful.