{"slug": "how-to-run-your-own-benchmark-on-harness-arena", "title": "How to Run Your Own Benchmark on Harness Arena", "summary": "Harness Arena, an open-source platform for comparing AI agent harnesses, now lets signed-in users upload their own task datasets, select harnesses such as Codex, Claude Code, OpenCode and Hermes, choose a model, and submit the comparison for blind evaluation. The platform scores outputs from 1 to 10 against a task rubric before revealing harness names, and its leaderboard breaks down rating, win rate, votes, wins and losses, and median completion time by categories including code and research operations. Harness Arena's back end is released under an MIT license, allowing self-hosting, though users still cover model and hosting costs.", "body_md": "# How to Run Your Own Benchmark on Harness Arena\n\nA walkthrough of submitting custom tasks, picking harnesses and models, and judging results on Harness Arena's benchmark platform.\n\n## What is Harness Arena?\n\nHarness Arena is a platform for comparing AI agent harnesses by putting them through the same tasks and judging the work they produce, blind, before revealing which tool made which output. A harness is the software layer around a model: its instructions, tools, and execution environment. Since the model can be held constant across a comparison, differences in output come down to how each harness handles the task, whether that’s reading code, editing it, or checking whether the edit actually worked.\n\n## TL;DR\n\n- **Harness Arena** compares coding and agent tasks across multiple harnesses (Codex, Claude Code, OpenCode, Hermes, and others) using the same model and task for each run.\n- The **battle log** shows recorded runs you can filter by status, category, or outcome, including rounds still awaiting judgment that you can evaluate right away.\n- **Blind judging** means you score outputs from 1 to 10 against a task rubric before the harness names are revealed, which keeps ratings from being skewed by brand recognition.\n- The **leaderboard** breaks down rating, win rate, votes, wins and losses, and median completion time, separated by categories like code or research operations.\n- The **new benchmark flow** lets signed-in users upload their own task datasets, pick which harnesses to test, choose a model, and submit the comparison for evaluation.\n- Early leaderboard data can be thin, so **vote counts matter** : a rating built on one or two votes is not the same as one built on dozens.\n- The platform’s **back end is open source under an MIT license** , so anyone can self-host it, though running it still means covering model and hosting costs.\n\n## Remy doesn't write the code. It manages the agents who do.\n\nRemy runs the project. The specialists do the work. You work with the PM, not the implementers.\n\n## How does the battle log work?\n\nThe battle log is the record of everything that has run through the arena. You can filter by status (queued, in progress, or awaiting judgment), by category, or by outcome. If you just want to try judging without setting up your own run, filtering for rounds awaiting judgment gives you a ready-made starting point.\n\nEach recorded round shows a task, like a “3D globe application, targeted overlay removal” web app task, alongside the harnesses that competed on it. Clicking the expand arrow opens a comparison table with one column per harness, plus rows for the model used, your score, community ratings, and deliverables. Deliverable counts and community ratings stay hidden until judging is complete, so there’s no way to peek at a leading answer before you score it yourself.\n\n## How does blind judging work?\n\nClicking into a task opens the evaluation view, which requires an account. The site’s methodology follows a fixed sequence: read the task and its rubric, inspect the anonymous outputs, and score every required output from 1 to 10. Harness names only appear after all scores are submitted.\n\nFor a task like the globe overlay removal, the useful approach is to check the specific requirement stated in the prompt (does the overlay disappear, does the globe still function), and then also check for regressions the task didn’t ask about, such as rotation or zoom controls that existed before the edit. A harness that satisfies the letter of the task while breaking something else isn’t actually doing better work. Keeping a short checklist based on the rubric helps keep scoring consistent across multiple outputs, especially when one submission just looks more polished without meeting the actual requirement.\n\n## How do you read the leaderboard?\n\nThe leaderboard sits in the sidebar and can be switched between categories like code and research operations depending on what kind of agent work you care about. The table includes rating, win rate, votes, wins and losses, and median completion time.\n\nBoth numbers matter here. A harness with a fast median completion time but a low win rate is finishing quickly and getting rejected, which isn’t actually useful. Vote counts are worth checking before trusting a ranking, too: if several harnesses show the same rating and each has only a single vote behind it, that’s not enough signal to call a winner. Early-stage rankings on any new benchmark platform should be read with that caveat in mind.\n\n## How do you submit your own benchmark?\n\nThe “new benchmark” option in the sidebar is where the submission flow lives, and it also requires signing in. From there:\n\n1. **Bring or build a dataset.** There’s a link to Artificial Analysis for creating a dataset, or you can download an Excel template, put one task per row, and upload it. Existing tasks are also preloaded, so you don’t have to start from a blank dataset.\n2. **Choose harnesses.** Pick at least two from the available options (Claude Code, Codex CLI, OpenCode, and others were shown in testing). Including a harness like On Demand requires adding an API key for that provider first, and the same key requirement applies to the skills section.\n3. **Filter and select a task.** Tasks can be filtered by category, such as code. The “view task” option lets you read the full prompt and any reference files before committing, which matters because tasks can carry specific constraints (like “remove this overlay but don’t touch the rotation controls”).\n4. **Pick a model.** Options observed in the interface included Kimi K2, GLM 4.5 Flash, and Qwen 3, several of which were listed as free to use at the time of testing.\n5. **Check your counts.** Before submitting, the interface shows how many tasks and harnesses are selected. Starting small (one task, two harnesses) keeps the comparison easy to read.\n6. **Submit and track.** After clicking submit for evaluation, “view progress” lets you follow the run. Finished tasks become available for judging immediately through the evaluate view or the battle log, so you don’t have to wait for the full dataset to complete before reviewing results.\n\n## Remy doesn't build the plumbing. It inherits it.\n\nOther agents wire up auth, databases, models, and integrations from scratch every time you ask them to build something.\n\nRemy ships with all of it from MindStudio — so every cycle goes into the app you actually want.\n\n## Is running your own benchmark worth it?\n\nIf you have coding tasks you handle repeatedly, whether that’s a specific refactor pattern, a UI change, or a recurring bug type, building a small custom benchmark gives you a more direct answer than a general leaderboard. General rankings tell you which harness wins most often across a broad task mix. A narrow, self-submitted benchmark tells you which harness handles the exact kind of work you do.\n\nThe tradeoff is effort. You need a dataset (or the willingness to build one from a template), an account, and time to judge outputs yourself if you want scores that reflect your own standards rather than someone else’s rubric. For a quick gut check, browsing the battle log for a relevant category and judging an existing round awaiting judgment is faster.\n\n## Frequently Asked Questions\n\n### What counts as a harness versus a model in Harness Arena?\n\nA model is the underlying LLM (Qwen 3, Kimi K2, GLM 4.5 Flash, etc.). A harness is everything wrapped around it: the system instructions, the tools it can call, and the environment it executes in. Harness Arena holds the model constant within a round so differences in output reflect the harness, not the model.\n\n### Do I need an account to browse the leaderboard?\n\nNo. The public leaderboard and battle log are accessible without signing in. An account is only required to submit judgments on an evaluation or to create a new benchmark.\n\n### How many harnesses can I compare at once?\n\nYou need a minimum of two harnesses to run a comparison. Options included Claude Code, Codex CLI, OpenCode, Hermes, and On Demand’s harness, though On Demand requires its own API key to include in a run.\n\n### Can I use my own tasks instead of the preloaded ones?\n\nYes. You can download an Excel template, add your own tasks (one per row), and upload the dataset, or use the linked Artificial Analysis tool to build one. Preloaded tasks are also available if you want to start without building a dataset from scratch.\n\n### Is Harness Arena open source?\n\nThe back end is available on GitHub under an MIT license, meaning it can be self-hosted. Running your own instance still requires covering model and hosting costs, since the platform itself doesn’t cover inference or infrastructure.", "url": "https://wpnews.pro/news/how-to-run-your-own-benchmark-on-harness-arena", "canonical_source": "https://www.mindstudio.ai/blog/harness-arena-run-own-benchmark/", "published_at": "2026-09-18 00:00:00+00:00", "updated_at": "2026-09-18 10:26:13.917523+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "ai-products", "developer-tools"], "entities": ["Harness Arena", "Codex", "Claude Code", "OpenCode", "Hermes"], "alternates": {"html": "https://wpnews.pro/news/how-to-run-your-own-benchmark-on-harness-arena", "markdown": "https://wpnews.pro/news/how-to-run-your-own-benchmark-on-harness-arena.md", "text": "https://wpnews.pro/news/how-to-run-your-own-benchmark-on-harness-arena.txt", "jsonld": "https://wpnews.pro/news/how-to-run-your-own-benchmark-on-harness-arena.jsonld"}}