Hugging Face Harness Leaderboard

Hugging Face's Harness Leaderboard updated OpenClaw to version 2026.11.1, where OpenClaw scores about 54 on DeepSeek-V4.1-Flash, up from 33 and 38, while Hermes Agent fell from first to ninth on DeepSeek-V4.1-Flash and now ranks first on Flint, tied with four others. The board re-ran eight harnesses on tasks where the model never saw the images they opened, so their scores and ranks moved, and every cell holds all 594 attempts except Kimi Code's on Flint at 593, with 38 more passes counted as cheating. A new FAQ documents how each harness was run and what counts as cheating, with each run executing on HF Jobs and trials reused from earlier runs copied rather than re-run.

Harness Leaderboard - OpenClaw moves to 2026.11.1 with each model's limits: on DeepSeek-V4.1-Flash it scores about 54, up from 33 and 38. - Eight harnesses ran again where the model never saw the images they opened, so their scores and ranks move. - Hermes Agent drops from first to ninth on DeepSeek-V4.1-Flash and now ranks first on Flint, tied with four others. - Every cell has all 594 attempts except Kimi Code's on Flint 593 . Thirty-eight more passes count as cheating. - A new FAQ shows how each harness was run and what counts as cheating. Full changelog https://huggingface.co/spaces/joelniklaus/harness-leaderboard/blob/main/CHANGELOG.md Which coding-agent harness should you run your open model in? Same model and weights; providers checked equivalent. Same reasoning effort, three attempts with confidence intervals, every trajectory public. Accuracy vs. cost Each dot is a harness running the same model. Up and to the left is better. The highlighted staircase is the Pareto frontier : harnesses no other harness beats on both accuracy and cost. Hover a dot for its numbers, confidence interval included. Leaderboard Click a column to sort. Highlighted cells are the best value in each column. Harnesses in a bracket are statistically tied and share a rank. Per-trial values average over scored trials. Tokens, calls and model time are measured outside the harness by the same metering proxy; agent time is Harbor's timer around the harness process. Reliability over three attempts For unattended work, solving a task every time matters more than solving it once: each bar runs from the share of tasks a harness solved on all three attempts to the share it solved on at least one, with a mark at its average attempt. Harness vs. noise A task flips when one attempt passes and the other fails. If swapping the harness flips no more tasks than re-running the same harness, the harness choice changes little beyond chance. Runs Each run executes on HF Jobs. Trials reused from earlier runs are copied, not re-run. Click a run for per-benchmark progress and its cost & time files. Trajectory scan atif-scan over the trials this view scores, per harness: how many trajectories it read, and the flagged trials critical, high or medium severity , split by how the board counts them. Below, the review queue. Method Everything that could favour one harness is pinned or equalised. FAQ Short answers to questions about the board. Open a question to read its answer. Does the ranking tell me which agent to use in my editor? No. Every harness runs headless on a fixed task: there is no person in the loop, and the benchmark's time limit applies. The ranking says how well a harness works on its own. In an editor, with you steering, a harness can rank differently. How exactly was each harness run? Each harness runs at the version in the table, headless, with its own system prompt and tools. The table gives its launch command and the main changes made to run it here. Claude Code runs in its standard mode, not its minimal mode. The Method says how each harness reaches the model and gets the model's limits. <task , <effort and <model stand for the task text, the reasoning effort and the model of the run. Paths and the proxy's settings are left out. Why only open models? The board answers one question: which harness to run an open model in. The models are open weights, served by Hugging Face Inference Providers or the board's own endpoints, and for some trials of one model by servers a volunteer lent the board, checked like a provider. The sandboxes get an inference-only token and no keys for closed-model APIs. Models behind a closed API are out of scope for now. What does "statistically tied" mean? Harnesses in one bracket of the leaderboard are statistically tied and share a rank. The board compares two harnesses on the same tasks and computes a 95% confidence interval for the difference in their scores, from a bootstrap over tasks. If that interval includes zero, the gap could be chance, and the two count as tied: the first group is the leader and every harness tied with it, and the next starts at the best harness not yet grouped. Tied is not equal: the data cannot separate them. How is cost computed? Cost is the tokens a trial used, as the provider reports them, times a price per token, with cached input at the cached rate. The price is the cheapest list price among the providers checked to behave the same, whichever provider served the trial. A model on a dedicated GPU endpoint is priced by tokens too: the hourly price of its GPU divided by the tokens it serves per hour in a load test. This is a comparison price, not an invoice. How is cheating handled? An attempt that gets the task's answer instead of working it out counts as a fail, even when the grader passes it. Access is enough: we do not need to show that the agent used it. Only a pass can change: a failed trial already counts as a fail. Counts as a fail: - Reading the task's hidden tests or its reference solution. - Receiving a reference answer. - Getting the fix or the answer from outside and being able to use it: the upstream version of the code that already holds the fix, another agent's published solution or transcript, or a search result that shows or describes the reference solution. - Calling another model to do the task's work. - Tampering with the tests, the verifier or the reward, or special-casing the grader's expected output, so that the trial passes without solving the task. Does not count as a fail: - An attempt that returned nothing: a blocked request, a DNS error or an empty result. - General documentation, package installs, and library source that does not hold the task's answer. - Reading the task's visible files, its repository's git history and the installed libraries, and running its visible tests, unless the history holds the task's fix. - Something that arrives only after the agent has finished and checked its own solution, because it could not use it. Unknown: the record is missing or cut off, so we cannot tell. The trial keeps its result. Some trajectories can be checked only in part, so what we confirm is a minimum. How we check: every trajectory is scanned with atif-scan https://github.com/huggingface/atif-scan , which flags answer lookups, reads of test or solution paths, tampering and network use that the benchmark does not allow. We read every flagged pass, and the decision cites the step that shows the access. Trials of a benchmark with a public answer key are also matched against it automatically. Can I get my harness added? Ask in a discussion on the Space's Community tab https://huggingface.co/spaces/joelniklaus/harness-leaderboard/discussions : there is no submission form. A harness fits when it runs headless against a custom model endpoint and needs no vendor account or key, and open source ones come first. A harness that is added joins the board in a release of its own, once it has run on every model. What does a harness need for us to evaluate it? The board needs six things from a harness: - A custom model endpoint. A base URL, an API key and a model name, on the OpenAI Chat Completions or Responses API or the Anthropic Messages API. We point it at our metering proxy, which measures cost and sets the reasoning effort. - A headless mode. It takes the task as a prompt, runs its tools in the task's container and finishes without asking for approval. - A plain install. It installs on Linux in a container at a version we can pin, with no account, sign-in or vendor key. - Settings for the model's limits. An option for the model's context window and longest reply, or it reads both from the endpoint. Without them, a harness falls back on defaults for models it does not know: OpenClaw, for one, asks for at most 8,192 tokens per reply unless it is told the model's longest reply. - No other services. A run needs no service but the model's endpoint, because some tasks run without network access. Telemetry and update checks must be switchable off or fail without harm. - Terms that let us publish. Its licence and terms of service must not forbid publishing benchmark results. We run a pinned version with the harness's own prompts and tools, and disclose the changes we make to run it. To ask for yours, open a discussion on the Space's Community tab https://huggingface.co/spaces/joelniklaus/harness-leaderboard/discussions ; there is no submission form.