{"slug": "what-is-an-ai-agent-harness-why-does-it-matter-for-the-model", "title": "What Is an AI Agent Harness? Why Does It Matter for the Model?", "summary": "OpenAI's GPT-6.2 Astra scored 70.3% on ARC-AGI-3 with ARC Prize's Standard harness and 99.8% with its Provider Adapter harness at the same max reasoning effort, per ARC Prize leaderboard data as of November 17, 2026, at a cost of $4,107 versus $24,169. A May 2026 position paper, \"Stop Comparing LLM Agents Without Disclosing the Harness,\" found that swapping harnesses for one model moved pass@1 on 100 SWE-bench Verified tasks by 8.5 to 13.0 points, while swapping models under one harness moved scores by 2.5 to 5.0 points, with two models trading places in 6 of 9 cross-harness comparisons. The author concludes an agent score belongs to the model, harness, settings and test setup together, and that a score without those details cannot be compared with another.", "body_md": "Hi everyone. OpenAI’s GPT-6.2 Astra scored **70.3%** on ARC-AGI-3 with one harness and **99.8%** with another. Same model, same benchmark, same reasoning effort (max). ARC Prize’s leaderboard lists both numbers, data as of November 17, 2026.\n\nIf you didn’t know, a harness is the code around a model, the loop that calls it, the tools, the memory, the checks. In March 2026 LangChain’s Vivek Trivedy wrote it as a formula, “Agent = Model + Harness”. Since then a popular claim says the harness matters more than the model. This story was written with the assistance of AI: Claude helped draft the text, and ChatGPT made the cover and the diagram.\n\nThe short version:\n\nYou know that a model on its own only writes text. To work as an agent, it needs code that calls it in a loop and runs the tools it asks for. The same code decides what the model sees, keeps memory between steps, limits the damage it can do and checks its work. That code is the **harness**. For example, Claude Code and Codex are harnesses around a model.\n\nThe name “harness engineering” took off in February 2026, when Mitchell Hashimoto used it. An eval harness is something else, it is just the code that runs a benchmark.\n\n**The same model can score very differently.** ARC Prize ran GPT-6.2 Astra twice with the same settings. The first run used a basic harness (ARC calls it Standard). The second used a harness that lets the model remember what it thought in earlier steps (ARC calls it Provider Adapter). The second run scored much higher and cost far less: $4,107 against $24,169.\n\nIt is not a one-off. At max effort, **Claude Fable 5.2 goes from 65.7% to 94.9% and GPT-6.2 Sol from 54.9% to 98.2%** when ARC Prize swaps the harness. The gain depends on the model: **Claude Haiku 5.5 only goes from 0.5% to 2.8%**.\n\nTwo more things helped. In February 2026 OpenAI described a product of about a million lines of code, built in about 1,500 pull requests, with no code typed by hand. And **a team can change the harness, but not the model it rents.**\n\nMost numbers here come from the companies that built the tools, so treat them as signs, not proof.\n\nThe most controlled test I found is a May 2026 position paper, “Stop Comparing LLM Agents Without Disclosing the Harness”. The authors ran 3 models (GLM-5.1, GPT-5.4, Kimi K2.6) with 3 harnesses (minimal, improved, full). Each pair ran the same 100 SWE-bench Verified tasks, twice, and was scored by pass@1 (the share of tasks solved on the first try).\n\n**Changing the harness for one model moved the score by 8.5 to 13.0 points. Changing the model with one harness moved it by 2.5 to 5.0 points.** In 6 of 9 comparisons across harnesses, two models swapped places.\n\nBut the paper has limits. The authors built the three harnesses themselves, the sample is small, and they only claim this for long multi-step tasks and models of similar strength. Other work points the other way:\n\nHarness effects are real and can be large. They depend on the model, the task and the setup.\n\nThe harness is real engineering, and it produced the biggest gaps above. The numbers do not prove it matters more than the model. They do show that **an agent score belongs to the model, harness, settings and test setup together**. A score without those details cannot be compared with another one.\n\nWhich part of your own agent setup would you measure first?\n\n**If you enjoyed this article, I’d be grateful for your support**\n\nI’m a Product Manager with an engineering background (ex SWE), focused on building, scaling and growing products. I’m especially interested in new technologies, particularly AI/ML, and how they can be applied in real workflows.\n\nEmail: [akzhankalimatov@gmail.com](mailto:akzhankalimatov@gmail.com)\n\n✅ Connect with me on [LinkedIn](https://www.linkedin.com/in/akzhankalimatov/), [X (Twitter)](https://x.com/akzh4n)\n\n✅ Press and hold the 👏 to give up to 50 claps for this article\n\n[What Is an AI Agent Harness? Why Does It Matter for the Model?](https://pub.towardsai.net/what-is-an-ai-agent-harness-why-does-it-matter-for-the-model-73dca33707a1) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/what-is-an-ai-agent-harness-why-does-it-matter-for-the-model", "canonical_source": "https://pub.towardsai.net/what-is-an-ai-agent-harness-why-does-it-matter-for-the-model-73dca33707a1?source=rss----98111c9905da---4", "published_at": "2026-11-21 15:01:05+00:00", "updated_at": "2026-11-21 15:43:10.056593+00:00", "lang": "en", "topics": ["ai-agents", "large-language-models", "ai-research"], "entities": ["OpenAI", "GPT-6.2 Astra", "ARC Prize", "ARC-AGI-3", "Claude Fable 5.2", "Claude Haiku 5.5", "LangChain", "Vivek Trivedy"], "also_reported_by": [], "takeaway": "On 100 SWE-bench Verified tasks, swapping harnesses for a single model shifted pass@1 by 8.5 to 13.0 points versus 2.5 to 5.0 points for swapping models, so report the harness, settings and test setup alongside any agent score.", "recommendation": {"advice": "Disclose your harness, settings and test setup with every agent benchmark result, and re-run comparisons across at least three harnesses before attributing score gaps to the model.", "playbook": "evals-testing", "playbook_url": "https://wpnews.pro/playbooks/evals-testing"}, "alternates": {"html": "https://wpnews.pro/news/what-is-an-ai-agent-harness-why-does-it-matter-for-the-model", "markdown": "https://wpnews.pro/news/what-is-an-ai-agent-harness-why-does-it-matter-for-the-model.md", "text": "https://wpnews.pro/news/what-is-an-ai-agent-harness-why-does-it-matter-for-the-model.txt", "jsonld": "https://wpnews.pro/news/what-is-an-ai-agent-harness-why-does-it-matter-for-the-model.jsonld"}}