{"slug": "show-hn-frontierharness-eval-9-harness-same-model-cost-per-pass-varies-17x", "title": "Show HN: FrontierHarness Eval – 9 harness, same model, cost per pass varies 17x", "summary": "FrontierHarness Eval, a new benchmark from Runta, tested nine AI coding harnesses on the same model and found median cost per successful task varies by 17x, with Claude Code v2.1.237 passing 19 tasks but reaching $18.34 per task, while OpenCode v1.18.19's cost rises to $3.24 per task when failed attempts are included. The evaluation, run on Runta agent runtimes with identical golden checkpoints, highlights that cache hit rate is not cost and quality and cost can diverge.", "body_md": "### Median cost per successful task\n\n### Median cost per task\n\n### Median cache hit rate per successful task\n\n### Median time per successful task\n\n## Beyond the numbers\n\n- 01\n**OpenCode: failures excluded.** It only covers 15 passes. Count failed attempts and the number becomes $3.24 per task.\n\n- 02\n**Cache hit rate is not cost.** A cached 300-turn failure can still burn more than a short cache miss.\n\n- 03\n**Quality and cost can diverge.** Claude Code passes 19 tasks, but reaches $18.34 in cost per task.\n\n## Run your harness on Runta.\n\nIf you want to test your own harness on Runta, we’ll give you $100 in credits to get started.\n\n## Tested harnesses\n\n**Codex**\n\n`v0.148.0`\n\n**DeepSeek Harness**\n\n`v0.1.0-rc.8`\n\n**Claude Code**\n\n`v2.1.237`\n\n**Pi**\n\n`v0.84.2`\n\n**Oh My Pi**\n\n`v17.4.0`\n\n**Kimi Code**\n\n`v0.37.2`\n\n**Exo Harness**\n\n`v0.1.0`\n\n**OpenCode**\n\n`v1.18.19`\n\n**Hermes**\n\n`v0.20.4`\n\n- FrontierHarness v1.0 focuses on software engineering contexts and terminal-based tasks. It may not generalize to other areas of knowledge work.\n- Evaluated on Runta agent runtimes. All harnesses and the task environment are prepared once as a golden checkpoint. Every run is a fresh restore with identical vCPU, memory, disk size, disk contents, and memory state.", "url": "https://wpnews.pro/news/show-hn-frontierharness-eval-9-harness-same-model-cost-per-pass-varies-17x", "canonical_source": "https://frontierharness.org", "published_at": "2026-09-02 16:14:52+00:00", "updated_at": "2026-09-02 16:23:00.701436+00:00", "lang": "en", "topics": ["ai-research", "ai-tools", "ai-infrastructure"], "entities": ["FrontierHarness", "Runta", "Claude Code", "OpenCode", "Codex", "DeepSeek Harness", "Kimi Code", "Hermes"], "alternates": {"html": "https://wpnews.pro/news/show-hn-frontierharness-eval-9-harness-same-model-cost-per-pass-varies-17x", "markdown": "https://wpnews.pro/news/show-hn-frontierharness-eval-9-harness-same-model-cost-per-pass-varies-17x.md", "text": "https://wpnews.pro/news/show-hn-frontierharness-eval-9-harness-same-model-cost-per-pass-varies-17x.txt", "jsonld": "https://wpnews.pro/news/show-hn-frontierharness-eval-9-harness-same-model-cost-per-pass-varies-17x.jsonld"}}