{"slug": "cheapest-per-token-is-not-cheapest-per-finished-job-what-2400-runs-show", "title": "Cheapest per token is not cheapest per finished job: what 2,400 runs show", "summary": "Arize and Fireworks benchmarked 10 models across 40 Terminal-Bench command-line tasks with 6 trials each, totaling 2,400 runs, and found that cost per successful task diverges sharply from per-token pricing. The cheapest model per finished job, gpt-oss-120b, passed only 33% of tasks but cost $0.054 per success versus $1.014 for Claude Sonnet 5 at a 49% pass rate, while GPT-5.5 led on hard tasks at 51%. The authors caution that the dollar figures are specific to agentic terminal work, but the metric — total spend across all attempts divided by passing attempts — transfers to other prompts.", "body_md": "A model with a 33% pass rate turned out to be the cheapest way to finish a task in a recent 10-model benchmark. The same benchmark shows why that result is less of a free lunch than it looks. Here is the metric behind it and how to compute it on your own prompts.\n\n**The metric**\n\nCost per successful task = everything the model spent across every attempt (failed, retried, timed out) divided by the number of attempts that passed.\n\nPer-token price cannot see retries, malformed outputs or runs that never finish. This metric sees all of them.\n\n**The numbers**\n\nArize and Fireworks ran 10 models on 40 real command-line tasks from Terminal-Bench, 6 trials each, 2,400 runs. The graded outcome is the task's own test script. A selection:\n\n| Model | Pass rate | Mean $ per attempt | $ per successful task | Attempts per success | \n|---|---|---|---|---|\n| gpt-oss-120b | 33% | $0.018 | $0.054 | 3.0 | \n| gemini-3.1-flash-lite | 40% | $0.026 | $0.063 | 2.5 | \n| GPT-5.5 | 67% | $0.424 | $0.636 | 1.5 | \n| Kimi K3 | 66% | $0.441 | $0.670 | 1.5 | \n| Claude Sonnet 5 | 49% | $0.494 | $1.014 | 2.1 | \n| gemini-3.5-flash | 23% | $0.288 | $1.233 | 4.3 | \n\nSource: [Arize and Fireworks, cost per successful task](https://arize.com/blog/cost-per-successful-task-ai-model-benchmark/).\n\nThree things to read from it:\n\nBy difficulty, price tier did not predict who wins the easy tasks: GPT-5.5 passed 69% of them, Kimi K3 (also frontier-priced) 100%, and gpt-oss-120b 65% at roughly a twenty-fourth of the cost per attempt. On hard tasks the gap was large (GPT-5.5 51% against 14% for gpt-oss-120b). With 6 trials per cell, the source puts the pass rates at about plus or minus 6 points, so read the difficulty split as a direction, not a decimal.\n\n**What does and does not transfer**\n\nThese are agentic terminal tasks, not classification or extraction, so do not read the dollar figures as your numbers. The method transfers.\n\n**Run it on your own prompts (an afternoon)**\n\nThe headline of that benchmark is the one worth remembering: the label (open or closed, frontier or budget) was the wrong axis. Cost per successful task, on your task, was the right one.", "url": "https://wpnews.pro/news/cheapest-per-token-is-not-cheapest-per-finished-job-what-2400-runs-show", "canonical_source": "https://dev.to/grain64/cheapest-per-token-is-not-cheapest-per-finished-job-what-2400-runs-show-1484", "published_at": "2026-11-05 00:47:31+00:00", "updated_at": "2026-11-05 00:52:17.040587+00:00", "lang": "en", "topics": ["ai-research", "large-language-models", "ai-agents", "ai-tools"], "entities": ["Arize", "Fireworks", "Terminal-Bench", "gpt-oss-120b", "GPT-5.5", "Kimi K3", "Claude Sonnet 5", "gemini-3.5-flash"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/cheapest-per-token-is-not-cheapest-per-finished-job-what-2400-runs-show", "markdown": "https://wpnews.pro/news/cheapest-per-token-is-not-cheapest-per-finished-job-what-2400-runs-show.md", "text": "https://wpnews.pro/news/cheapest-per-token-is-not-cheapest-per-finished-job-what-2400-runs-show.txt", "jsonld": "https://wpnews.pro/news/cheapest-per-token-is-not-cheapest-per-finished-job-what-2400-runs-show.jsonld"}}