Cheapest per token is not cheapest per finished job: what 2,400 runs show

Arize and Fireworks benchmarked 10 models across 40 Terminal-Bench command-line tasks with 6 trials each, totaling 2,400 runs, and found that cost per successful task diverges sharply from per-token pricing. The cheapest model per finished job, gpt-oss-120b, passed only 33% of tasks but cost $0.054 per success versus $1.014 for Claude Sonnet 5 at a 49% pass rate, while GPT-5.5 led on hard tasks at 51%. The authors caution that the dollar figures are specific to agentic terminal work, but the metric — total spend across all attempts divided by passing attempts — transfers to other prompts.

A model with a 33% pass rate turned out to be the cheapest way to finish a task in a recent 10-model benchmark. The same benchmark shows why that result is less of a free lunch than it looks. Here is the metric behind it and how to compute it on your own prompts. The metric Cost per successful task = everything the model spent across every attempt failed, retried, timed out divided by the number of attempts that passed. Per-token price cannot see retries, malformed outputs or runs that never finish. This metric sees all of them. The numbers Arize and Fireworks ran 10 models on 40 real command-line tasks from Terminal-Bench, 6 trials each, 2,400 runs. The graded outcome is the task's own test script. A selection: | Model | Pass rate | Mean $ per attempt | $ per successful task | Attempts per success | |---|---|---|---|---| | gpt-oss-120b | 33% | $0.018 | $0.054 | 3.0 | | gemini-3.1-flash-lite | 40% | $0.026 | $0.063 | 2.5 | | GPT-5.5 | 67% | $0.424 | $0.636 | 1.5 | | Kimi K3 | 66% | $0.441 | $0.670 | 1.5 | | Claude Sonnet 5 | 49% | $0.494 | $1.014 | 2.1 | | gemini-3.5-flash | 23% | $0.288 | $1.233 | 4.3 | Source: Arize and Fireworks, cost per successful task https://arize.com/blog/cost-per-successful-task-ai-model-benchmark/ . Three things to read from it: By difficulty, price tier did not predict who wins the easy tasks: GPT-5.5 passed 69% of them, Kimi K3 also frontier-priced 100%, and gpt-oss-120b 65% at roughly a twenty-fourth of the cost per attempt. On hard tasks the gap was large GPT-5.5 51% against 14% for gpt-oss-120b . With 6 trials per cell, the source puts the pass rates at about plus or minus 6 points, so read the difficulty split as a direction, not a decimal. What does and does not transfer These are agentic terminal tasks, not classification or extraction, so do not read the dollar figures as your numbers. The method transfers. Run it on your own prompts an afternoon The headline of that benchmark is the one worth remembering: the label open or closed, frontier or budget was the wrong axis. Cost per successful task, on your task, was the right one.