A model with a 33% pass rate turned out to be the cheapest way to finish a task in a recent 10-model benchmark. The same benchmark shows why that result is less of a free lunch than it looks. Here is the metric behind it and how to compute it on your own prompts.
The metric
Cost per successful task = everything the model spent across every attempt (failed, retried, timed out) divided by the number of attempts that passed.
Per-token price cannot see retries, malformed outputs or runs that never finish. This metric sees all of them.
The numbers
Arize and Fireworks ran 10 models on 40 real command-line tasks from Terminal-Bench, 6 trials each, 2,400 runs. The graded outcome is the task's own test script. A selection:
| Model | Pass rate | Mean $ per attempt | $ per successful task | Attempts per success |
|---|---|---|---|---|
| gpt-oss-120b | 33% | $0.018 | $0.054 | 3.0 |
| gemini-3.1-flash-lite | 40% | $0.026 | $0.063 | 2.5 |
| GPT-5.5 | 67% | $0.424 | $0.636 | 1.5 |
| Kimi K3 | 66% | $0.441 | $0.670 | 1.5 |
| Claude Sonnet 5 | 49% | $0.494 | $1.014 | 2.1 |
| gemini-3.5-flash | 23% | $0.288 | $1.233 | 4.3 |
Source: Arize and Fireworks, cost per successful task. Three things to read from it:
By difficulty, price tier did not predict who wins the easy tasks: GPT-5.5 passed 69% of them, Kimi K3 (also frontier-priced) 100%, and gpt-oss-120b 65% at roughly a twenty-fourth of the cost per attempt. On hard tasks the gap was large (GPT-5.5 51% against 14% for gpt-oss-120b). With 6 trials per cell, the source puts the pass rates at about plus or minus 6 points, so read the difficulty split as a direction, not a decimal.
What does and does not transfer
These are agentic terminal tasks, not classification or extraction, so do not read the dollar figures as your numbers. The method transfers.
Run it on your own prompts (an afternoon)
The headline of that benchmark is the one worth remembering: the label (open or closed, frontier or budget) was the wrong axis. Cost per successful task, on your task, was the right one.