# Cheapest per token is not cheapest per finished job: what 2,400 runs show

> Source: <https://dev.to/grain64/cheapest-per-token-is-not-cheapest-per-finished-job-what-2400-runs-show-1484>
> Published: 2026-11-05 00:47:31+00:00

A model with a 33% pass rate turned out to be the cheapest way to finish a task in a recent 10-model benchmark. The same benchmark shows why that result is less of a free lunch than it looks. Here is the metric behind it and how to compute it on your own prompts.

**The metric**

Cost per successful task = everything the model spent across every attempt (failed, retried, timed out) divided by the number of attempts that passed.

Per-token price cannot see retries, malformed outputs or runs that never finish. This metric sees all of them.

**The numbers**

Arize and Fireworks ran 10 models on 40 real command-line tasks from Terminal-Bench, 6 trials each, 2,400 runs. The graded outcome is the task's own test script. A selection:

| Model | Pass rate | Mean $ per attempt | $ per successful task | Attempts per success | 
|---|---|---|---|---|
| gpt-oss-120b | 33% | $0.018 | $0.054 | 3.0 | 
| gemini-3.1-flash-lite | 40% | $0.026 | $0.063 | 2.5 | 
| GPT-5.5 | 67% | $0.424 | $0.636 | 1.5 | 
| Kimi K3 | 66% | $0.441 | $0.670 | 1.5 | 
| Claude Sonnet 5 | 49% | $0.494 | $1.014 | 2.1 | 
| gemini-3.5-flash | 23% | $0.288 | $1.233 | 4.3 | 

Source: [Arize and Fireworks, cost per successful task](https://arize.com/blog/cost-per-successful-task-ai-model-benchmark/).

Three things to read from it:

By difficulty, price tier did not predict who wins the easy tasks: GPT-5.5 passed 69% of them, Kimi K3 (also frontier-priced) 100%, and gpt-oss-120b 65% at roughly a twenty-fourth of the cost per attempt. On hard tasks the gap was large (GPT-5.5 51% against 14% for gpt-oss-120b). With 6 trials per cell, the source puts the pass rates at about plus or minus 6 points, so read the difficulty split as a direction, not a decimal.

**What does and does not transfer**

These are agentic terminal tasks, not classification or extraction, so do not read the dollar figures as your numbers. The method transfers.

**Run it on your own prompts (an afternoon)**

The headline of that benchmark is the one worth remembering: the label (open or closed, frontier or budget) was the wrong axis. Cost per successful task, on your task, was the right one.