Cheapest per token is not cheapest per finished job: what 2,400 runs show
Arize and Fireworks benchmarked 10 models across 40 Terminal-Bench command-line tasks with 6 trials each, totaling 2,400 runs, and found that cost per successful task diverges sharply from per-token pricing. The cheapest…