Eight Headless Coding Agents, One Task: Tokens and Cost A new benchmark testing eight headless coding-agent CLIs on a single Python task found that seven of eight agents passed on both runs, but list prices per run varied 18-fold, from $0.0165 (DeepSeek V4 Flash off-peak) to $0.3039 (Claude Opus 5). The benchmark, which independently verified results with unittest, also found that two CLIs reported costs inconsistent with vendor rate cards: Claude Code's total_cost_usd matched Anthropic's list price for Opus 5 but overstated DeepSeek by ~27×, while Grok's reported cost was exactly 17% of its rate card. This is a headless coding-agent cost-per-task benchmark: one small, fixed Python task run through eight agent CLIs — Claude Opus 5, GPT-5.6 Sol, Gemini 3.7 Flash, Grok 4.6, Kimi K3, Qwen3.8-Max, GLM-5.3 and DeepSeek V4 Flash — twice each, with the tokens each CLI reported, the seconds each took, whether the result actually passed its tests, and what those tokens cost at the vendor’s published rate. It is an original measurement, not a survey. Nothing here was estimated: every token count was read from the CLI’s own output or, for one CLI that prints none, from its local session log , and every pass/fail comes from us running python3 -m unittest -v in the agent’s directory rather than trusting the agent’s “DONE”. The complete 18-row dataset is in section 03; the prose around it is packaging. Two things turned out to matter more than the headline cost figures. First, the harnesses themselves report cost inconsistently — one CLI’s dollar figure matches our rate-card calculation to the cent on its own vendor and is 27× wrong on a third-party endpoint, another’s is exactly 17% of its published rate card with no explanation found. Second, the cheapest-looking agent produced no usable output on the invocation most people would write. Both are in the table, not footnoted out of it. - 01Seven of eight agents shipped a passing deliverable on both runs.Opus 5, GPT-5.6 Sol, Grok 4.6, Kimi K3, Qwen3.8-Max, GLM-5.3 and DeepSeek V4 Flash each produced dedupe.py and test dedupe.py that passed our independent unittest run, twice. Gemini 3.7 Flash passed in three of four attempts but only once into the working directory. - 02List price per run spans $0.0165 to $0.3039 — an 18× spread.DeepSeek V4 Flash at off-peak rates sits at the bottom; Claude Opus 5 at the top, with cache writes — priced at the $10 rate in Table 2 — accounting for 76% and 71% of its two list-price figures. The chart in section 03 shows every run as its own bar rather than an average. - 03Cache reads were 46.0% to 95.4% of input tokens in every run.Uncached input ran from 4 to 61K tokens; cache reads from 22K to 203K. Pricing all input at the uncached rate would have overstated the full-run list price by between 1.4× and 4.8× across these 18 rows. - 04Two CLI cost fields disagree with the rate card, in opposite directions.Claude Code’s total cost usd matched Anthropic list price to the cent for Opus 5 but overstated DeepSeek by ~27× and Z.ai by ~2.8–2.9× through the Anthropic-compatible shim. Grok’s CLI-reported cost was exactly 0.17 of the rate-card figure in both runs — an open discrepancy we do not explain. - 05Free is the plan, not the model.Six agents ran on flat subscriptions, Qwen on a prepaid plan at 10% of standard rate, and only DeepSeek was pay-as-you-go — about $0.03 for two runs. The list-price column is what the same tokens would cost on the vendor’s API; the plan column is what was actually paid. 01 — The instrumentOne task, eight CLIs, empty directories. The task was chosen to be small, unambiguous and self-verifying: a command-line CSV deduplicator plus a three-test unittest suite, standard library only, with an exact stdout contract. It fits in two files of roughly 140 lines and has a binary outcome — either python3 -m unittest -v passes three tests in the agent’s directory or it does not. The prompt, verbatim: You are in an empty directory. Build a small Python 3 command-line tool and its tests, using ONLY the standard library. 1. Create dedupe.py. When run as python3 dedupe.py contacts.csv deduped.csv it must: - read the input CSV, which has a header row with exactly the columns name,email,phone - normalise email strip surrounding whitespace, lowercase and phone keep digits only - drop duplicate rows by normalised email, keeping the first occurrence - write the surviving rows with their normalised email and phone to the output CSV, with the same header - print one line to stdout: read=