This is a headless coding-agent cost-per-task benchmark: one small, fixed Python task run through eight agent CLIs — Claude Opus 5, GPT-5.6 Sol, Gemini 3.7 Flash, Grok 4.6, Kimi K3, Qwen3.8-Max, GLM-5.3 and DeepSeek V4 Flash — twice each, with the tokens each CLI reported, the seconds each took, whether the result actually passed its tests, and what those tokens cost at the vendor’s published rate.
It is an original measurement, not a survey. Nothing here was estimated: every token count was read from the CLI’s own output (or, for one CLI that prints none, from its local session log), and every pass/fail comes from us running python3 -m unittest -v
in the agent’s directory rather than trusting the agent’s “DONE”. The complete 18-row dataset is in section 03; the prose around it is packaging.
Two things turned out to matter more than the headline cost figures. First, the harnesses themselves report cost inconsistently — one CLI’s dollar figure matches our rate-card calculation to the cent on its own vendor and is 27× wrong on a third-party endpoint, another’s is exactly 17% of its published rate card with no explanation found. Second, the cheapest-looking agent produced no usable output on the invocation most people would write. Both are in the table, not footnoted out of it.
- 01Seven of eight agents shipped a passing deliverable on both runs.Opus 5, GPT-5.6 Sol, Grok 4.6, Kimi K3, Qwen3.8-Max, GLM-5.3 and DeepSeek V4 Flash each produced dedupe.py and test_dedupe.py that passed our independent unittest run, twice. Gemini 3.7 Flash passed in three of four attempts but only once into the working directory.
- 02List price per run spans $0.0165 to $0.3039 — an 18× spread.DeepSeek V4 Flash at off-peak rates sits at the bottom; Claude Opus 5 at the top, with cache writes — priced at the $10 rate in Table 2 — accounting for 76% and 71% of its two list-price figures. The chart in section 03 shows every run as its own bar rather than an average.
- 03Cache reads were 46.0% to 95.4% of input tokens in every run.Uncached input ran from 4 to 61K tokens; cache reads from 22K to 203K. Pricing all input at the uncached rate would have overstated the full-run list price by between 1.4× and 4.8× across these 18 rows.
- 04Two CLI cost fields disagree with the rate card, in opposite directions.Claude Code’s total_cost_usd matched Anthropic list price to the cent for Opus 5 but overstated DeepSeek by ~27× and Z.ai by ~2.8–2.9× through the Anthropic-compatible shim. Grok’s CLI-reported cost was exactly 0.17 of the rate-card figure in both runs — an open discrepancy we do not explain.
- 05Free is the plan, not the model.Six agents ran on flat subscriptions, Qwen on a prepaid plan at 10% of standard rate, and only DeepSeek was pay-as-you-go — about $0.03 for two runs. The list-price column is what the same tokens would cost on the vendor’s API; the plan column is what was actually paid.
01 — The instrumentOne task, eight CLIs, empty directories. #
The task was chosen to be small, unambiguous and self-verifying: a command-line CSV deduplicator plus a three-test unittest suite, standard library only, with an exact stdout contract. It fits in two files of roughly 140 lines and has a binary outcome — either python3 -m unittest -v
passes three tests in the agent’s directory or it does not. The prompt, verbatim:
You are in an empty directory. Build a small Python 3 command-line tool and its tests, using ONLY the standard library.
1. Create dedupe.py. When run as `python3 dedupe.py contacts.csv deduped.csv` it must:
- read the input CSV, which has a header row with exactly the columns name,email,phone
- normalise email (strip surrounding whitespace, lowercase) and phone (keep digits only)
- drop duplicate rows by normalised email, keeping the first occurrence
- write the surviving rows with their normalised email and phone to the output CSV, with the same header
- print one line to stdout: read=<rows read> written=<rows written> removed=<rows removed>
2. Create test_dedupe.py using unittest (not pytest) with three tests: (a) emails differing only by case or surrounding whitespace collapse to one row, (b) a phone like "+1 (555) 010-2020" becomes "15550102020", (c) an input file with only the header produces an output file with only the header and prints read=0 written=0 removed=0. Tests must create their own temporary input files.
3. Run `python3 -m unittest -v` and make all three tests pass.
Do not create any other files. When finished, reply with exactly one line: DONE
Each agent received the identical prompt as a single headless invocation, in its own freshly created empty directory, with its permission checks defeated so nothing would block on approval. All eight were launched at the same moment. Two full rounds were run (run1
, run2
), plus two additional Gemini-only re-runs that isolate a harness-configuration finding described in section 07. The eight invocations, exactly as executed:
| Agent / model | CLI · version | Exact headless command (cwd = agent’s empty dir) | Effort | Plan it ran on |
|---|---|---|---|---|
| Claude Opus 5 | Claude Code · claude 2.1.241 | claude -p "$TASK" --model opus --effort high --dangerously-skip-permissions --output-format json |
high | Claude Max subscription ($200/mo) |
| GPT-5.6 Sol | Codex CLI · codex-cli 0.149.0 | codex exec --json -s workspace-write --skip-git-repo-check --ephemeral -c model="gpt-5.6-sol" -c model_reasoning_effort="high" "$TASK" |
high | ChatGPT subscription (Codex) |
| Gemini 3.7 Flash | Antigravity CLI · agy 1.1.19 | agy -p "$TASK" --model gemini-3.7-flash-high --dangerously-skip-permissions [--add-dir "$cwd"] --output-format json |
high (model suffix) | Google Workspace / Antigravity plan |
| Grok 4.6 | Grok Build · grok 1.0.5 | grok -p "$TASK" -m grok-4.6 --effort high --output-format json |
high | SuperGrok subscription |
| Kimi K3 | Kimi Code · kimi 0.38.0 | kimi -p "$TASK" --output-format stream-json (headless = auto permission; no flag exists) |
default (always-thinking) | Kimi Allegro subscription (monthly quota) |
| Qwen3.8-Max | Qwen Code · qwen 0.22.0 | QWEN_CODE_SUPPRESS_YOLO_WARNING=1 qwen -y -o json -p "$TASK" |
default | QwenCloud Token Plan (prepaid, 10% of standard pricing) |
| GLM-5.3 | Claude Code → Z.ai Anthropic-compatible endpoint · claude 2.1.241 | ANTHROPIC_BASE_URL=https://api.z.ai/api/anthropic … claude -p "$TASK" --model glm-5.3 --effort high --dangerously-skip-permissions --output-format json |
high | Z.ai Coding Plan subscription |
| DeepSeek V4 Flash | Claude Code → DeepSeek Anthropic-compatible endpoint · claude 2.1.241 | ANTHROPIC_BASE_URL=https://api.deepseek.com/anthropic … claude -p "$TASK" --model deepseek-v4-flash --effort high --dangerously-skip-permissions --output-format json |
high | Pay-as-you-go API key |
Effort is therefore not equalised. Six agents ran at their explicit “high”; Kimi and Qwen ran at their CLI defaults, because neither exposes a comparable headless effort flag in the form used here. Kimi is always-thinking. That asymmetry is stated rather than hidden, and it is one reason this page does not rank models — our earlier reasoning-effort cost-vs-quality benchmarks show how much the effort dial alone moves token counts.
02 — MethodologyWhat was read, what was calculated, and what was left out. #
What was collected. One fixed coding task (reproduced in full in section 01), identical prompt sent to eight headless coding-agent CLIs, each in its own empty directory, each on its own subscription or API key. Per run: wall-clock seconds (measured by the harness from process start to exit, not self-reported); turns or API calls where the CLI reports them; input tokens split into uncached / cache-read / cache-write, output tokens and thinking tokens, all as reported by the CLI itself; the files present in the working directory afterwards; and whether the deliverable works, verified by us running python3 -m unittest -v
in the agent’s directory. The agent’s own “DONE” claim was ignored.
From which sources. Each CLI’s own headless JSON or stream output — Claude Code --output-format json
, Codex --json
, Antigravity --output-format json
, Grok --output-format json
, Qwen -o json
— or, for Kimi only, its local session log under ~/.kimi-code/sessions/
, because the Kimi CLI printed no usage on stdout at all.
On what date. Runs executed 23 August 2026 (UTC), 11:16–11:23, on one machine (an Apple-silicon MacBook Pro), all eight agents launched concurrently. That is the dataset’s as-of date, distinct from this post’s publish date. Vendor rates were read from each vendor’s own pricing page on the same day, with one exception flagged in its own row: the Qwen3.8-Max rate rests on a routing note plus secondary trackers (section 04). The run window sits outside both of DeepSeek’s published peak bands (01:00–04:00 and 06:00–10:00 UTC), so DeepSeek is priced at its off-peak rate.
How many items. 18 run-rows: 8 agents × 2 full runs (run1
, run2
), plus 2 additional Gemini-only re-runs (run1-gemini-retry
, run1-gemini-retry2
) that isolate a harness-configuration finding (section 07).
What was excluded and why. No third run per agent (budget). No attempt to equalise reasoning effort across CLIs: Kimi has no headless effort flag and runs always-thinking, Qwen exposes no comparable flag either, so six of eight ran at an explicit “high” and two ran at their CLI default. No quality scoring beyond pass/fail — the task has a binary outcome and was scored as one.
What was READ and what was CALCULATED. Every token count, turn count, files-in-cwd entry and pass/fail result in the table is READ directly from the CLI’s own output or (Kimi) its session log — none is inferred or estimated. The wall-clock column is MEASURED by the harness around each subprocess, not read from the CLI. The list-price USD column is CALCULATED: uncached input × input rate + cache-read tokens × cache-read rate + cache-write tokens × cache-write rate + output tokens × output rate, using the rates in section 04; thinking tokens are not separately priced, and every published list-price cell reproduces from that formula to four decimal places. The CLI-reported USD column is READ verbatim where the CLI prints a cost, and says NOT REPORTED where it prints none.
Known limitations (as written in the dataset’s methodology file):
- One task, two runs per agent, run concurrently at one time of day on one machine. This is a measurement of this task on this day, not a ranking of models. Variance between run1 and run2 is visible in every column. - The task is small (two files, ~140 lines) and deliberately standard-library-only, so it rewards fast, direct agents; it says nothing about long-horizon work.
- Effort settings are not equalised (see the fleet table). Kimi is always-thinking; Qwen’s thinking tokens are reported separately.
- Kimi’s token figures come from its session log, not its stdout, and may exclude any background calls; Qwen’s stats block may exclude its background memory-extractor call (documented as a second API call per run).
- Prices are the registry values as of the run date and were re-verified on each vendor’s own pricing page (section 04); DeepSeek is priced at off-peak (runs fell outside its 01–04 / 06–10 UTC peak). One rate, Qwen3.8-Max, rests on a routing note plus secondary trackers rather than a row on the vendor’s own pricing table. Opus 5’s cache writes are priced at the $10 1-hour tier because that is the rate at which Claude Code’s own cost figure reproduces; the invocation requested no cache TTL, so the tier is inferred rather than observed.
- Wall-clock includes CLI start-up and, for Codex, skill/plugin ; API-side duration is recorded separately where the CLI reports it. Because all eight launched at once, wall-clock also includes whatever API-side queueing each vendor applied at that moment.
Reproducibility. A stranger can redo this exactly: the same task text, the same eight invocation strings, the same two verification steps (an independent unittest
run plus token counts lifted from the CLI output), and the same vendor pricing pages, dated to the day they redo it.
"This is a measurement of this task on this day, not a ranking of models."— Limitations section of this dataset’s methodology
03 — The datasetAll eighteen rows, every column. #
The table is the asset. Rows are ordered as the runs were recorded: the eight run1
rows, the two Gemini re-runs, then the eight run2
rows. “NOT REPORTED” means the CLI printed no such figure; it is not zero. “NO FILES IN CWD” means the agent replied DONE but nothing existed in its working directory for us to test. Token counts are as reported by each CLI; list price is calculated from them at the rates in section 04.
| Agent | Run | Wall s | Turns | Input (uncached) | Cache read | Cache write | Output | Thinking | Total tok | Files in cwd | Tests (our run) | List-price USD | CLI-reported USD | Plan / marginal cost |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| run1 — eight agents, launched concurrently | ||||||||||||||
| Anthropic · Claude Opus 5 (Claude Code) | run1 | 20.2 | 2 | 4 | 49,439 | 21,219 | 1,622 | 0 | 72,284 | dedupe.py, test_dedupe.py | PASS | 0.2775 | 0.2774795 | Claude Max subscription ($200/mo) — $0 marginal |
| OpenAI · GPT-5.6 Sol (Codex CLI) | run1 | 34.3 | NOT REPORTED | 25,834 | 22,016 | 0 | 1,273 | 70 | 49,123 | dedupe.py, test_dedupe.py | PASS | 0.1376 | NOT REPORTED | ChatGPT subscription (Codex) — $0 marginal |
| Google · Gemini 3.7 Flash (Antigravity CLI) | run1 | 64.5 | 1 | 50,061 | 146,474 | 0 | 6,135 | 3,563 | 202,670 | (none) | NO FILES IN CWD (status=ERROR; files never reached cwd) | 0.0715 | NOT REPORTED | Google Workspace / Antigravity plan — $0 marginal |
| xAI · Grok 4.6 (Grok Build) | run1 | 24.2 | 3 | 21,642 | 34,176 | 0 | 1,344 | 126 | 57,162 | dedupe.py, test_dedupe.py | PASS | 0.0684 | 0.01163412 | SuperGrok subscription — $0 marginal; the CLI prints its own total_cost_usd |
| Moonshot · Kimi K3 (Kimi Code) | run1 | 44.9 | NOT REPORTED | 7,252 | 126,976 | 0 | 2,046 | NOT REPORTED | 136,274 | dedupe.py, test_dedupe.py | PASS | 0.0905 | NOT REPORTED | Kimi Allegro subscription (monthly quota) — $0 marginal |
| Alibaba · Qwen3.8-Max (Qwen Code) | run1 | 107.1 | 4 | 38,887 | 110,223 | 0 | 5,298 | 3,768 | 154,408 | dedupe.py, test_dedupe.py | PASS | 0.1371 | NOT REPORTED | QwenCloud Token Plan (prepaid, 10% of standard pricing) — effectively $0 marginal until quota expiry |
| Z.ai · GLM-5.3 (Claude Code → Z.ai shim) | run1 | 39.5 | 5 | 38,677 | 203,392 | 0 | 1,451 | 0 | 243,520 | dedupe.py, test_dedupe.py | PASS | 0.1134 | 0.331356 | Z.ai Coding Plan subscription — $0 marginal |
| DeepSeek · V4 Flash (Claude Code → DeepSeek shim) | run1 | 24.9 | 4 | 61,389 | 129,024 | 0 | 3,657 | 0 | 194,070 | dedupe.py, test_dedupe.py | PASS | 0.0168 | 0.462882 | Pay-as-you-go API key — REAL marginal spend (the only one) |
| Gemini-only re-runs — isolating the --add-dir finding | ||||||||||||||
| Google · Gemini 3.7 Flash (Antigravity CLI) | run1-gemini-retry | 86.8 | 1 | 53,221 | 130,126 | 0 | 1,338 | 816 | 184,685 | (none) | NO FILES IN CWD (status=SUCCESS, “DONE”, but files landed in ~/.gemini/antigravity-cli/scratch/, not the cwd) | 0.0547 | NOT REPORTED | Google Workspace / Antigravity plan — $0 marginal |
| Google · Gemini 3.7 Flash (Antigravity CLI) | run1-gemini-retry2 | 71.1 | 1 | 43,980 | 154,555 | 0 | 5,419 | 3,271 | 203,954 | dedupe.py, test_dedupe.py | PASS (with --add-dir "$cwd") | 0.0649 | NOT REPORTED | Google Workspace / Antigravity plan — $0 marginal |
| run2 — eight agents, launched concurrently | ||||||||||||||
| Anthropic · Claude Opus 5 (Claude Code) | run2 | 26.4 | 3 | 6 | 86,608 | 21,462 | 1,839 | 59 | 109,915 | dedupe.py, test_dedupe.py | PASS | 0.3039 | 0.303929 | Claude Max subscription ($200/mo) — $0 marginal |
| OpenAI · GPT-5.6 Sol (Codex CLI) | run2 | 53.0 | NOT REPORTED | 24,193 | 98,816 | 0 | 1,853 | 227 | 124,862 | dedupe.py, test_dedupe.py | PASS | 0.1734 | NOT REPORTED | ChatGPT subscription (Codex) — $0 marginal |
| Google · Gemini 3.7 Flash (Antigravity CLI) | run2 | 58.9 | 1 | 32,668 | 65,113 | 0 | 1,474 | 1,231 | 99,255 | (none) | NO FILES IN CWD (no --add-dir; reproduced the global-scratch behaviour) | 0.0349 | NOT REPORTED | Google Workspace / Antigravity plan — $0 marginal |
| xAI · Grok 4.6 (Grok Build) | run2 | 37.1 | 6 | 9,450 | 108,928 | 0 | 2,403 | 346 | 120,781 | dedupe.py, test_dedupe.py | PASS | 0.0878 | 0.01492294 | SuperGrok subscription — $0 marginal; the CLI prints its own total_cost_usd |
| Moonshot · Kimi K3 (Kimi Code) | run2 | 26.7 | NOT REPORTED | 7,052 | 126,976 | 0 | 2,076 | NOT REPORTED | 136,104 | dedupe.py, test_dedupe.py | PASS | 0.0904 | NOT REPORTED | Kimi Allegro subscription (monthly quota) — $0 marginal |
| Alibaba · Qwen3.8-Max (Qwen Code) | run2 | 91.2 | 3 | 37,569 | 70,484 | 0 | 4,086 | 2,765 | 112,139 | dedupe.py, test_dedupe.py | PASS | 0.1173 | NOT REPORTED | QwenCloud Token Plan (prepaid, 10% of standard pricing) — effectively $0 marginal until quota expiry |
| Z.ai · GLM-5.3 (Claude Code → Z.ai shim) | run2 | 43.6 | 4 | 8,368 | 174,848 | 0 | 2,382 | 0 | 185,598 | dedupe.py, test_dedupe.py | PASS | 0.0677 | 0.188814 | Z.ai Coding Plan subscription — $0 marginal |
| DeepSeek · V4 Flash (Claude Code → DeepSeek shim) | run2 | 22.0 | 4 | 61,461 | 128,128 | 0 | 3,208 | 0 | 192,797 | dedupe.py, test_dedupe.py | PASS | 0.0165 | 0.451569 | Pay-as-you-go API key — REAL marginal spend (the only one) |
One chart, one column. List price per run is the single figure every reader can compare apples-to-apples across all eight agents: CLI-reported USD is NOT REPORTED for four of eight and wrong for two, and wall-clock is confounded by the concurrent launch. Run1 and run2 are shown side by side rather than averaged, so the run-to-run variance stays visible. The three Gemini runs that left no files in the working directory are drawn as dashed outlines rather than omitted: their cost was real even though the deliverable was not.
Reading across the rows rather than down the cost column: every agent that wrote files into its directory passed, on both runs. The fastest completion was Opus 5 at 20.2 s; the slowest was Qwen3.8-Max at 107.1 s. Opus 5 used almost no uncached input (4 and 6 tokens) but wrote 21K tokens to cache on each run, and at the $10 cache-write rate Table 2 uses, those writes were 76% and 71% of its two list-price figures. GLM-5.3 read the most from cache (203,392 tokens in run1). The two DeepSeek runs were within 2% of each other on price and within 3 s on wall-clock.
04 — Rate cardThe eight rates the list-price column uses. #
Seven of the eight rates below were read from the vendor’s own pricing or model page on the dataset’s as-of date; the eighth, Qwen3.8-Max, rests on a routing note plus secondary trackers and is marked as weaker in its own row. Where the page states a promotional or introductory period, the end date is recorded; where a rate depends on a threshold the runs did not cross, the tier that applied is named.
| Vendor / model | Input | Cache read | Cache write | Output | Source page | Note |
|---|---|---|---|---|---|---|
| Anthropic · Claude Opus 5 | 5.00 | 0.50 | 10.00 | 25.00 | ||
developers.openai.com model pageai.google.dev pricingdocs.x.ai modelsplatform.kimi.ai K3 pricingalibabacloud.com routing notedocs.z.ai pricingapi-docs.deepseek.com pricingThree of these eight rates are explicitly promotional or time-bounded on the vendor’s own page — Sol’s promotional rate is guaranteed only through November 21, 2026, Gemini 3.7 Flash’s introductory rate runs through December 31, 2026, and Z.ai’s free cache storage is “for a limited time”. Anyone re-pricing these tokens later should expect at least the Gemini row to move. Our LLM API pricing index carries the broader rate card; the eight rows above are only the ones this dataset needs.
05 — Finding 1 · ReportingSix CLIs, six different ideas of what usage means. #
The first finding is about the instruments, not the models. Getting a comparable token split out of eight headless runs meant reading five different output schemas, and a sixth CLI offered none at all. The NOT REPORTED cells in Table 1 are the direct consequence.
--output-format json
Also reports turns, API duration and a total_cost_usd. Used for three of the eight agents here — Opus 5 natively, GLM-5.3 and DeepSeek V4 Flash through each vendor’s Anthropic-compatible endpoint. The cost field is the subject of section 06.
--json
Token counts arrive on the turn-completion event of a JSON-Lines stream. No cost field and no turn count are emitted, which is why the Sol rows show NOT REPORTED in both columns. The cached count was subtracted from total input to produce the uncached figure.
--output-format json
Reports a status field alongside usage. Run1 returned status=ERROR after 60 s yet still replied DONE; the two retries returned SUCCESS. No cost field.
--output-format json
The only non-Anthropic CLI that prints a dollar figure. xAI’s cost-tracking documentation describes the underlying field as the billed amount, not a client-side estimate — which makes the gap in section 06 an open question rather than a bug.
-o json
A full stats block. A documented background memory-extractor call per run may not be included in these counts — listed as a limitation, not corrected for.
--output-format stream-json
Kimi printed no usage at all. Per-call usage (inputOther, output, inputCacheRead, inputCacheCreation) was recovered from the session log under ~/.kimi-code/sessions/ — six API calls per run — and may exclude background calls.
The practical consequence for anyone building a cost dashboard on top of headless agents: there is no common field to read. Token splits have to be normalised per CLI, “input” means total input in some schemas and uncached input in others, thinking tokens are sometimes a separate count and sometimes folded into output, and a cost figure — where one exists — is not necessarily the vendor’s. Our per-task, per-user agent cost framework assumes you can get a clean per-task token count; this table is what obtaining one actually looked like.
06 — Finding 2 · Cost fieldsOne cost field is 27× too high; another is 0.17 of the rate card. #
Two CLIs print a dollar figure. Neither should be read without knowing what produced it. The comparison that makes this visible is already in Table 1: the CLI-reported USD column next to the list-price column, row by row.
Opus 5 · Claude Code on Anthropic
Run1: CLI said $0.2774795, list price $0.2775. Run2: CLI said $0.303929, list price $0.3039. Claude Code’s total_cost_usd agrees with our calculation to the cent when the endpoint is Anthropic’s own — but only with cache writes priced at $10, and we chose that tier because it is the one that reproduces the CLI’s figure. The invocation carries no cache-TTL flag, and Anthropic’s 5-minute write rate is $6.25. Read the row as the same-vendor case agreeing with itself, not as an independent check.
DeepSeek V4 Flash · Claude Code → DeepSeek
Run1: CLI said $0.462882, DeepSeek off-peak list price $0.0168 (27.5×). Run2: $0.451569 vs $0.0165 (27.4×). GLM-5.3 through the Z.ai endpoint: $0.331356 vs $0.1134 (2.9×) and $0.188814 vs $0.0677 (2.8×). The CLI priced third-party tokens with Anthropic’s table.
Grok 4.6 · CLI-reported vs rate card
Run1: CLI said $0.01163412; tokens × published rates = $0.068436. Run2: $0.01492294 vs $0.087782. In both runs the CLI figure is exactly 0.17 of the rate-card computation (0.068436 × 0.17 = 0.01163412; 0.087782 × 0.17 = 0.01492294). We do not know why.
Claude Code through a third-party endpoint. Pointed at Z.ai or DeepSeek via the Anthropic-compatible base URL, Claude Code still prints a total_cost_usd
— computed with Anthropic’s price table, and it logs unrecognized_model
while doing so. This is consistent with how Anthropic documents the field. Claude Code’s headless-mode documentation says of total_cost_usd
and the per-model cost breakdown that “Both figures are client-side estimates and can differ from your actual bill.” The figure was never claimed to be a bill; on Anthropic’s own models it happens to be exact, and on someone else’s models it is Anthropic’s rate applied to someone else’s tokens. A closed issue on the Claude Code repository (#53371, filed April 25, 2026) separately reports the field returning roughly 10× too high for a same-vendor call — a different failure mode from ours, but corroboration that the field is documented as an estimate rather than a ledger. Anyone reading /cost
against a third-party endpoint is reading a number that can be off by an order of magnitude. The column stays in Table 1 for exactly that reason.
Grok’s field is a different kind of number. xAI’s cost-tracking documentation describes the underlying cost field as “the actual amount billed, after all applicable discounts (including prompt caching reductions) have been applied, and inclusive of all token costs and server-side tool invocation costs” — a server-computed billed figure, not a client-side estimate. That is what makes the 0.17 ratio worth reporting rather than dismissing. A promotional or subscription-linked discount, an undisclosed caching credit, or a rate-card page that lags the billing system would each produce a gap like this; none of them is documented on any page we read, and the consistency across two independent runs rules out random variance. The page states the discrepancy with its exact numbers and does not pick a cause.
total_cost_usd
across agents, classify each source as client-side estimate(Claude Code, by Anthropic’s own description) or
server-side billed figure(Grok, by xAI’s), and recompute from tokens × the vendor’s current rate card as the comparable baseline. Where the two disagree, that gap is itself a finding to record, not to average away.
07 — Findings 3 & 4 · Cache and cwdCache reads outweigh fresh input in 17 of 18 rows; the cheapest-looking agent wrote nowhere useful. #
Cache reads are most of the input in all but one row. Across the 18 rows, uncached input ran from 4 tokens (Opus 5, run1) to 61,461 (DeepSeek, run2), while cache reads ran from 22,016 (Sol, run1) to 203,392 (GLM-5.3, run1). Expressed as a share of all input tokens (uncached + cache read + cache write), cache reads were between 46.0% and 95.4% in every run, and above half in seventeen of the eighteen — the exception is GPT-5.6 Sol’s run1 at 46.0%. The bars below show run1 per agent; run2 is in the sub-label.
Cache reads as a share of all input tokens · run1, by agent
Source: Table 1 — cache-read tokens ÷ (uncached + cache-read + cache-write), run1; run2 share in the sub-labelThe pricing consequence follows directly. Re-pricing each row with every input token at the uncached rate — the mistake a naive “input tokens × input price” estimate makes — would have overstated the full-run list price by between 1.4× (Opus 5, run1) and 4.8× (both Kimi runs) across these 18 rows. Table 1 prices cache reads at the vendor’s published cache-read rate wherever one exists, which is all eight vendors here.
The cheapest-looking agent and the working directory. Gemini 3.7 Flash’s list-price rows are among the lowest on the page: $0.0715, $0.0547, $0.0649 and $0.0349. Three of those four runs produced nothing in the directory the agent was launched in. Run1, with the cwd not in the CLI’s trusted-workspace list, errored after 60 s and 202K tokens with a message that artifacts must live under ~/.gemini/antigravity-cli/brain/<conversation>/
— status ERROR, final response still “DONE”. The first retry, with the cwd pre-trusted, returned SUCCESS and “DONE” with zero files in the cwd: the files had been written to ~/.gemini/antigravity-cli/scratch/
, where they did pass the tests. Only the second retry, with --add-dir
pointing at the cwd, put the files where the task said and passed in 71.1 s. Run2, launched without --add-dir
, reproduced the global-scratch behaviour.
For a cost benchmark the consequence is that the cheapest-looking agent produced nothing usable, twice, on the invocation most people would write. Those rows stay in the table and in Figure 1, drawn as outlines: the tokens were consumed and would have been billed on a metered plan whether or not the deliverable arrived. A reader who wants a single Gemini figure for this task should take the --add-dir
run, $0.0649, and note that it took the third attempt to obtain.
08 — Finding 5 · Plans“Free” is the plan, not the model. #
Table 1 carries two cost columns on purpose. List price is what the same tokens would cost on the vendor’s metered API. The plan column is what was actually paid, and for seven of eight agents the answer was nothing at the margin.
Six agents, $0 marginal
Claude Max ($200/mo), ChatGPT (Codex), a Google Workspace / Antigravity plan, SuperGrok, Kimi Allegro (monthly quota) and the Z.ai Coding Plan. The tokens in Table 1 drew down quota or nothing at all; they did not generate a bill.
One agent, effectively $0 until expiry
Qwen3.8-Max ran on a QwenCloud Token Plan that Alibaba’s promotional note describes as billing as low as 10% of the standard rate. Prepaid and expiring, so the marginal cost of a run is the quota it consumes, not a charge.
One agent, real spend
DeepSeek V4 Flash on an API key, priced at off-peak: $0.0168 and $0.0165 per run, about $0.03 for both. This was the only money that left an account during the whole exercise.
Summed across all 18 rows, the list-price column comes to $1.92 for 2,579,601 tokens; the amount actually paid was about $0.03. That gap is the reason the page refuses to call any agent “cheap” or “expensive” without naming the plan. A team on flat subscriptions experiences cost as quota pressure and rate limits; a team on metered keys experiences it as a bill; the same token counts sit under both. The off-peak detail matters too — DeepSeek’s rate doubles inside its published peak bands, so the same two runs inside a peak window would have cost about $0.07 rather than $0.03. Our earlier post on DeepSeek and GLM off-peak pricing windows defines the windows; this dataset is a dated data point inside one.
09 — LimitsHow to read this without over-reading it. #
The limitations in the Methodology block are not boilerplate; each one rules out a conclusion a reader might be tempted to draw. One task, two runs, one time of day, one machine, concurrent launch. Wall-clock in particular is confounded: eight agents hitting eight vendors at the same second means each wall-clock figure includes whatever queueing that vendor applied at that moment, so a 20 s versus 107 s gap is a fact about this run, not a latency ranking. Effort was not equalised, and our effort benchmarks show that dial alone can move token counts by multiples.
What the dataset does support is narrower and, we think, more useful than a ranking. It shows the shape of a small headless coding task in 2026: a few thousand output tokens, uncached input running from single digits to tens of thousands, and cache reads that outweigh fresh input in seventeen of the eighteen rows. Because a cache read is billed at a fraction of the uncached rate, cache reads dominate the token count by more than they dominate the bill — which is why re-pricing them as fresh input inflates these rows by 1.4× to 4.8× rather than by an order of magnitude, and why that re-pricing is the estimating mistake worth catching first. It shows that the cost fields CLIs print are not interchangeable with each other or with a bill. And it shows that a DONE reply is not a deliverable: the one agent whose output we could not test said DONE every time.
Projecting forward, the part of this table most likely to change is not the token counts but the rate card under them. Two of eight rates are explicitly promotional with stated end dates and a third is free “for a limited time” with no end date given, one rests on a routing note rather than a pricing-table row, and one vendor doubles its rate by the clock. A refresh of this dataset will re-price the same tokens against whatever the pages say on that day, and the difference between the two list-price columns will be a measurement of rate-card drift rather than of the models. The harness findings are likelier to age the other way: a CLI that prints no usage, or prices third-party tokens with its own vendor’s table, tends to get fixed once enough people notice.
Two companion pieces frame where this sits. We defined the metric — cost per successful task, not cost per token — in our cost-per-successful-task post, with worked hypothetical comparisons; this page is the first real measurement behind it: on this task seven agents succeeded on every attempt, and the eighth needed a third attempt to deliver into the working directory. And every CLI here ran with its permission checks defeated — the census of coding-agent permission defaults records what each of these CLIs does when you do not. Teams deciding which of these agents to route which work to are the usual starting point of our AI transformation engagements, which begin with a measurement like this one on the team’s own tasks rather than ours.
10 — ConclusionThe instruments are the finding. #
The token counts were easy to read. The cost was not.
Seven of eight headless coding agents built the tool and passed the tests on both runs, at list prices from $0.0165 to $0.3039 per run. Every one of those figures is reproducible from the token counts in Table 1 and the rates in Table 2, and every token count is the CLI’s own report or, for Kimi, its own session log. That is the part of this page a citing writer can lean on.
The harder part is that the dollar figures the CLIs themselves print are not a substitute for that calculation. One is exact on its own vendor and 27× high on another’s; one is described by its vendor as the billed amount and comes to exactly 0.17 of the rate card, for reasons no page we read explains; four print no cost at all. Cache reads, not fresh input, are where most of the tokens sit in seventeen of the eighteen rows, and one agent’s cheap rows bought nothing in the working directory three times out of four.
None of that ranks a model. It describes what it took to measure one small task honestly across eight harnesses on one day, and it leaves the rows in place that make the measurement look less tidy. The dataset is maintained; the numbers will be re-read against the vendor pages when it is refreshed, and the as-of date will move with them.