One pass of my board eval bills $9.14 on the Anthropic API. Through Claude Code it bills $0. Same model, claude-opus-4-8.
That is 27 calls, and it is not an estimate. The CLI prints a total_cost_usd
in its envelope: what the run would have cost on the API. It bills the subscription instead, so the number is a receipt for money nobody spent.
Running with `--output-format json`
and `--json-schema`
rides the same structured-output machinery the API does, an internal forced tool call. Format reliability on my suite went from 7 out of 15 to 15 out of 15.
The schema needs relaxing first: strip pattern
, minLength
, maxLength
, minItems
, maxItems
, format
and the $schema
meta-ref, because the CLI validator rejects draft-2020-12. The strict version stays in Zod on the caller side, so nothing is actually loosened, the validation just moves to where it can run.
If ANTHROPIC_API_KEY
sits in the child process environment, the CLI quietly bills the API account rather than the subscription. Nothing errors. Nothing warns. The invoice arrives. It gets stripped explicitly at spawn.
This is the failure mode I would look for first in anyone else's runner: the money leak is silent, and the only symptom is a bill at the end of the month for a run you believed was free.
This is a dev-loop tool. Anthropic's consumer terms prohibit automated access "except when you are accessing our Services via an Anthropic API Key or where we otherwise explicitly permit it", and the commercial terms governing API use do not cover consumer subscriptions. An eval runner on my own machine is the CLI used as designed. A shipped service is not.
Iterate on the CLI, ship on the API. What does one pass of your eval suite cost, and does anyone know it?
Originally published at dylan.merigaud.com.