{"slug": "one-pass-of-my-eval-bills-9-14-on-the-api-and-0-through-the-cli", "title": "One pass of my eval bills $9.14 on the API and $0 through the CLI", "summary": "A developer found that running their evaluation suite through Anthropic's Claude Code CLI costs $0 per run, while the same 27 calls via the API would bill $9.14. The CLI prints a total_cost_usd receipt for money not spent, and supports structured output with JSON schema, improving format reliability from 7/15 to 15/15. The developer warns that if an API key is present in the environment, the CLI silently bills the API account instead of the subscription, creating a potential money leak.", "body_md": "*One pass of my board eval bills $9.14 on the Anthropic API. Through Claude Code it bills $0. Same model, claude-opus-4-8.*\n\nThat is 27 calls, and it is not an estimate. The CLI prints a `total_cost_usd`\n\nin its envelope: what the run would have cost on the API. It bills the subscription instead, so the number is a receipt for money nobody spent.\n\nRunning with `--output-format json`\n\nand `--json-schema`\n\nrides the same structured-output machinery the API does, an internal forced tool call. Format reliability on my suite went from 7 out of 15 to 15 out of 15.\n\nThe schema needs relaxing first: strip `pattern`\n\n, `minLength`\n\n, `maxLength`\n\n, `minItems`\n\n, `maxItems`\n\n, `format`\n\nand the `$schema`\n\nmeta-ref, because the CLI validator rejects draft-2020-12. The strict version stays in Zod on the caller side, so nothing is actually loosened, the validation just moves to where it can run.\n\nIf `ANTHROPIC_API_KEY`\n\nsits in the child process environment, the CLI quietly bills the API account rather than the subscription. Nothing errors. Nothing warns. The invoice arrives. It gets stripped explicitly at spawn.\n\nThis is the failure mode I would look for first in anyone else's runner: the money leak is silent, and the only symptom is a bill at the end of the month for a run you believed was free.\n\nThis is a dev-loop tool. Anthropic's consumer terms prohibit automated access \"except when you are accessing our Services via an Anthropic API Key or where we otherwise explicitly permit it\", and the commercial terms governing API use do not cover consumer subscriptions. An eval runner on my own machine is the CLI used as designed. A shipped service is not.\n\nIterate on the CLI, ship on the API. What does one pass of your eval suite cost, and does anyone know it?\n\n*Originally published at dylan.merigaud.com.*", "url": "https://wpnews.pro/news/one-pass-of-my-eval-bills-9-14-on-the-api-and-0-through-the-cli", "canonical_source": "https://dev.to/dylanmerigaud/one-pass-of-my-eval-bills-914-on-the-api-and-0-through-the-cli-3eof", "published_at": "2026-08-12 18:52:47+00:00", "updated_at": "2026-08-12 19:17:40.905864+00:00", "lang": "en", "topics": ["developer-tools", "large-language-models", "ai-products", "ai-tools"], "entities": ["Anthropic", "Claude Code", "claude-opus-4-8", "Zod"], "alternates": {"html": "https://wpnews.pro/news/one-pass-of-my-eval-bills-9-14-on-the-api-and-0-through-the-cli", "markdown": "https://wpnews.pro/news/one-pass-of-my-eval-bills-9-14-on-the-api-and-0-through-the-cli.md", "text": "https://wpnews.pro/news/one-pass-of-my-eval-bills-9-14-on-the-api-and-0-through-the-cli.txt", "jsonld": "https://wpnews.pro/news/one-pass-of-my-eval-bills-9-14-on-the-api-and-0-through-the-cli.jsonld"}}