{"slug": "catch-token-cost-regressions-in-ci-before-they-ship", "title": "Catch token-cost regressions in CI before they ship", "summary": "Wartzar-bee released a GitHub Action called ci-guardrail that catches token-cost regressions in CI before they ship. The action performs static analysis of agent code to estimate structural token costs and can block pull requests if costs exceed a configurable threshold. It was inspired by a real incident where a team's Claude bill jumped 40% from a small prompt change.", "body_md": "Last year a team shipped a \"small prompt improvement\" on a Friday. By Monday their Claude bill had jumped 40%. Nobody caught it in review — the diff looked fine. The tokens were invisible.\n\nWe wrote about the extreme end of this in [We burned 136 million tokens running an autonomous agent studio — here's how we cut the bill ~90%](https://dev.to/wartzarbee/we-burned-136-million-tokens-running-an-autonomous-agent-studio-heres-how-we-cut-the-bill-90-17gf). That was a runaway agent. But most cost regressions aren't dramatic — they're a prompt that grew by 200 tokens, a context window that stopped being trimmed, a new tool whose description is 800 words long. They compound quietly until the bill arrives.\n\nWe built the guardrail we wish we'd had.\n\n** wartzar-bee/ci-guardrail** is a GitHub Action that:\n\n`@wartzar-bee/tokenscope`\n\nHere's what the PR comment looks like when a regression is caught:\n\n```\n## 🚨 wartzar-bee Cost Guardrail\n\n| Metric          | Value      |\n|-----------------|------------|\n| Base tokens     | 12,400     |\n| This PR tokens  | 16,200     |\n| Delta           | **+30.6%** |\n| Threshold       | 20%        |\n\n### Top token consumers (HEAD)\n| File                      | Tokens |\n|---------------------------|--------|\n| src/agent/prompts.ts      | 8,100  |\n| src/agent/context.ts      | 4,200  |\n| src/tools/search.ts       | 3,900  |\n\n⛔ Build blocked — cost regression exceeds the 20% threshold.\nReduce prompt size, add caching, or raise the threshold if intentional.\n```\n\nThe comment is **idempotent** — it updates in place on each push, no spam.\n\nAdd one file to your repo:\n\n```\n# .github/workflows/cost-guardrail.yml\nname: Cost Guardrail\non: [pull_request]\n\njobs:\n  cost-check:\n    runs-on: ubuntu-latest\n    permissions:\n      pull-requests: write\n    steps:\n      - uses: actions/checkout@v4\n        with:\n          fetch-depth: 0\n\n      - uses: wartzar-bee/ci-guardrail@v1\n        with:\n          github-token: ${{ secrets.GITHUB_TOKEN }}\n          threshold-pct: 20   # block if tokens grow >20% vs base\n```\n\nThat's it. No API keys. No external service. `GITHUB_TOKEN`\n\nis the only secret — it's already in every repo.\n\nNot ready to block builds yet? Set `threshold-pct: 0`\n\n:\n\n```\n- uses: wartzar-bee/ci-guardrail@v1\n  with:\n    github-token: ${{ secrets.GITHUB_TOKEN }}\n    threshold-pct: 0   # always comment, never block\n```\n\nYou get visibility without the gate. Add the gate when you're ready.\n\nThe action uses `tokenscope scan`\n\n— a static analysis of your agent code (prompts, context configs, tool definitions). It doesn't run your agent. That means:\n\nThe tradeoff: it estimates *structural* token cost, not runtime cost (which depends on conversation history, tool outputs, etc.). For catching regressions from prompt edits and config changes, static analysis catches the majority of real-world cases. Live sandbox execution is on the roadmap.\n\nThe 136M-token burn postmortem identified three root causes that appear in almost every cost incident:\n\nThe CI guardrail addresses all three: it establishes a per-PR baseline, provides an optional gate, and shows exactly which files are responsible.\n\nThe action is available now. The repo is at ** wartzar-bee/ci-guardrail** — copy the workflow above and you're running in under two minutes.\n\nIf you hit an edge case or want to discuss the approach, open an issue or find us at [@wartzarbee](https://dev.to/wartzarbee).\n\n*wartzar-bee builds cost-efficient autonomous agents. *\n\n`@wartzar-bee/tokenscope`\n\nis our open-source token analysis tool — ~90 installs/month and growing.", "url": "https://wpnews.pro/news/catch-token-cost-regressions-in-ci-before-they-ship", "canonical_source": "https://dev.to/wartzarbee/catch-token-cost-regressions-in-ci-before-they-ship-35o3", "published_at": "2026-07-21 20:48:12+00:00", "updated_at": "2026-07-21 21:31:45.007072+00:00", "lang": "en", "topics": ["developer-tools", "ai-infrastructure", "ai-agents", "large-language-models"], "entities": ["wartzar-bee", "ci-guardrail", "tokenscope", "GitHub", "Claude"], "alternates": {"html": "https://wpnews.pro/news/catch-token-cost-regressions-in-ci-before-they-ship", "markdown": "https://wpnews.pro/news/catch-token-cost-regressions-in-ci-before-they-ship.md", "text": "https://wpnews.pro/news/catch-token-cost-regressions-in-ci-before-they-ship.txt", "jsonld": "https://wpnews.pro/news/catch-token-cost-regressions-in-ci-before-they-ship.jsonld"}}