Catch token-cost regressions in CI before they ship

Wartzar-bee released a GitHub Action called ci-guardrail that catches token-cost regressions in CI before they ship. The action performs static analysis of agent code to estimate structural token costs and can block pull requests if costs exceed a configurable threshold. It was inspired by a real incident where a team's Claude bill jumped 40% from a small prompt change.

Last year a team shipped a "small prompt improvement" on a Friday. By Monday their Claude bill had jumped 40%. Nobody caught it in review — the diff looked fine. The tokens were invisible. We wrote about the extreme end of this in We burned 136 million tokens running an autonomous agent studio — here's how we cut the bill ~90% https://dev.to/wartzarbee/we-burned-136-million-tokens-running-an-autonomous-agent-studio-heres-how-we-cut-the-bill-90-17gf . That was a runaway agent. But most cost regressions aren't dramatic — they're a prompt that grew by 200 tokens, a context window that stopped being trimmed, a new tool whose description is 800 words long. They compound quietly until the bill arrives. We built the guardrail we wish we'd had. wartzar-bee/ci-guardrail is a GitHub Action that: @wartzar-bee/tokenscope Here's what the PR comment looks like when a regression is caught: 🚨 wartzar-bee Cost Guardrail | Metric | Value | |-----------------|------------| | Base tokens | 12,400 | | This PR tokens | 16,200 | | Delta | +30.6% | | Threshold | 20% | Top token consumers HEAD | File | Tokens | |---------------------------|--------| | src/agent/prompts.ts | 8,100 | | src/agent/context.ts | 4,200 | | src/tools/search.ts | 3,900 | ⛔ Build blocked — cost regression exceeds the 20% threshold. Reduce prompt size, add caching, or raise the threshold if intentional. The comment is idempotent — it updates in place on each push, no spam. Add one file to your repo: .github/workflows/cost-guardrail.yml name: Cost Guardrail on: pull request jobs: cost-check: runs-on: ubuntu-latest permissions: pull-requests: write steps: - uses: actions/checkout@v4 with: fetch-depth: 0 - uses: wartzar-bee/ci-guardrail@v1 with: github-token: ${{ secrets.GITHUB TOKEN }} threshold-pct: 20 block if tokens grow 20% vs base That's it. No API keys. No external service. GITHUB TOKEN is the only secret — it's already in every repo. Not ready to block builds yet? Set threshold-pct: 0 : - uses: wartzar-bee/ci-guardrail@v1 with: github-token: ${{ secrets.GITHUB TOKEN }} threshold-pct: 0 always comment, never block You get visibility without the gate. Add the gate when you're ready. The action uses tokenscope scan — a static analysis of your agent code prompts, context configs, tool definitions . It doesn't run your agent. That means: The tradeoff: it estimates structural token cost, not runtime cost which depends on conversation history, tool outputs, etc. . For catching regressions from prompt edits and config changes, static analysis catches the majority of real-world cases. Live sandbox execution is on the roadmap. The 136M-token burn postmortem identified three root causes that appear in almost every cost incident: The CI guardrail addresses all three: it establishes a per-PR baseline, provides an optional gate, and shows exactly which files are responsible. The action is available now. The repo is at wartzar-bee/ci-guardrail — copy the workflow above and you're running in under two minutes. If you hit an edge case or want to discuss the approach, open an issue or find us at @wartzarbee https://dev.to/wartzarbee . wartzar-bee builds cost-efficient autonomous agents. @wartzar-bee/tokenscope is our open-source token analysis tool — ~90 installs/month and growing.