# Catch token-cost regressions in CI before they ship

> Source: <https://dev.to/wartzarbee/catch-token-cost-regressions-in-ci-before-they-ship-35o3>
> Published: 2026-07-21 20:48:12+00:00

Last year a team shipped a "small prompt improvement" on a Friday. By Monday their Claude bill had jumped 40%. Nobody caught it in review — the diff looked fine. The tokens were invisible.

We wrote about the extreme end of this in [We burned 136 million tokens running an autonomous agent studio — here's how we cut the bill ~90%](https://dev.to/wartzarbee/we-burned-136-million-tokens-running-an-autonomous-agent-studio-heres-how-we-cut-the-bill-90-17gf). That was a runaway agent. But most cost regressions aren't dramatic — they're a prompt that grew by 200 tokens, a context window that stopped being trimmed, a new tool whose description is 800 words long. They compound quietly until the bill arrives.

We built the guardrail we wish we'd had.

** wartzar-bee/ci-guardrail** is a GitHub Action that:

`@wartzar-bee/tokenscope`

Here's what the PR comment looks like when a regression is caught:

```
## 🚨 wartzar-bee Cost Guardrail

| Metric          | Value      |
|-----------------|------------|
| Base tokens     | 12,400     |
| This PR tokens  | 16,200     |
| Delta           | **+30.6%** |
| Threshold       | 20%        |

### Top token consumers (HEAD)
| File                      | Tokens |
|---------------------------|--------|
| src/agent/prompts.ts      | 8,100  |
| src/agent/context.ts      | 4,200  |
| src/tools/search.ts       | 3,900  |

⛔ Build blocked — cost regression exceeds the 20% threshold.
Reduce prompt size, add caching, or raise the threshold if intentional.
```

The comment is **idempotent** — it updates in place on each push, no spam.

Add one file to your repo:

```
# .github/workflows/cost-guardrail.yml
name: Cost Guardrail
on: [pull_request]

jobs:
  cost-check:
    runs-on: ubuntu-latest
    permissions:
      pull-requests: write
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - uses: wartzar-bee/ci-guardrail@v1
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          threshold-pct: 20   # block if tokens grow >20% vs base
```

That's it. No API keys. No external service. `GITHUB_TOKEN`

is the only secret — it's already in every repo.

Not ready to block builds yet? Set `threshold-pct: 0`

:

```
- uses: wartzar-bee/ci-guardrail@v1
  with:
    github-token: ${{ secrets.GITHUB_TOKEN }}
    threshold-pct: 0   # always comment, never block
```

You get visibility without the gate. Add the gate when you're ready.

The action uses `tokenscope scan`

— a static analysis of your agent code (prompts, context configs, tool definitions). It doesn't run your agent. That means:

The tradeoff: it estimates *structural* token cost, not runtime cost (which depends on conversation history, tool outputs, etc.). For catching regressions from prompt edits and config changes, static analysis catches the majority of real-world cases. Live sandbox execution is on the roadmap.

The 136M-token burn postmortem identified three root causes that appear in almost every cost incident:

The CI guardrail addresses all three: it establishes a per-PR baseline, provides an optional gate, and shows exactly which files are responsible.

The action is available now. The repo is at ** wartzar-bee/ci-guardrail** — copy the workflow above and you're running in under two minutes.

If you hit an edge case or want to discuss the approach, open an issue or find us at [@wartzarbee](https://dev.to/wartzarbee).

*wartzar-bee builds cost-efficient autonomous agents. *

`@wartzar-bee/tokenscope`

is our open-source token analysis tool — ~90 installs/month and growing.