Last year a team shipped a "small prompt improvement" on a Friday. By Monday their Claude bill had jumped 40%. Nobody caught it in review — the diff looked fine. The tokens were invisible.
We wrote about the extreme end of this in We burned 136 million tokens running an autonomous agent studio — here's how we cut the bill ~90%. That was a runaway agent. But most cost regressions aren't dramatic — they're a prompt that grew by 200 tokens, a context window that stopped being trimmed, a new tool whose description is 800 words long. They compound quietly until the bill arrives.
We built the guardrail we wish we'd had.
** wartzar-bee/ci-guardrail** is a GitHub Action that:
@wartzar-bee/tokenscope
Here's what the PR comment looks like when a regression is caught:
## 🚨 wartzar-bee Cost Guardrail
| Metric | Value |
|-----------------|------------|
| Base tokens | 12,400 |
| This PR tokens | 16,200 |
| Delta | **+30.6%** |
| Threshold | 20% |
### Top token consumers (HEAD)
| File | Tokens |
|---------------------------|--------|
| src/agent/prompts.ts | 8,100 |
| src/agent/context.ts | 4,200 |
| src/tools/search.ts | 3,900 |
⛔ Build blocked — cost regression exceeds the 20% threshold.
Reduce prompt size, add caching, or raise the threshold if intentional.
The comment is idempotent — it updates in place on each push, no spam.
Add one file to your repo:
name: Cost Guardrail
on: [pull_request]
jobs:
cost-check:
runs-on: ubuntu-latest
permissions:
pull-requests: write
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: wartzar-bee/ci-guardrail@v1
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
threshold-pct: 20 # block if tokens grow >20% vs base
That's it. No API keys. No external service. GITHUB_TOKEN
is the only secret — it's already in every repo.
Not ready to block builds yet? Set threshold-pct: 0
:
- uses: wartzar-bee/ci-guardrail@v1
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
threshold-pct: 0 # always comment, never block
You get visibility without the gate. Add the gate when you're ready.
The action uses tokenscope scan
— a static analysis of your agent code (prompts, context configs, tool definitions). It doesn't run your agent. That means:
The tradeoff: it estimates structural token cost, not runtime cost (which depends on conversation history, tool outputs, etc.). For catching regressions from prompt edits and config changes, static analysis catches the majority of real-world cases. Live sandbox execution is on the roadmap.
The 136M-token burn postmortem identified three root causes that appear in almost every cost incident:
The CI guardrail addresses all three: it establishes a per-PR baseline, provides an optional gate, and shows exactly which files are responsible.
The action is available now. The repo is at ** wartzar-bee/ci-guardrail** — copy the workflow above and you're running in under two minutes.
If you hit an edge case or want to discuss the approach, open an issue or find us at @wartzarbee.
*wartzar-bee builds cost-efficient autonomous agents. *
@wartzar-bee/tokenscope
is our open-source token analysis tool — ~90 installs/month and growing.