{"slug": "wattage-a-token-spend-profiler-and-cost-regression-gate-for-ai-agents", "title": "Wattage: A token-spend profiler and cost-regression gate for AI agents", "summary": "Wattage, an open-source token-spend profiler and cost-regression gate for AI agents, analyzes OpenTelemetry traces to identify token waste, price inefficiencies in real dollars, and prescribe fixes, including failing CI when changes increase costs. Its convergence engine detects non-convergent agent loops with 1.00 F1 score versus 0.25 for exact-match baselines, and a prefix_churn fix simulation shows 44.7% cost reduction on a real trace. The tool is fully offline, requires no config or API key, and outputs terminal reports or HTML flame graphs.", "body_md": "**A Kill-A-Watt meter for your AI agents.** Point it at a trace and it tells\nyou exactly where your tokens are being burned and wasted, prices each waste\npattern in real dollars, prescribes a fix, and can fail your CI when a change\nmakes your agent measurably more expensive.\n\n*A real captured agent trace (see provenance) —\nWattage catches a stable prompt prefix being re-sent instead of cached, prices\nthe waste, and prescribes the fix. Regenerate this GIF with\nvhs docs/assets/demo.tape (see the tape file for the exact command).*\n\n```\nuvx wattage report trace.json\n```\n\nNo config file, no API key, fully offline — point it at an [OTLP JSON](https://opentelemetry.io/docs/specs/otlp/)\ntrace export and it prices every call and runs every detector. Don't have a\ntrace yet? [Getting your first trace](/faizannraza/wattage/blob/main/docs/getting-started.md) covers both\n\"I already have OTel traces\" and \"I have zero instrumentation\" (a runnable,\n5-minute path from nothing to a real, priced report). Or try it right now\nagainst the fixture shipped in this repo:\n\n```\ngit clone https://github.com/faizannraza/wattage\ncd wattage && uv sync\nuv run wattage report examples/sample_trace.json\n╭──── ⚡ wattage — examples/sample_trace.json ────╮\n│ Token Efficiency: A (100)   Total cost: $0.0602 │\n│ quality: unmeasured                             │\n╰─────────────────────────────────────────────────╯\n      Token breakdown\n┏━━━━━━━━━━━━━━━━┳━━━━━━━━┓\n┃ Category       ┃ Tokens ┃\n┡━━━━━━━━━━━━━━━━╇━━━━━━━━┩\n│ input          │  18450 │\n│ output         │    320 │\n│ cache_read     │      0 │\n│ cache_creation │      0 │\n│ reasoning      │      0 │\n└────────────────┴────────┘\nNo findings — this trace looks efficient.\npricing: 2026-07-18-verified\n```\n\nOr get a self-contained, shareable HTML flame graph instead of the terminal view:\n\n```\nuv run wattage report examples/sample_trace.json --html report.html\n```\n\nWattage's standout feature is the **convergence engine** — the\n`nonconvergence`\n\ndetector, which catches an agent thrashing through a loop\nwithout making real progress, including patterns a naive exact-match\nduplicate detector structurally cannot see (a retry with a fresh timestamp\neach time, an oscillation between two strategies, a \"productive-looking\"\nstall where every call is technically unique but nothing is actually\nlearned).\n\nRather than assert that, we built a hand-reviewed set of 10 labeled synthetic loops and benchmarked Wattage's classifier against a real SHA-256 exact-match baseline implementation:\n\n| Classifier | Precision | Recall | F1 |\n|---|---|---|---|\nWattage |\n1.00 | 1.00 | 1.00 |\n| SHA-256 exact-match | 1.00 | 0.14 | 0.25 |\n\nReproduce it yourself — no cherry-picking, no hidden setup:\n\n```\nuv run python -m benchmarks.harness\n```\n\nAnd on a genuine captured agent trace (not synthetic — see\n[ benchmarks/traces/README.md](/faizannraza/wattage/blob/main/benchmarks/traces/README.md) for provenance),\nWattage's\n\n`prefix_churn`\n\nfix simulation shows a **44.7% cost reduction**(\n\n`$0.000199 → $0.000110`\n\n) from enabling prompt caching on the stable prefix —\nsmall dollar figures because it's a 3-turn demo trace, but the mechanism is\nidentical at production scale. Run it against your own traces for numbers\nthat matter:\n\n``` python\nuv run python -c \"from benchmarks.frontier import build_frontier; print(build_frontier())\"\n```\n\nFull methodology: [The Convergence Engine](/faizannraza/wattage/blob/main/docs/convergence.md).\n\n```\nuv run wattage badge trace.json --out wattage-badge.svg\n![Wattage](wattage-badge.svg)\n```\n\nWire `--badge-out`\n\ninto your CI job (see below) so it regenerates on every\nmerge to your default branch, and the badge in your README stays live.\n\nThree surfaces, one normalized data model underneath\n(`sessions → tasks → loops → iterations → calls`\n\n), built from\n[OpenTelemetry GenAI semantic-convention](https://opentelemetry.io/docs/specs/semconv/gen-ai/)\ntraces:\n\n-\n— ingests a trace, prices every call against a vendored, dated pricing snapshot, and runs eight detectors:`wattage report`\n\nDetector Catches `prefix_churn`\n\nStable context re-sent instead of cached `cache_gap`\n\nCaching attempted but under-redeemed by later reads `verbosity`\n\nOutput far beyond what the step needed `redundant_tool_calls`\n\nThe same tool call repeated (exact or fuzzy) `nonconvergence`\n\nLoops that thrash, oscillate, or stall without progress `retrieval_thrash`\n\nRepeated retrieval that never yields relevant results `model_mismatch`\n\nA pricier model doing work a cheaper one could handle `reasoning_overspend`\n\nHeavy reasoning-token spend on a simple step Every finding is priced in real dollars, includes a concrete fix, and is tagged with a\n\n`quality_risk`\n\ntier (`none`\n\n/`low`\n\n/`review`\n\n) — a fix that could plausibly change output quality (a model downgrade, less reasoning) only counts toward your score once a`--quality`\n\nmap backs it with real evidence. Full detail:[Detectors](/faizannraza/wattage/blob/main/docs/detectors/index.md). -\n— a single 0–100 Token Efficiency grade for a README badge or a CI gate.`wattage score`\n\n/`wattage badge`\n\n-\n— the cost-regression gate (below).`wattage ci`\n\nWattage never fabricates a number: an unpriced model leaves that call's cost\nat zero (and fails `wattage ci`\n\nloudly, exit code 4) rather than guessing;\nan unmeasured quality signal is reported as `unmeasured`\n\n, not assumed fine.\n\n```\n# .github/workflows/wattage.yml\nname: Wattage\non:\n  pull_request:\n    paths: [\"agents/**\", \"prompts/**\", \"src/**\"]\nconcurrency:\n  group: wattage-${{ github.ref }}\n  cancel-in-progress: true\njobs:\n  token-efficiency:\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions/checkout@v4\n      - name: Generate trace fixture\n        run: python scripts/run_agent_fixture.py > trace.json\n      - name: Wattage cost-regression gate\n        uses: faizannraza/wattage/action@v0.1.0\n        with:\n          source: trace.json\n          baseline: .wattage/baseline.json\n          fail-on: \"score_below:80,cost_delta_pct_above:5,any_critical:true\"\n          pr-comment: \"true\"\n```\n\nFails the build (exit code 1) when your agent regresses past the threshold you set, posts a per-detector delta table as a PR comment, and emits SARIF (shows up in GitHub's Security tab) and JUnit XML for any other CI system. The baseline is a small committed JSON file — noise-floor protection is structural, not statistical: it only ever updates on a run that actually passed the gate.\n\n**This is only half the setup.** A PR job runs on a throwaway checkout, so\nit can't be the thing that updates `.wattage/baseline.json`\n\non disk — that\nupdate needs a second workflow, triggered on push to your default branch,\nthat commits the refreshed baseline (and badge) back after each merge.\nSkipping it means every PR compares against the same stale baseline\nforever. Full reference, with both workflows: [CI Integration](/faizannraza/wattage/blob/main/docs/ci.md).\n\nDetectors are discovered through a Python entry-point group, so adding one\ndoesn't require touching this repo's core pipeline — see\n[CONTRIBUTING.md](/faizannraza/wattage/blob/main/CONTRIBUTING.md) for the full \"write a detector\" walkthrough,\nusing [ cache_gap](/faizannraza/wattage/blob/main/src/wattage/detectors/cache_gap.py) as the reference\nexample.", "url": "https://wpnews.pro/news/wattage-a-token-spend-profiler-and-cost-regression-gate-for-ai-agents", "canonical_source": "https://github.com/faizannraza/wattage", "published_at": "2026-07-26 23:27:35+00:00", "updated_at": "2026-07-26 23:52:27.242484+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "mlops"], "entities": ["Wattage", "OpenTelemetry", "SHA-256"], "alternates": {"html": "https://wpnews.pro/news/wattage-a-token-spend-profiler-and-cost-regression-gate-for-ai-agents", "markdown": "https://wpnews.pro/news/wattage-a-token-spend-profiler-and-cost-regression-gate-for-ai-agents.md", "text": "https://wpnews.pro/news/wattage-a-token-spend-profiler-and-cost-regression-gate-for-ai-agents.txt", "jsonld": "https://wpnews.pro/news/wattage-a-token-spend-profiler-and-cost-regression-gate-for-ai-agents.jsonld"}}