{"slug": "track-ai-token-spend-in-grafana-claude-codex-and-ollama", "title": "Track AI Token Spend in Grafana: Claude, Codex, and Ollama", "summary": "A developer built a monitoring setup to track AI token spend and usage across Claude Code, Codex, and Ollama in a homelab, integrating metrics into Prometheus and Grafana. The system handles three different billing models—subscription quotas for the CLIs and per-token API costs for an automated agent—using OpenTelemetry, a custom Python exporter, and Pushgateway. The developer also documented PromQL pitfalls and privacy scrubbing for publishable dashboards.", "body_md": "I let three different AIs work in my homelab every day — a coding assistant, a second CLI for pair-work, and a small agent that triages alerts overnight. One evening I realized I couldn't answer a basic question: **what is all of this actually costing me?** Two burn subscription quota I've already paid for, one spends real API dollars, and none showed up on the Grafana dashboards I'd built for everything else in the rack.\n\nSo I fixed it. Every call from every AI in the lab — tokens, latency, cache hits, and dollars — now lands in Prometheus and one Grafana dashboard. This walks through the four measurement legs, the PromQL traps that made my first dashboard lie, and the privacy scrub that makes the screenshots publishable.\n\n**Make the addresses your own.** Every machine-specific value here is a placeholder: the monitoring host `10.0.0.5`, agent host `10.0.0.7`, Ollama nodes `10.0.0.1`–` 10.0.0.3`, exporter ports, and any `/home/youradmin` paths.\n\nThe AI layer has a genuinely weird cost structure. Two interactive CLIs run on flat subscriptions, so their \"cost\" is quota — a percentage of a weekly allowance. The automated agent calls a hosted API and pays per token. Same lab, three billing models. So the dashboard has two columns: **quota burn** (a percentage that resets) and **real dollars** (the metered agent).\n\nThe coding CLI needs no wrapper — it has [native OpenTelemetry support](https://code.claude.com/docs/en/monitoring-usage). Switch it on with env vars:\n\n```\nexport CLAUDE_CODE_ENABLE_TELEMETRY=1\nexport OTEL_METRICS_EXPORTER=otlp\nexport OTEL_EXPORTER_OTLP_PROTOCOL=grpc\nexport OTEL_EXPORTER_OTLP_ENDPOINT=http://10.0.0.5:4317\n```\n\nEvery session exports `claude_code.token.usage` (by type — input, output, cache read), `claude_code.cost.usage` in USD, and `claude_code.session.count`. Those arrive over OTLP, which Prometheus doesn't scrape directly — so an [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) on the monitoring host listens on `:4317`, applies processors, and re-exposes everything in Prometheus format on `:8889`.\n\nThe automated agent doesn't speak OTel, but it writes an honest log line per call: provider, model, token counts, latency, cache hit, and the occasional \"fallback activated\". A ~200-line stdlib-only Python exporter tails the log with a regex, keeps counters in memory, and serves them on `:9109`. It also reads the agent's task database (read-only) for queue-depth gauges — so the dashboard shows quality alongside spend: is the agent finishing work, or just burning tokens?\n\nMy three-node Ollama cluster was stubborn: Ollama ships no Prometheus endpoint (as of the 0.30 releases). But it leaks what you need in two places — its journald request log (status, latency, caller IP) and the documented `/api/ps` endpoint (which models are loaded right now). One exporter per node on `:9110`. The caller-IP label turned out useful: it shows *who* is using the local models.\n\nThe second CLI ([Codex](https://github.com/openai/codex)) records tokens and quota in session files, but the laptop sleeps and moves, so Prometheus can't reliably scrape it. That's exactly the [Pushgateway](https://prometheus.io/docs/practices/pushing/) case: a systemd *user* timer parses the session files every 5 minutes and pushes lifetime totals plus quota percentage to `:9091`.\n\nWiring it together is four scrape jobs:\n\n```\nscrape_configs:\n  - job_name: ai_claude_code\n    static_configs: [{ targets: [\"10.0.0.5:8889\"] }]   # OTel Collector\n  - job_name: ai_codex\n    honor_labels: true\n    static_configs: [{ targets: [\"10.0.0.5:9091\"] }]   # Pushgateway\n  - job_name: ai_agent\n    static_configs: [{ targets: [\"10.0.0.7:9109\"] }]   # log-tail exporter\n  - job_name: ai_ollama\n    static_configs: [{ targets: [\"10.0.0.1:9110\", \"10.0.0.2:9110\", \"10.0.0.3:9110\"] }]\n```\n\n**Trap 1: per-session counters break `increase()`.** Claude Code's counters are per-session and ephemeral; a short session leaves one sample, and range functions need two, so `increase()` returns nothing while a real 30k-token session sits invisible. Read the last value each session reported and sum:\n\n```\nsum(max_over_time(claude_code_token_usage_tokens_total[1d]))\n```\n\n**Trap 2: composite cost math collapses on absent series.** In PromQL, arithmetic with an empty operand makes the *whole expression* empty — so before any cache reads existed, real spend rendered as $0.00. Guard every component with `or vector(0)`:\n\n```\n(sum(rate(input_tokens[1h]))  or vector(0)) * 1.00 / 1e6\n+ (sum(rate(cache_tokens[1h])) or vector(0)) * 0.10 / 1e6\n+ (sum(rate(output_tokens[1h])) or vector(0)) * 5.00 / 1e6\n```\n\n**Trap 3: `histogram_quantile` returns literal `NaN` over idle windows.** Documented behavior with zero observations — my latency panel drew garbage across every quiet hour. Consumers need to drop non-finite samples; Grafana panels just go sparse when the lab is idle, which is the honest picture.\n\nClaude Code's telemetry attaches identity by default — your **email**, account ids, org id — as labels on every metric. Useful in a company; radioactive on a public screenshot. Going forward, the OTel Collector deletes those before they reach Prometheus:\n\n```\nprocessors:\n  attributes/scrub:\n    actions:\n      - { key: user.email,        action: delete }\n      - { key: user.account_uuid, action: delete }\n      - { key: user.account_id,   action: delete }\n      - { key: user.id,           action: delete }\n      - { key: organization.id,   action: delete }\n```\n\nDelete by key, not value, so anyone who ever exported from that laptop gets scrubbed. For history already on disk, open a temporary Prometheus admin window (`--web.enable-admin-api`), delete the identity-labeled series, then close it:\n\n```\ncurl -X POST 'http://10.0.0.5:9090/api/v1/admin/tsdb/delete_series?match[]={user_email!=\"\"}'\ncurl -X POST 'http://10.0.0.5:9090/api/v1/admin/tsdb/clean_tombstones'\n# now remove --web.enable-admin-api from the unit and restart again\n```\n\nOne deliberate non-deletion: I kept `session_id`. Dropping it merges per-session cumulative counters into one series (last write wins, totals undercount). Scrub identity; keep cardinality that's structurally load-bearing.\n\nMy Node Down alert was `up == 0`. A sleeping laptop is not an outage, so it's now scoped to exclude the AI jobs — a dead AI exporter shows as a dashboard gap, real infra still pages:\n\n```\nup{job!~\"ai_.*\"} == 0\n```\n\nNot spend — the **cache-hit rate**. At ~78% cached input, the agent's metered bill stays in coffee money. The day that rate drops is the day something changed in how it builds prompts, and now I'll see it the same morning. Total cost of the measurement layer: two tiny Python exporters, one collector, one gateway, and an evening.\n\n*Originally published on [peira.dev](https://peira.dev/blog/ai-spend-telemetry-grafana/).*", "url": "https://wpnews.pro/news/track-ai-token-spend-in-grafana-claude-codex-and-ollama", "canonical_source": "https://dev.to/josh_hall_b54941047f33661/track-ai-token-spend-in-grafana-claude-codex-and-ollama-5alm", "published_at": "2026-09-08 20:35:30+00:00", "updated_at": "2026-09-08 20:51:52.303697+00:00", "lang": "en", "topics": ["developer-tools", "ai-tools", "mlops", "ai-infrastructure"], "entities": ["Claude Code", "Codex", "Ollama", "Prometheus", "Grafana", "OpenTelemetry", "Pushgateway"], "alternates": {"html": "https://wpnews.pro/news/track-ai-token-spend-in-grafana-claude-codex-and-ollama", "markdown": "https://wpnews.pro/news/track-ai-token-spend-in-grafana-claude-codex-and-ollama.md", "text": "https://wpnews.pro/news/track-ai-token-spend-in-grafana-claude-codex-and-ollama.txt", "jsonld": "https://wpnews.pro/news/track-ai-token-spend-in-grafana-claude-codex-and-ollama.jsonld"}}