Tokenomics: View token usage, costs, and rate-limit metrics

Tokenomics Viewer, a local-first tool for scanning Codex and Claude Code session logs to estimate token usage and costs, has been released. The tool reads files from the user's machine without uploading data and supports output as text, JSON, SQLite, or a local web dashboard. It requires Node.js 26 or newer and optionally integrates with ClickHouse for data storage.

Tokenomics Viewer scans local Codex and Claude Code session logs, normalizes token usage, estimates costs from static pricing tables, and reports the results as text, JSON, SQLite-backed data, or a local web dashboard. The tool is local-first. It reads files from your machine and does not upload logs or reports anywhere. On macOS or Linux, install or update Tokenomics Viewer and start the guided setup with: /bin/sh -c "$ curl -fsSL https://raw.githubusercontent.com/skuznetsov/tokenomics-viewer/main/install.sh " The installer does not use sudo . It installs versioned application files in ~/.local/share/tokenomics-viewer , creates tokenomics , tokenomics-viewer , and tokenomics-launch commands in ~/.local/bin , and then starts the guided launcher. If Node.js 26 or newer is unavailable, it installs a private Node.js 26 runtime after verifying the archive against the official Node.js SHA-256 manifest. The launcher asks whether to install and use ClickHouse, remembers the answer, runs the initial sync, starts the dashboard, and opens it in the default browser. SQLite data is kept across application updates at: ~/.local/share/tokenomics-viewer/tokenomics.sqlite Run the same one-line command again to update. To install without starting the dashboard, use: TOKENOMICS NO LAUNCH=1 /bin/sh -c "$ curl -fsSL https://raw.githubusercontent.com/skuznetsov/tokenomics-viewer/main/install.sh " If ~/.local/bin is not already on PATH , the installer prints the exact export command to add it. It does not modify shell startup files. - Node.js 26 or newer - No npm dependencies node:sqlite is used for the SQLite-backed sync and web dashboard modes. ClickHouse mode uses the ClickHouse HTTP interface and still needs no npm dependencies. clickhousectl https://clickhouse.com/blog/getting-started-clickhousectl is the official ClickHouse CLI for installing versions and managing isolated local servers. The installer also creates the shorter chctl command used below.Install it on macOS or Linux: curl -fsSL https://clickhouse.com/cli | sh The binary is installed under ~/.local/bin . If your shell cannot find it, add that directory to PATH and restart the shell: echo 'export PATH="$HOME/.local/bin:$PATH"' ~/.zshrc source ~/.zshrc clickhousectl --version chctl --version Download the current stable ClickHouse release and make it the local default: chctl local use stable chctl local which From the Tokenomics Viewer repository, start a persistent named server on the HTTP port expected by the application: chctl local server start --name tokenomics --http-port 8123 --tcp-port 9000 chctl local server list curl http://127.0.0.1:8123/ping The server runs in the background and stores its data under .clickhouse/servers/tokenomics/ . Connect with the bundled ClickHouse client: chctl local client --name tokenomics --query "SELECT version " Stop, restart, or permanently remove the local instance with: chctl local server stop tokenomics chctl local server start --name tokenomics chctl local server stop tokenomics chctl local server remove tokenomics If port 8123 is already occupied, omit --http-port and inspect the assigned port with chctl local server list , then pass it to Tokenomics Viewer: chctl local server start --name tokenomics-alt ./app.js --sync --db-engine clickhouse \ --clickhouse-url http://127.0.0.1:ASSIGNED HTTP PORT For the guided local workflow, run: ./launcher.js The launcher checks for an existing dashboard and ClickHouse, asks once whether ClickHouse should be installed and used, remembers that answer in ~/.config/tokenomics-viewer/launcher.json , starts Tokenomics with sync enabled, waits for the dashboard to become ready, and opens the default browser. A declined ClickHouse prompt falls back to SQLite. Useful launcher controls: ./launcher.js --sqlite one-time SQLite override ./launcher.js --clickhouse one-time ClickHouse override ./launcher.js --no-open do not open a browser ./launcher.js --reset-clickhouse-choice ./launcher.js -- --source codex pass Tokenomics options after -- The reset command removes only the remembered ClickHouse decision and exits; the next normal launch asks again. In non-interactive environments with no remembered choice, the launcher uses SQLite without persisting a decline. When ClickHouse installation is accepted, the launcher downloads and executes the official https://clickhouse.com/cli installer, selects the stable local release, and starts the named tokenomics server on HTTP 8123 and TCP 9000 . Automatic installation is supported on macOS and Linux. Run an in-memory text report over the default local roots: ./app.js Write JSON: ./app.js --json --output report.json Scan explicit paths: ./app.js /path/to/session.jsonl /path/to/archived sessions.zip Build or update a SQLite database: ./app.js --sync --db tokenomics.sqlite Serve the browser dashboard: ./app.js --webserver --db tokenomics.sqlite Open the printed local URL, usually: http://127.0.0.1:8787 Serve an existing database without rescanning logs: ./app.js --webserver --db tokenomics.sqlite --no-sync Use a local ClickHouse server instead of SQLite: chctl local server start --name tokenomics --http-port 8123 --tcp-port 9000 ./app.js --sync --webserver --db-engine clickhouse By default ClickHouse mode connects to http://127.0.0.1:8123 and uses the tokenomics database. Override with --clickhouse-url , --clickhouse-database , --clickhouse-user , and --clickhouse-password , or the matching TOKENOMICS CLICKHOUSE environment variables. ClickHouse inserts use large bounded batches by default: up to 100,000 rows or 32 MiB per JSONEachRow request, whichever comes first. Tune that balance with --clickhouse-insert-batch-rows and --clickhouse-insert-batch-bytes when the server can absorb larger inserts or the client needs a lower memory ceiling. New ClickHouse tables are created with per-column codecs: ZSTD for text and stored JSON, Delta+ZSTD for counters and timestamps, Gorilla+ZSTD for floats, and T64+ZSTD for compact flags. Compatible schema additions are applied in place. To discard and rebuild all Tokenomics-owned ClickHouse tables, use: ./app.js --sync --db-engine clickhouse --clickhouse-reset To render a report from an already-synced ClickHouse database without rescanning logs: ./app.js --db-engine clickhouse --json When no paths are passed, Tokenomics Viewer scans: ~/.claude/projects/ / .jsonl ${CODEX HOME:-~/.codex}/sessions/ / .{jsonl,jsonl.zst} ${CODEX HOME:-~/.codex}/archived sessions/ / .{jsonl,jsonl.zst,zip} Use --source claude , --source codex , --archives , and --no-archives to control default discovery. ZIP archives and Codex Zstd-compressed rollouts are read directly without extracting entries to disk. When both .jsonl and .jsonl.zst representations exist during a Codex compression transition, Tokenomics reads only the plain file. The report includes: - totals by provider, model, project, day, week, month, year - per-session processing metrics - input, cache-create, cache-read, output, and reasoning-output token counts - cost breakdown by token category - Codex rate-limit burn summaries when rate-limit snapshots are present - unpriced model buckets when a model is missing from the static pricing table --sync --db <path stores normalized rows in SQLite: sources sessions usage events rate limit samples Sync is incremental by source fingerprint. If a JSONL file or ZIP entry changes, that source is replaced in a transaction instead of duplicated. Generated SQLite files and reports are ignored by .gitignore . --db-engine clickhouse stores the same normalized rows in ClickHouse tables: sources sessions usage events rate limit samples Each source version is immutable. A sync stages changed sources and a complete source manifest, then publishes one global generation marker last. Reports pin that generation for every aggregation query, so a failed multi-source sync leaves the previous complete report visible instead of exposing a partial mix. Source fingerprints include independent analytics-derivation and pricing-catalog versions. Bumping either version automatically reimports unchanged source files, so stored token splits and estimated costs cannot silently outlive the code that derived them. Database schema versioning remains a separate concern. The web dashboard reuses the report produced by startup sync instead of rebuilding it for every API request. In ClickHouse mode, summary buckets are computed with ClickHouse aggregations rather than streaming all usage rows into Node.js. Sync streams normalized rows into ClickHouse in bounded chunks, so large session files do not need to fit in the JavaScript heap. --clickhouse-reset drops and recreates all Tokenomics-owned tables before sync; normal upgrades do not require a reset. --webserver serves: / dashboard HTML /api/summary /api/sessions /api/report /api/sync sync status and protected start action /api/sync/events live sync progress over server-sent events The dashboard shows canvas-based daily token-flow, cost-mix, and per-project daily cost charts with accumulated mouse-wheel zoom and drag selection. Small trackpad deltas are accumulated before changing the visible range. Hover labels use the same tokens / $amount / percent format for input, cache, and output. Analyst mode includes Cost & Resource Diagnostics . It reuses the Models date range and compares effort rows only within one selected provider/model cohort. The table reports usage-event count, tariff coverage, estimated spend, covered input/cache/output per event, amortized total spend per output, output tariff, cache-read share, and reasoning share. Tariff coverage means that the local pricing catalog recognized an event; it does not prove that the event was billed. Usage event is also deliberately not labeled as a user request or completed task. Without outcome or quality data the section remains descriptive and does not rank effort levels by efficiency. The header switches between Overview and Analyst modes. Overview keeps the model ranking to ten rows and hides project/diagnostics detail. Analyst exposes the full stored model list and the detailed project and diagnostics sections. Both modes show deterministic recommendation findings generated from the same report. Recommendations include evidence, confidence, a concrete action, and a caveat; unpriced traffic is not assumed to be billable because subscription or intentionally non-billable models may have no API tariff. The server binds to 127.0.0.1 by default. Use --host only if you understand that reports can contain local file paths, project names, usage patterns, and estimated spending. The dashboard Sync button is enabled only when the webserver is bound to a loopback host. Bindings such as 0.0.0.0 and LAN addresses remain read-only: reports are served, but sync mutations are rejected. This is intentional; the custom action header prevents browser CSRF but is not remote-user authentication. Pricing is a static table in lib/core/pricing.js . Treat estimates as audit aids, not billing truth. Verify current provider pricing before relying on the numbers for financial decisions. GPT-5.6 Sol, Terra, and Luna use separate input, 30-minute cache-write, and cache-read prices. Codex logs are interpreted conservatively: the legacy input tokens plus cached input tokens format treats the cached amount as a subset of input read only , while the explicit cache creation input tokens plus cache read input tokens format records a 30-minute cache write separately. A legacy log cannot prove a cache-write quantity, so the estimate leaves that bucket at zero instead of inferring it from input. Source: https://developers.openai.com/api/docs/models/gpt-5.6-sol https://developers.openai.com/api/docs/models/gpt-5.6-sol . Session logs, generated reports, and SQLite databases can contain sensitive metadata such as local paths, project names, timestamps, model names, and usage patterns. Do not publish generated output unless you have reviewed it. app.js is the executable composition root. Implementation is split by responsibility: lib/core/ contains report state, usage normalization, pricing, aggregation, and rate-limit calculations. lib/ingest/ contains the JSONL parser, fork/replay handling, source discovery, and ZIP reader. lib/storage/ contains independent SQLite and ClickHouse backends plus their shared facade. lib/report/ , lib/web-server.js , and lib/cli.js contain presentation, HTTP, and command-line boundaries. test/ .test.js mirrors those domains; shared fixture builders live under test/support/ . Run tests: node --test Check syntax: node --check app.js