{"slug": "tokenomics-view-token-usage-costs-and-rate-limit-metrics", "title": "Tokenomics: View token usage, costs, and rate-limit metrics", "summary": "Tokenomics Viewer, a local-first tool for scanning Codex and Claude Code session logs to estimate token usage and costs, has been released. The tool reads files from the user's machine without uploading data and supports output as text, JSON, SQLite, or a local web dashboard. It requires Node.js 26 or newer and optionally integrates with ClickHouse for data storage.", "body_md": "Tokenomics Viewer scans local Codex and Claude Code session logs, normalizes token usage, estimates costs from static pricing tables, and reports the results as text, JSON, SQLite-backed data, or a local web dashboard.\n\nThe tool is local-first. It reads files from your machine and does not upload logs or reports anywhere.\n\nOn macOS or Linux, install or update Tokenomics Viewer and start the guided setup with:\n\n```\n/bin/sh -c \"$(curl -fsSL https://raw.githubusercontent.com/skuznetsov/tokenomics-viewer/main/install.sh)\"\n```\n\nThe installer does not use `sudo`\n\n. It installs versioned application files in\n`~/.local/share/tokenomics-viewer`\n\n, creates `tokenomics`\n\n, `tokenomics-viewer`\n\n,\nand `tokenomics-launch`\n\ncommands in `~/.local/bin`\n\n, and then starts the guided\nlauncher. If Node.js 26 or newer is unavailable, it installs a private Node.js\n26 runtime after verifying the archive against the official Node.js SHA-256\nmanifest.\n\nThe launcher asks whether to install and use ClickHouse, remembers the answer, runs the initial sync, starts the dashboard, and opens it in the default browser. SQLite data is kept across application updates at:\n\n```\n~/.local/share/tokenomics-viewer/tokenomics.sqlite\n```\n\nRun the same one-line command again to update. To install without starting the dashboard, use:\n\n```\nTOKENOMICS_NO_LAUNCH=1 /bin/sh -c \"$(curl -fsSL https://raw.githubusercontent.com/skuznetsov/tokenomics-viewer/main/install.sh)\"\n```\n\nIf `~/.local/bin`\n\nis not already on `PATH`\n\n, the installer prints the exact\n`export`\n\ncommand to add it. It does not modify shell startup files.\n\n- Node.js 26 or newer\n- No npm dependencies\n\n`node:sqlite`\n\nis used for the SQLite-backed sync and web dashboard modes.\nClickHouse mode uses the ClickHouse HTTP interface and still needs no npm\ndependencies.\n\n[ clickhousectl](https://clickhouse.com/blog/getting-started-clickhousectl) is\nthe official ClickHouse CLI for installing versions and managing isolated local\nservers. The installer also creates the shorter\n\n`chctl`\n\ncommand used below.Install it on macOS or Linux:\n\n```\ncurl -fsSL https://clickhouse.com/cli | sh\n```\n\nThe binary is installed under `~/.local/bin`\n\n. If your shell cannot find it, add\nthat directory to `PATH`\n\nand restart the shell:\n\n```\necho 'export PATH=\"$HOME/.local/bin:$PATH\"' >> ~/.zshrc\nsource ~/.zshrc\nclickhousectl --version\nchctl --version\n```\n\nDownload the current stable ClickHouse release and make it the local default:\n\n```\nchctl local use stable\nchctl local which\n```\n\nFrom the Tokenomics Viewer repository, start a persistent named server on the HTTP port expected by the application:\n\n```\nchctl local server start --name tokenomics --http-port 8123 --tcp-port 9000\nchctl local server list\ncurl http://127.0.0.1:8123/ping\n```\n\nThe server runs in the background and stores its data under\n`.clickhouse/servers/tokenomics/`\n\n. Connect with the bundled ClickHouse client:\n\n```\nchctl local client --name tokenomics --query \"SELECT version()\"\n```\n\nStop, restart, or permanently remove the local instance with:\n\n```\nchctl local server stop tokenomics\nchctl local server start --name tokenomics\nchctl local server stop tokenomics\nchctl local server remove tokenomics\n```\n\nIf port `8123`\n\nis already occupied, omit `--http-port`\n\nand inspect the assigned\nport with `chctl local server list`\n\n, then pass it to Tokenomics Viewer:\n\n```\nchctl local server start --name tokenomics-alt\n./app.js --sync --db-engine clickhouse \\\n  --clickhouse-url http://127.0.0.1:ASSIGNED_HTTP_PORT\n```\n\nFor the guided local workflow, run:\n\n```\n./launcher.js\n```\n\nThe launcher checks for an existing dashboard and ClickHouse, asks once whether\nClickHouse should be installed and used, remembers that answer in\n`~/.config/tokenomics-viewer/launcher.json`\n\n, starts Tokenomics with sync enabled,\nwaits for the dashboard to become ready, and opens the default browser. A\ndeclined ClickHouse prompt falls back to SQLite.\n\nUseful launcher controls:\n\n```\n./launcher.js --sqlite              # one-time SQLite override\n./launcher.js --clickhouse          # one-time ClickHouse override\n./launcher.js --no-open             # do not open a browser\n./launcher.js --reset-clickhouse-choice\n./launcher.js -- --source codex     # pass Tokenomics options after --\n```\n\nThe reset command removes only the remembered ClickHouse decision and exits; the next normal launch asks again. In non-interactive environments with no remembered choice, the launcher uses SQLite without persisting a decline.\n\nWhen ClickHouse installation is accepted, the launcher downloads and executes\nthe official `https://clickhouse.com/cli`\n\ninstaller, selects the stable local\nrelease, and starts the named `tokenomics`\n\nserver on HTTP `8123`\n\nand TCP `9000`\n\n.\nAutomatic installation is supported on macOS and Linux.\n\nRun an in-memory text report over the default local roots:\n\n```\n./app.js\n```\n\nWrite JSON:\n\n```\n./app.js --json --output report.json\n```\n\nScan explicit paths:\n\n```\n./app.js /path/to/session.jsonl /path/to/archived_sessions.zip\n```\n\nBuild or update a SQLite database:\n\n```\n./app.js --sync --db tokenomics.sqlite\n```\n\nServe the browser dashboard:\n\n```\n./app.js --webserver --db tokenomics.sqlite\n```\n\nOpen the printed local URL, usually:\n\n```\nhttp://127.0.0.1:8787\n```\n\nServe an existing database without rescanning logs:\n\n```\n./app.js --webserver --db tokenomics.sqlite --no-sync\n```\n\nUse a local ClickHouse server instead of SQLite:\n\n```\nchctl local server start --name tokenomics --http-port 8123 --tcp-port 9000\n./app.js --sync --webserver --db-engine clickhouse\n```\n\nBy default ClickHouse mode connects to `http://127.0.0.1:8123`\n\nand uses the\n`tokenomics`\n\ndatabase. Override with `--clickhouse-url`\n\n,\n`--clickhouse-database`\n\n, `--clickhouse-user`\n\n, and `--clickhouse-password`\n\n, or\nthe matching `TOKENOMICS_CLICKHOUSE_*`\n\nenvironment variables.\n\nClickHouse inserts use large bounded batches by default: up to 100,000 rows or\n32 MiB per JSONEachRow request, whichever comes first. Tune that balance with\n`--clickhouse-insert-batch-rows`\n\nand `--clickhouse-insert-batch-bytes`\n\nwhen the\nserver can absorb larger inserts or the client needs a lower memory ceiling.\n\nNew ClickHouse tables are created with per-column codecs: ZSTD for text and stored JSON, Delta+ZSTD for counters and timestamps, Gorilla+ZSTD for floats, and T64+ZSTD for compact flags. Compatible schema additions are applied in place. To discard and rebuild all Tokenomics-owned ClickHouse tables, use:\n\n```\n./app.js --sync --db-engine clickhouse --clickhouse-reset\n```\n\nTo render a report from an already-synced ClickHouse database without rescanning logs:\n\n```\n./app.js --db-engine clickhouse --json\n```\n\nWhen no paths are passed, Tokenomics Viewer scans:\n\n`~/.claude/projects/**/*.jsonl`\n\n`${CODEX_HOME:-~/.codex}/sessions/**/*.{jsonl,jsonl.zst}`\n\n`${CODEX_HOME:-~/.codex}/archived_sessions/**/*.{jsonl,jsonl.zst,zip}`\n\nUse `--source claude`\n\n, `--source codex`\n\n, `--archives`\n\n, and `--no-archives`\n\nto\ncontrol default discovery.\n\nZIP archives and Codex Zstd-compressed rollouts are read directly without\nextracting entries to disk. When both `.jsonl`\n\nand `.jsonl.zst`\n\nrepresentations\nexist during a Codex compression transition, Tokenomics reads only the plain\nfile.\n\nThe report includes:\n\n- totals by provider, model, project, day, week, month, year\n- per-session processing metrics\n- input, cache-create, cache-read, output, and reasoning-output token counts\n- cost breakdown by token category\n- Codex rate-limit burn summaries when rate-limit snapshots are present\n- unpriced model buckets when a model is missing from the static pricing table\n\n`--sync --db <path>`\n\nstores normalized rows in SQLite:\n\n`sources`\n\n`sessions`\n\n`usage_events`\n\n`rate_limit_samples`\n\nSync is incremental by source fingerprint. If a JSONL file or ZIP entry changes, that source is replaced in a transaction instead of duplicated.\n\nGenerated SQLite files and reports are ignored by `.gitignore`\n\n.\n\n`--db-engine clickhouse`\n\nstores the same normalized rows in ClickHouse tables:\n\n`sources`\n\n`sessions`\n\n`usage_events`\n\n`rate_limit_samples`\n\nEach source version is immutable. A sync stages changed sources and a complete source manifest, then publishes one global generation marker last. Reports pin that generation for every aggregation query, so a failed multi-source sync leaves the previous complete report visible instead of exposing a partial mix. Source fingerprints include independent analytics-derivation and pricing-catalog versions. Bumping either version automatically reimports unchanged source files, so stored token splits and estimated costs cannot silently outlive the code that derived them. Database schema versioning remains a separate concern.\n\nThe web dashboard reuses the report produced by startup sync instead of\nrebuilding it for every API request. In ClickHouse mode, summary buckets are\ncomputed with ClickHouse aggregations rather than streaming all usage rows into\nNode.js. Sync streams normalized rows into ClickHouse in bounded chunks, so large\nsession files do not need to fit in the JavaScript heap. `--clickhouse-reset`\n\ndrops and recreates all Tokenomics-owned tables before sync; normal upgrades do\nnot require a reset.\n\n`--webserver`\n\nserves:\n\n`/`\n\ndashboard HTML`/api/summary`\n\n`/api/sessions`\n\n`/api/report`\n\n`/api/sync`\n\nsync status and protected start action`/api/sync/events`\n\nlive sync progress over server-sent events\n\nThe dashboard shows canvas-based daily token-flow, cost-mix, and per-project\ndaily cost charts with accumulated mouse-wheel zoom and drag selection. Small\ntrackpad deltas are accumulated before changing the visible range. Hover labels\nuse the same `tokens / $amount / percent`\n\nformat for input, cache, and output.\n\nAnalyst mode includes `Cost & Resource Diagnostics`\n\n. It reuses the Models date\nrange and compares effort rows only within one selected provider/model cohort.\nThe table reports usage-event count, tariff coverage, estimated spend, covered\ninput/cache/output per event, amortized total spend per output, output tariff,\ncache-read share, and reasoning share. `Tariff coverage`\n\nmeans that the local\npricing catalog recognized an event; it does not prove that the event was billed.\n`Usage event`\n\nis also deliberately not labeled as a user request or completed\ntask. Without outcome or quality data the section remains descriptive and does\nnot rank effort levels by efficiency.\n\nThe header switches between `Overview`\n\nand `Analyst`\n\nmodes. Overview keeps the\nmodel ranking to ten rows and hides project/diagnostics detail. Analyst exposes\nthe full stored model list and the detailed project and diagnostics sections.\nBoth modes show deterministic recommendation findings generated from the same\nreport. Recommendations include evidence, confidence, a concrete action, and a\ncaveat; unpriced traffic is not assumed to be billable because subscription or\nintentionally non-billable models may have no API tariff.\n\nThe server binds to `127.0.0.1`\n\nby default. Use `--host`\n\nonly if you understand\nthat reports can contain local file paths, project names, usage patterns, and\nestimated spending.\n\nThe dashboard Sync button is enabled only when the webserver is bound to a\nloopback host. Bindings such as `0.0.0.0`\n\nand LAN addresses remain read-only:\nreports are served, but sync mutations are rejected. This is intentional;\nthe custom action header prevents browser CSRF but is not remote-user\nauthentication.\n\nPricing is a static table in `lib/core/pricing.js`\n\n. Treat estimates as audit aids, not\nbilling truth. Verify current provider pricing before relying on the numbers for\nfinancial decisions.\n\nGPT-5.6 Sol, Terra, and Luna use separate input, 30-minute cache-write, and\ncache-read prices. Codex logs are interpreted conservatively: the legacy\n`input_tokens`\n\nplus `cached_input_tokens`\n\nformat treats the cached amount as a\nsubset of input (read only), while the explicit\n`cache_creation_input_tokens`\n\nplus `cache_read_input_tokens`\n\nformat records a\n30-minute cache write separately. A legacy log cannot prove a cache-write\nquantity, so the estimate leaves that bucket at zero instead of inferring it\nfrom input. Source: [https://developers.openai.com/api/docs/models/gpt-5.6-sol](https://developers.openai.com/api/docs/models/gpt-5.6-sol).\n\nSession logs, generated reports, and SQLite databases can contain sensitive metadata such as local paths, project names, timestamps, model names, and usage patterns. Do not publish generated output unless you have reviewed it.\n\n`app.js`\n\nis the executable composition root. Implementation is split by\nresponsibility:\n\n`lib/core/`\n\ncontains report state, usage normalization, pricing, aggregation, and rate-limit calculations.`lib/ingest/`\n\ncontains the JSONL parser, fork/replay handling, source discovery, and ZIP reader.`lib/storage/`\n\ncontains independent SQLite and ClickHouse backends plus their shared facade.`lib/report/`\n\n,`lib/web-server.js`\n\n, and`lib/cli.js`\n\ncontain presentation, HTTP, and command-line boundaries.`test/*.test.js`\n\nmirrors those domains; shared fixture builders live under`test/support/`\n\n.\n\nRun tests:\n\n```\nnode --test\n```\n\nCheck syntax:\n\n```\nnode --check app.js\n```\n\n", "url": "https://wpnews.pro/news/tokenomics-view-token-usage-costs-and-rate-limit-metrics", "canonical_source": "https://github.com/skuznetsov/tokenomics-viewer", "published_at": "2026-07-14 17:36:02+00:00", "updated_at": "2026-07-14 17:47:49.723700+00:00", "lang": "en", "topics": ["developer-tools", "large-language-models"], "entities": ["Tokenomics Viewer", "Codex", "Claude Code", "ClickHouse", "Node.js"], "alternates": {"html": "https://wpnews.pro/news/tokenomics-view-token-usage-costs-and-rate-limit-metrics", "markdown": "https://wpnews.pro/news/tokenomics-view-token-usage-costs-and-rate-limit-metrics.md", "text": "https://wpnews.pro/news/tokenomics-view-token-usage-costs-and-rate-limit-metrics.txt", "jsonld": "https://wpnews.pro/news/tokenomics-view-token-usage-costs-and-rate-limit-metrics.jsonld"}}