{"slug": "tokenade-is-now-open-source", "title": "Tokenade is now Open Source", "summary": "Tokenade, a Rust binary that compresses context between AI coding agents and models, is now open source, with the project reporting it ranks #1 on the THOL (Token-Harness Optimizer Leaderboard) at 39% cheaper on long sessions. Tokenade folds build logs, verbose MCP payloads and redundant file reads before they are billed, keeps every folded byte recoverable via hash-based `tokenade expand-ref`, and installs through `npm install -g @tokenade/cli` followed by `tokenade install`, which detects coding agents such as Claude Code, Codex, Cursor, Copilot CLI, OpenCode, Kilo Code and Qwen Code. The release ships per-platform npm packages and Ed25519-signed GitHub Release archives for Linux (x64 glibc/musl, arm64), macOS (x64, arm64) and Windows (x64, arm64), requiring Node ≥ 18.", "body_md": "**Cut your AI coding agent's token bill — locally, automatically, on every prompt.**\n\n[Install](#install) ·\n  [What it does](#what-it-does) ·\n  [Benchmark](https://tokenade.net/en/thol) ·\n  [Website](https://tokenade.net)\n\nTokenade is a single Rust binary that sits between your coding agent (Claude Code, Codex, Cursor, Copilot CLI, OpenCode, Kilo Code, Qwen Code…) and the model. It strips what the model never needed to see — 2,000-line build logs, whole files when one function mattered, verbose MCP payloads, re-reads of content already in context — before it is billed, and keeps every byte it folds recoverable on demand.\n\nTokenade ranks **#1 on THOL**, the Token-Harness Optimizer Leaderboard —\nreal end-to-end agent runs, every optimizer measured the same way:\n**39% cheaper on long sessions**. Method and full results:\n[https://tokenade.net/en/thol](https://tokenade.net/en/thol).\n\n- **Command compaction** — test runs, builds, logs,`git` ,`kubectl` ,`docker` , package managers… noisy output is folded to what matters\n(failures, numbers, decisive lines) before the agent reads it.\n- **Read deduplication** — content the agent already has comes back as a\nshort reference, or as a diff when it changed.\n- **Code intelligence** —`map` ,`query` ,`skeleton` ,`impact` and hybrid`semantic` search, so the agent stops reading whole files to find one thing.\n- **MCP optimization** — MCP servers are wrapped: responses compacted in\ntransit, tool manifests trimmed to their callable skeleton.\n- **Web** —`tokenade web` /`tokenade search` return compact Markdown and\nresult lists instead of raw HTML.\n- **Context and output** — prompt-cache-aware compaction of stale context,\nand a lean-output style note that drops filler without dropping facts.\n- **Secret redaction** — credentials are scrubbed from what Tokenade logs,\ncaches and folds.\n- **Nothing is lost** — every fold carries a hash;`tokenade expand-ref` gives the exact bytes back.\n\n```\nnpm install -g @tokenade/cli\ntokenade install      # detects your coding agents and wires Tokenade in\ntokenade healthcheck  # optional: verify the wiring\n```\n\nThe binary for your platform ships as an optional per-platform npm package;\nif that is skipped, `postinstall` downloads the archive from this repository's\nGitHub Release and checks it against the release's Ed25519-signed manifest\n(SHA-256). Linux (x64 glibc/musl, arm64), macOS (x64, arm64), Windows (x64,\narm64). Node ≥ 18.\n\nPrebuilt archives are also attached to every\n[GitHub Release](https://github.com/pi-infected/tokenade/releases).\n\nRequires a recent stable Rust toolchain ([rustup](https://rustup.rs)) and a\nC/C++ compiler (tree-sitter grammars and SQLite are compiled from source).\n\n```\ngit clone https://github.com/pi-infected/tokenade\ncd tokenade\ncargo install --locked --path crates/tokenade-cli   # or use a script below\ntokenade install\n```\n\nPer-OS scripts in `scripts/build/` build the same targets as the release CI\nand print where the binary is:\n\n- **Linux** (x86_64 / aarch64):`scripts/build/build-linux.sh` for a native\nglibc build;`--musl` for a static binary (uses[`cargo zigbuild`](https://github.com/rust-cross/cargo-zigbuild) if\ninstalled);`--target <triple>` for anything else.`lld` is used as linker\non x86_64 when present.\n- **macOS** (Xcode Command Line Tools):`scripts/build/build-macos.sh` for\nthis Mac's architecture;`--target aarch64-apple-darwin` (Apple Silicon) or`--target x86_64-apple-darwin` (Intel);`--universal` for both, merged with`lipo` .\n- **Windows** (PowerShell):`.\\scripts\\build\\build-windows.ps1` builds`x86_64-pc-windows-gnu` with a static CRT; it needs MinGW-w64 (`gcc` ,`g++` )\non`PATH` , e.g. from MSYS2.`-Target aarch64-pc-windows-gnullvm` builds\nARM64 with[llvm-mingw](https://github.com/mstorsjo/llvm-mingw) .\n\nThe semantic-search embedding model is downloaded on first use (Hugging Face,\nwith a signed GitHub Release asset as fallback) and cached; set\n`TOKENADE_MODEL_DIR` to a directory that already holds it for offline machines.\n\nFull integration (hooks: command compaction, read dedup, web, MCP, code\nsearch, lean output, context optimization): **Claude Code**, **OpenCode**,\n**Kilo Code**, **Copilot CLI**, **Hermes**, **Cline**, **T3 Code**.\n\nMost features: **OpenClaw** and **Pi** (no MCP wrapping); **Qwen Code** (no\nweb replacement); **Codex CLI** and the Codex app, **Cursor**, **Grok** (no\nweb replacement or read dedup); **Droid** (no web, MCP wrapping or read\ndedup).\n\nPartial: **Aider** (run commands through `tokenade wrap …`),\n**Antigravity** and **Claude Desktop chat** (MCP optimization only).\nClaude Desktop's Code tab runs Claude Code and gets full integration.\n\n`tokenade install --only <agent>` limits the install to one agent;\n`tokenade uninstall` removes everything it added.\n\nYour savings\nledger and dashboard (`tokenade gain`, `tokenade dashboard`) stay on your\nmachine. The only network calls are:\n\n- the ones you ask for (`tokenade web` ,`tokenade search` );\n- the one-time embedding-model download (see above);\n- a daily **auto-update** check that reads the signed`manifest.json` of the\nlatest GitHub Release and only installs a binary whose signature and\nSHA-256 verify.\n\nTurn auto-update off with `TOKENADE_NO_AUTO_UPDATE=1` (or\n`tokenade upgrade --off`); `tokenade upgrade` updates on demand.\n`TOKENADE_NO_WEBSEARCH=1` / `TOKENADE_NO_WEBFETCH=1` keep your agent's own\nweb tools.\n\n1.x installs update themselves to 2.0 (or run `npm i -g @tokenade/cli@latest`).\nOn its first run, 2.0 deletes the leftover 1.x licence, quota and\nusage-report files under `~/.tokenade` once and keeps your savings history.\n`tokenade login`, `activate` and `report` no longer exist. Run\n`tokenade install` again if your agent's hooks look stale.\n\n```\ntokenade wrap 'cargo test'          # run a noisy command, get the compacted output\nsome-cmd | tokenade read -          # compact any blob; format auto-detected\ntokenade map                        # codebase structure\ntokenade query parse_config         # locate a symbol\ntokenade skeleton src/lib.rs        # signatures only\ntokenade semantic \"where is auth\"   # search by meaning\ntokenade expand-ref <hash>          # get folded bytes back\ntokenade dashboard                  # what you saved: graphical local report\ntokenade gain                       # same totals, in the terminal\ntokenade help                       # everything else\n```\n\nBug reports, compactor fixtures and pull requests are welcome — see\n[CONTRIBUTING.md](https://github.com/pi-infected/tokenade/blob/main/CONTRIBUTING.md) and the\n[Code of Conduct](https://github.com/pi-infected/tokenade/blob/main/CODE_OF_CONDUCT.md). Security issues: see\n[SECURITY.md](https://github.com/pi-infected/tokenade/blob/main/SECURITY.md).\n\nLicensed under either of\n\n- Apache License, Version 2.0 ([LICENSE-APACHE](https://github.com/pi-infected/tokenade/blob/main/LICENSE-APACHE) )\n- MIT license ([LICENSE-MIT](https://github.com/pi-infected/tokenade/blob/main/LICENSE-MIT) )\n\nat your option (`MIT OR Apache-2.0`).\n\nUnless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this project by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.", "url": "https://wpnews.pro/news/tokenade-is-now-open-source", "canonical_source": "https://github.com/pi-infected/tokenade", "published_at": "2026-10-19 23:06:30+00:00", "updated_at": "2026-10-19 23:46:22.414426+00:00", "lang": "en", "topics": ["ai-tools", "ai-agents", "developer-tools", "ai-infrastructure"], "entities": ["Tokenade", "THOL", "Claude Code", "Codex", "Cursor", "GitHub", "Hugging Face", "Rust"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/tokenade-is-now-open-source", "markdown": "https://wpnews.pro/news/tokenade-is-now-open-source.md", "text": "https://wpnews.pro/news/tokenade-is-now-open-source.txt", "jsonld": "https://wpnews.pro/news/tokenade-is-now-open-source.jsonld"}}