{"slug": "rtk-reports-huge-token-savings-but-our-cost-benchmarks-disagree", "title": "RTK reports huge token savings, but our cost benchmarks disagree", "summary": "JetBrains's SkillsBench benchmark found that RTK (Rust Token Killer), a terminal-output compression tool with over 79,000 GitHub stars, produced no consistent cost savings for AI coding agents, with DeepSeek V4 Pro 0813 task costs rising 17% on average and Fable 5.0 costs rising 1% on a task-level measure across 1,740 attempts. RTK's own README states it cuts up to 90% of bash output an agent reads but \"it is not the same as cutting your bill by 90%,\" and the tool's `rtk gain` metric reports 349.2 million tokens saved across 445 DeepSeek RTK attempts — an 89% reduction — which the benchmark authors say is not a measure of billed tokens. The findings, based on several days of testing and over $1,500 spent on tokens, contradict a viral X post claiming RTK could cut Claude Code tokens by up to 60%.", "body_md": "[RTK](https://github.com/rtk-ai/rtk) (Rust Token Killer) filters and compresses terminal output before the AI agent reads it. With over 79k GitHub stars today, RTK is one of the most popular tools to make AI coding cheaper.\n\nOne X post saying [RTK could cut Claude Code tokens by up to 60%](https://x.com/jasonzhou1993/status/2038215854584906078) reached 313K views.\n\nYet [JetBrains’s SkillsBench run found no savings](https://blog.jetbrains.com/ai/2026/07/rtk-claude-code-token-savings/). The [README](https://github.com/rtk-ai/rtk#how-savings-work) has a disclaimer:\n\nRTK cuts up to 90% of the bash output your agent reads. […] it is not the same as cutting your bill by 90%.\n\nSo “less terminal output” is not the same as “cheaper AI coding”. It can help, be a no-op, or backfire (more turns or lower quality). In this post, we present our findings after several days and over $1,500 spent on tokens.\n\n## How RTK works\n\nRTK can rewrite Git, test, package and file commands the agent runs through its shell tool (`Bash` in Claude Code, `bash` in OpenCode). Each rewrite returns a terser version of the same output.\n\nFor example, RTK keeps file names, sizes and permissions (`644` means `rw-r—r—`), but drops the owner and date:\n\n``` bash\n$ ls -la /app/warriors\n-rw-r--r-- 1 root root  824 Sep 13  2025 g2-clear.red\n-rw-r--r-- 1 root root  487 Sep 13  2025 paper.red\n\n$ rtk ls -la warriors/\n644  g2-clear.red  824B\n644  paper.red  487B\n```\n\n## Testing RTK on Terminal-Bench 2.1\n\nRTK compresses terminal output, so we tested it on [Terminal-Bench 2.1](https://www.tbench.ai/news/terminal-bench-2-1), a benchmark with heavy terminal interaction. We stayed on 2.1 rather than the newer [3.0](https://www.tbench.ai/news/terminal-bench-3-0) and [4.0](https://www.tbench.ai/news/terminal-bench-4-0): agents pass most 2.1 tasks, while 3.0 and 4.0 are still a challenge. Cost only matters for tasks that pass.\n\nWe ran Claude Code with Fable 5.0, and OpenCode with DeepSeek V4 Pro 0813 through OpenRouter. Each task was scheduled five times without RTK and five times with it, on the same model route, platform and task-specific timeout.\n\nAfter removing four Fable security tasks that got refusals, the final comparison covers 85 Fable tasks and 89 DeepSeek tasks, or 1,740 attempts.\n\n## The first chart was promising\n\nWith RTK, costs fell by 5% for Fable and rose by 5% for DeepSeek.\n\nPass rates were lower with RTK: by 1% for Fable and 2% for DeepSeek. Both pass-rate gaps are small.\n\nWhen we divided all spending, including failed attempts, by the number of passes, Fable was 3% cheaper with RTK, and DeepSeek was 7% more expensive.\n\nAnother way is to weight every task equally, because one expensive task can outweigh many cheap ones. We compared the mean of each task’s baseline attempts with the mean of its RTK attempts, then averaged those changes.\n\nOn this task-level measure, Fable was 1% more expensive, with no clear difference from zero. DeepSeek’s task cost rose **17% on average**.\n\nAccounting for failures does not change the trend. Across the 36 DeepSeek tasks where all ten attempts passed, the increase was still 18%.\n\n## One task made the difference in the whole benchmark\n\nAlmost all of Fable’s savings with RTK came from one task: `winning-avg-corewars`. Both setups passed every attempt, but with RTK it finished in about half as many turns. Across the other tasks, the savings were less than 1%.\n\nDeepSeek had the reverse result on that same task. Both setups passed every attempt, but RTK took more turns and cost more. Even without that task, costs remained higher with RTK.\n\n## `rtk gain` is useless as a cost metric\n\nRTK documents [`rtk gain`](https://github.com/rtk-ai/rtk/blob/develop/docs/guide/resources/savings-explained.md#how-to-read-rtk-gain) as raw minus filtered command output in bytes, divided by 4, not a count of billed tokens.\n\nAcross 445 DeepSeek RTK attempts, RTK reported **349.2 million tokens saved**, a 89% reduction.\n\nLarge reported token savings did not mean cheaper tasks.\n\nIn `train-fasttext`, the model requested `head -1 train.txt` twice. RTK credited 120.5 million tokens saved each time by comparing those limited reads with the whole file. Those two calls accounted for **69% of the comparison’s savings counter**, although the requested commands would never have returned the whole file.\n\nTreating `rtk gain` as money saved assumes the rest of the attempt would stay the same. RTK can change the agent’s next turns. `rtk gain` does not account for the cost of those turns.\n\nThis is where social posts go wrong: `rtk gain` counts removed output, not money saved, and it can make a more expensive attempt look optimized.\n\n## RTK bugs can bite you\n\nOne DeepSeek `git-multibranch` attempt got stuck in a loop. The agent ran a `find` with a flag that `rtk find` 0.45.0 did not support. The plugin rewrote it to `rtk find`, which failed with “Use `find` directly”. Every retry was rewritten again. RTK [fixed this](https://github.com/rtk-ai/rtk/commit/6370e79275ef8c1063fc9b682bc36e7e63041b53) in 0.46.0, after our runs.\n\nThe agent accumulated **339 consecutive errors** before its timeout. It still passed the task, but cost **about 9× as much** as the matching baseline attempt, which also passed. One outlier attempt; the trend holds without it.\n\n## Terminal output is a small share of the bill\n\nWithout RTK, tool output made up about 11% of Fable’s input tokens and 40% of DeepSeek’s.\n\nIn the RTK attempts, 31% of Claude Code’s terminal calls and 51% of OpenCode’s terminal calls used RTK.\n\nRTK rewrites only shell commands: its Claude Code hook matches the `Bash` tool and its OpenCode plugin acts on `bash` calls. Both platforms expose file reading and searching as separate `Read`, `Grep`, and `Glob` tools, which [bypass RTK](https://github.com/rtk-ai/rtk/blob/v0.45.0/README.md). About half of Claude Code’s Bash calls already limited their own output with `head`, `tail`, or `wc`.\n\nIn agentic coding, the context is cached after each turn, so later reads of terminal output mostly show up as cache reads. Those cost 1/10 of regular input tokens for Fable, and 1/30 for DeepSeek.\n\nIn DeepSeek, RTK reduced terminal-output characters by 9%, yet prompt tokens rose 9%. Uncached input fell 1% and cached input rose 9%. Model output, including reasoning, accounted for 56% of cost with RTK and 57% without it.\n\n## Extra turns can erase the savings\n\nWhen the agent took more turns, task cost usually rose with it.\n\nDeepSeek’s RTK attempts took more turns on 58 tasks, and 44 of them cost more. They took fewer turns on 28, and 23 of them cost less.\n\nThe average DeepSeek turn had 7% less input with RTK, but there were 18% more turns overall. Smaller turns did not add up to less total input.\n\nOne extra agent turn can cost more than the compression saved. It is the same [tokenflation](/blog/tokenflation-when-hi-triggers-33-tool-calls) problem in another form. JetBrains saw the same pattern on SkillsBench: RTK added turns at low effort and did not lower cost at high effort.\n\n## RTK does not make AI coding cheaper\n\nOn Terminal-Bench 2.1, Fable’s savings depended on one task and did not hold across tasks. We do not recommend RTK as a generic cost-saving tool.\n\nIndividual transcripts show that current frontier models already use the terminal efficiently (just ~7% of Fable’s context was terminal output). Models use techniques like `head -n` or `tail -n` themselves. RTK probably helped more with older models. Today it is a niche optimization, not a source of general savings.\n\n*Tested with RTK 0.45.0, Claude Code 2.1.220, OpenCode 1.18.25 and Harbor 0.20. Trajectories available [on request](/cdn-cgi/l/email-protection#5b3834352f3a382f1b2a2e3e28363a75383436) for follow-up research. [Subscribe](#mce-EMAIL) for future posts, including our planned benchmark of [Headroom](https://github.com/headroomlabs-ai/headroom). Thanks to Piotr Migdał for his review and feedback.*", "url": "https://wpnews.pro/news/rtk-reports-huge-token-savings-but-our-cost-benchmarks-disagree", "canonical_source": "https://quesma.com/blog/does-rtk-make-ai-coding-cheaper/", "published_at": "2026-09-11 08:00:00+00:00", "updated_at": "2026-09-11 09:03:25.173721+00:00", "lang": "en", "topics": ["ai-tools", "ai-agents", "developer-tools", "ai-products"], "entities": ["RTK", "JetBrains", "SkillsBench", "Claude Code", "OpenCode", "DeepSeek V4 Pro 0813", "Fable 5.0", "Terminal-Bench 2.1"], "alternates": {"html": "https://wpnews.pro/news/rtk-reports-huge-token-savings-but-our-cost-benchmarks-disagree", "markdown": "https://wpnews.pro/news/rtk-reports-huge-token-savings-but-our-cost-benchmarks-disagree.md", "text": "https://wpnews.pro/news/rtk-reports-huge-token-savings-but-our-cost-benchmarks-disagree.txt", "jsonld": "https://wpnews.pro/news/rtk-reports-huge-token-savings-but-our-cost-benchmarks-disagree.jsonld"}}