JetBrains tested Caveman, RTK and Ponytail's token-saving claims against Claude Code. Here's how their real results compare to the marketing.
Do Claude token-saving plugins actually save tokens? #
Mostly not by much. JetBrains ran an independent test of three popular Claude Code token-saving tools, Caveman, RTK, and Ponytail, against real coding tasks. The results fell far short of what each tool advertises. Caveman claimed roughly 65% savings on output tokens but delivered about 8.5%. RTK didn’t save anything meaningful and actually made tasks more expensive at low effort settings. Ponytail was the only one that came out ahead, cutting costs by about 10% per task, which is still half of its advertised number.
TL;DR #
- Caveman advertises around 65% output token savings but only delivered about 8.5% in JetBrains’ testing on real coding tasks.
- RTK showed no real savings and in some configurations, particularly at low effort settings, it increased the cost of completing a task.
- Ponytail was the only tool that worked , saving roughly 10% per task, half of its marketed savings figure.
- Ponytail has a known Windows bug that can stall every message for several seconds, which is worth weighing against its modest savings.
- A separate tool called Codeburn doesn’t promise savings directly but shows where your tokens actually go, including MCP servers that silently load on every message without being used.
- Two previously recommended memory tools, Claude Mem and Context Mode, come with caveats : Claude Mem sends usage data by default, and Context Mode has an open bug that can burn through usage unexpectedly.
- The gap between marketing and measured results is the real story here : anyone relying on these plugins to cut Claude Code bills should test on their own workload rather than trust the advertised percentage.
Everyone else built a construction worker.
We built the contractor.
One file at a time.
UI, API, database, deploy.
What are Caveman, RTK, and Ponytail? #
These are third-party, open-source plugins built for Claude Code that claim to reduce the number of tokens Claude consumes during coding sessions, and therefore reduce API costs. The pitch is simple: Claude Code burns through tokens quickly on long sessions, especially when context gets reloaded or outputs run longer than necessary. Each of these tools proposes a different mechanism to trim that usage, with marketed savings in the tens of percent.
The appeal is obvious for anyone running Claude Code regularly, whether that’s a solo developer or someone building client projects where API costs eat into margin. A tool that genuinely cuts token spend by even 10-20% adds up fast over a month of heavy use.
How did JetBrains test these claims? #
JetBrains ran the three plugins against real coding tasks on Claude Code rather than synthetic benchmarks, measuring actual output token counts and cost per task. This matters because marketed savings figures from plugin developers are often based on best-case scenarios, specific prompt types, or internal benchmarks that don’t reflect typical day-to-day coding work. Independent testing on realistic workloads is a better signal of what you’ll actually see in production.
The gap between advertised and measured numbers was large enough to matter. Caveman’s claimed 65% turned into roughly 8.5% in practice, which is nearly an eightfold difference. That’s not a rounding error, it suggests the advertised figure came from a narrow or favorable test case that doesn’t generalize.
Is Ponytail worth installing? #
Of the three, Ponytail is the only one that showed a real, measurable benefit in JetBrains’ testing, saving about 10% per task. That’s meaningful if you’re running Claude Code constantly, but it’s also half of what Ponytail advertises, so expectations should be calibrated accordingly.
The catch is a known bug on Windows that can stall every message for several seconds. If you’re on Windows and value responsiveness during a coding session, that latency cost might outweigh a 10% token saving, especially since interrupted flow has its own cost in time and focus. Mac and Linux users don’t appear to hit this issue based on the available information, which makes Ponytail a more reasonable pick there.
Why did RTK make tasks more expensive? #
RTK’s failure is the most counterintuitive result. A tool built specifically to reduce token usage ended up increasing cost in low-effort configurations. This likely comes down to overhead: if a token-saving mechanism adds its own processing, instructions, or context to every request, it can offset or exceed whatever it trims elsewhere. When that overhead is fixed and the task itself is small or low effort, the overhead becomes a larger proportion of the total cost, which is exactly the scenario where RTK underperformed.
The lesson generalizes beyond RTK: any tool that modifies how Claude handles context or output needs to be tested against the kind of tasks you actually run, not just large or complex ones where overhead is negligible by comparison.
What’s a better way to manage Claude Code token costs? #
Remy doesn't build the plumbing. It inherits it. #
Other agents wire up auth, databases, models, and integrations from scratch every time you ask them to build something.
Remy ships with all of it from MindStudio — so every cycle goes into the app you actually want.
Rather than installing a plugin and trusting its marketing, a more reliable approach is to measure where your tokens are actually going. A tool called Codeburn does this: it shows token spend broken down by source and flags waste, including MCP servers that load into context on every single message even if you never use them in a given session. That’s a common and invisible cost in Claude Code setups, since every connected MCP server adds to the baseline context regardless of whether it’s called.
Identifying and removing unused MCP servers, trimming overly verbose system prompts, and being deliberate about what stays loaded in context will often save more than a plugin promising a flat percentage reduction. It’s less convenient than installing one tool and forgetting about it, but it’s grounded in your actual usage pattern rather than someone else’s benchmark.
Should you still use memory tools like Claude Mem or Context Mode? #
Caution is warranted for both right now. Claude Mem, a tool for persisting context and memory across Claude Code sessions, sends usage data by default, which is a privacy consideration worth knowing about before enabling it on client work or anything sensitive. Context Mode has an open bug that can burn through usage unexpectedly, which defeats the purpose of a tool meant to help you manage consumption. Both were recommended in earlier coverage of Claude Code tooling, but the current state of each suggests holding off until the issues are resolved.
Frequently Asked Questions #
What is the actual token savings rate for Caveman?
JetBrains measured about 8.5% reduction in output tokens, far below the roughly 65% Caveman advertises.
Does RTK save money on Claude Code tasks?
No. JetBrains found no meaningful savings, and in low-effort task configurations RTK actually increased cost compared to not using it.
Is Ponytail safe to use on Windows?
Ponytail has an open bug on Windows that can stall every message for several seconds. It still delivers the best measured savings of the three tools tested (about 10%), but that latency issue is a real tradeoff on Windows machines.
How can I check where my Claude Code tokens are actually going?
A tool called Codeburn tracks and displays token spend by source, including flagging MCP servers or other integrations that load into every message’s context without actually being used.
Should I trust advertised savings percentages from Claude plugins?
Based on independent testing, advertised figures can overstate real-world performance significantly. Caveman’s claim was off by roughly eightfold. It’s worth testing any token-saving tool against your own typical workload before relying on the marketed number.