{"slug": "caveman-saves-tokens-by-doing-less-not-just-saying-less", "title": "Caveman Saves Tokens by Doing Less, Not Just Saying Less", "summary": "Caveman, an open-source skill for coding agents, reduces output tokens by 65% on standalone answers but only cuts token usage by 18% in Claude Code and 3.9% in Codex CLI across 60 tasks, while also reducing steps and tool calls. The experiment, run by Julius Brussee, found that medium reasoning effort in Claude Code achieves similar Pass@3 to Caveman with high effort at about 20% lower cost, suggesting a direct reasoning knob may be more effective.", "body_md": "# Caveman Saves Tokens by Doing Less, Not Just Saying Less\n\nCaveman promises an appealing optimization for coding agents:\n\nSame answers, 65% fewer output tokens. Brain still big. Mouth small.\n\nThe idea is simple. Models often produce unnecessary explanations, narrate their tool calls, and pad conclusions with filler. Remove that verbosity and the agent should become faster and cheaper without becoming less capable.\n\nCoding agents can consume a surprising number of tokens while inspecting repositories, running commands, and correcting their own mistakes. A small skill that removes wasted language sounds like an easy efficiency win. But a coding agent operates through a trajectory: call a tool, interpret the result, and decide whether to continue. An instruction that changes how the agent communicates can also change how it acts.\n\nTo measure that effect, I ran 60 tasks across Claude Code and Codex CLI using SWE Bench Verified, Terminal Bench 2, and USACO tasks. Caveman reduced Claude Code token usage by 18% and its reported cost by 25%, but in Codex token usage fell by only 3.9%.\n\nThe most important result was not how many tokens disappeared, but where they disappeared from. With Caveman enabled, Claude Code performed 17% fewer steps and made almost 20% fewer tool calls, and in Codex hidden reasoning tokens fell by 14%. Caveman didn’t merely make the agents speak less. It changed how much work they performed.\n\nAnd if the skill mostly works by making the agent do less, adjusting the reasoning effort might be a better choice. Claude Code exposes reasoning effort as a native setting, and in this experiment medium effort reached roughly the same Pass@3 as Caveman with high effort while costing about 20% less. A style instruction that arrives at a similar operating point indirectly is harder to justify when a direct knob is one flag away.\n\n## What Caveman Promises\n\n[Caveman](https://github.com/JuliusBrussee/caveman) is a skill for Claude Code, Codex, Gemini, and other coding agents. It instructs the model to avoid narration, drop hedging, and communicate in compact fragments.\n\nIts README summarizes the intended behavior with a memorable distinction:\n\nBrain still big. Mouth small.\n\nThe main benchmark compares ten standalone answers, including prompts such as explaining Git rebase, React rerenders, and Docker multistage builds. The reported average falls from about 1,200 output tokens to under 300, the headline reduction of 65%.\n\nThat result is plausible for those prompts. They ask for prose, and Caveman aggressively compresses prose. The problem is the extrapolation.\n\nA single conversational answer and a repository level coding task have very different token distributions. In a coding agent, useful output is source code, patches, and exact error messages. Caveman leaves much of that unchanged.\n\nThe project itself acknowledges this distinction. Its documentation warns that Caveman only targets output tokens, adds input tokens of its own, and may become net negative when the original model is already concise. It also describes the honest evaluation control as Caveman versus a simple terse instruction, rather than Caveman versus an unconstrained baseline.\n\nThose caveats are good. Caveman is free, open source, and doesn’t collect telemetry. The issue is that the headline claims are much broader than the evidence supporting them.\n\n## The Experiment\n\nI evaluated five agent configurations:\n\n| Configuration | Agent | Treatment |\n|---|---|---|\n`claude-baseline` | Claude Code with Sonnet 5 | High effort |\n`claude-caveman` | Claude Code with Sonnet 5 | Caveman injected at session start (high effort) |\n`claude-medium` | Claude Code with Sonnet 5 | Medium reasoning effort |\n`codex-baseline` | Codex CLI with GPT 5.6 Terra | High effort |\n`codex-caveman` | Codex CLI with GPT 5.6 Terra | Caveman forcibly activated (high effort) |\n\nThe benchmark contained 60 tasks: 20 from SWE Bench Verified, 30 from USACO, and 10 from Terminal Bench 2. Each task was executed three times under every configuration, for 900 trials in total.\n\nThe `claude-medium`\n\nconfiguration is particularly important. Comparing only baseline against Caveman mixes together two possible effects:\n\n- Caveman may compress communication.\n- Caveman may cause the model to spend less effort.\n\nThe medium effort arm provides a native way to reduce model effort without installing Caveman. It lets us ask whether the skill offers something beyond indirectly making the agent do less work.\n\n## Results\n\nClaude Code showed substantial savings:\n\n| Metric | Baseline | Caveman | Change |\n|---|---|---|---|\n| Total tokens | 1,595,199 | 1,302,271 | 18.4% lower |\n| Output tokens | 19,860 | 16,059 | 19.1% lower |\n| Reported cost | $1.003 | $0.758 | 24.5% lower |\n| Steps | 25.5 | 21.2 | 17.0% lower |\n| Tool calls | 24.6 | 19.7 | 19.7% lower |\n| Agent prose characters | 2,956 | 1,741 | 41.1% lower |\n| Pass@1 | 81.1% | 80.0% | 1.1 points lower |\n| Pass@3 | 91.7% | 86.7% | 5.0 points lower |\n\nCodex showed a much smaller reduction:\n\n| Metric | Baseline | Caveman | Change |\n|---|---|---|---|\n| Total tokens | 418,852 | 402,402 | 3.9% lower |\n| Output tokens | 5,252 | 4,757 | 9.4% lower |\n| Reasoning tokens | 2,542 | 2,200 | 13.5% lower |\n| Reported cost | $0.257 | $0.249 | 3.2% lower |\n| Steps | 17.8 | 17.5 | 1.8% lower |\n| Tool calls | 11.8 | 11.5 | 2.7% lower |\n| Pass@1 | 89.4% | 85.0% | 4.4 points lower |\n| Pass@3 | 93.3% | 90.0% | 3.3 points lower |\n\nIf we stop here, Caveman looks like a reasonable tradeoff for Claude and a poor one for Codex. Claude saves approximately one quarter of its modeled cost while losing little Pass@1, while Codex gives up considerably more performance for a 3.2% cost reduction.\n\nBut these aggregated numbers hide the most important part of the experiment.\n\n## The Wrong Denominator\n\nCaveman promotes a reduction in output tokens. Coding agent cost, however, isn’t dominated by visible prose.\n\nFor Claude Code, the baseline token distribution looked like this:\n\n| Quantity | Share |\n|---|---|\n| Cache hit rate | 97.8% |\n| Output tokens as a share of all tokens | 1.24% |\n| Visible prose as a share of output tokens | 3.7% |\n| Visible prose as a share of all tokens | 0.046% |\n\nDirectly generated prose accounts for only about 0.05% of Claude’s reported session token volume. 1 A 41% reduction therefore removes only about 0.02% of the total tokens directly.\n\nShorter replies also make subsequent contexts slightly smaller, so the full mechanical saving from prose compression is larger than 0.02%. But even a deliberately generous fixed trajectory estimate puts that effect well below 1% of total token volume. Most output tokens are code, diffs, and commands that must remain exact. That’s nowhere near the observed 18% reduction.\n\nThe missing savings have to come from somewhere else. The trajectory data provides the answer: with Caveman, Claude took 17% fewer steps and made almost 20% fewer tool calls. Each removed turn avoided processing a large context again.\n\nThe expensive part of an agent is taking another turn.\n\n## Decomposing the Reduction\n\nThe total token count can be decomposed approximately into two factors:\n\n```\ntotal tokens = number of steps × tokens per step\n```\n\nApplying that decomposition gives the following:\n\n| Comparison | Total tokens | Steps | Tokens per step |\n|---|---|---|---|\n| Codex baseline to Caveman | 3.9% lower | 1.8% lower | 2.2% lower |\n| Claude baseline to Caveman | 18.4% lower | 17.0% lower | 1.7% lower |\n| Claude baseline to medium | 37.1% lower | 20.7% lower | 20.6% lower |\n\nFor Claude, approximately 92% of the token reduction is explained by taking fewer steps. **Tokens per step barely changed.**\n\nCaveman certainly changed the writing style. Visible prose fell by 41%, and prose per step also decreased. But this compressed prose was too small to explain the session level savings. The savings appeared because the model ended its trajectory earlier.\n\nThat interpretation is also visible in the tool usage:\n\n| Metric | Claude baseline to Caveman |\n|---|---|\n| Steps | 17.0% lower |\n| Tool calls | 19.7% lower |\n| Shell calls | 19.0% lower |\n| Agent execution time | 15.6% lower |\n\nA style instruction propagated into the action policy of the agent. This doesn’t automatically mean that the removed steps were useful. Agents can overinspect repositories, repeat tests unnecessarily, and continue reasoning after they already have a valid solution. Some trajectories should be shorter.\n\nBut it does mean that Caveman shrinks the brain along with the mouth.\n\n## The Control That Changes the Conclusion\n\nThe strongest argument against using Caveman isn’t the small drop in quality. It’s the `claude-medium`\n\ncontrol.\n\nBoth configurations reached exactly the same Pass@3:\n\n```\nClaude Caveman: 86.7%\nClaude medium:  86.7%\n```\n\nBut Caveman used 30% more tokens and cost 23% more than medium.\n\nThe chart makes the relevant comparison visible. On this plane, a configuration is dominated when another configuration achieves at least the same quality at a lower cost. Claude medium sits directly to the left of Claude Caveman: same Pass@3, lower cost.\n\nThe difference remains after accounting for successful trials:\n\n| Metric | Medium | Caveman | Caveman overhead |\n|---|---|---|---|\n| Pass@1 | 77.2% | 80.0% | 2.8 points higher |\n| Pass@3 | 86.7% | 86.7% | no change |\n| Tokens per resolved trial | 1.30 million | 1.63 million | 25.3% more |\n| Cost per resolved trial | $0.749 | $0.874 | 16.6% more |\n\nThe quality comparison between Caveman and medium is inconclusive at Pass@1: Caveman was 2.8 points higher, but the difference was not statistically significant. The efficiency comparison is much clearer. Both configurations reached the same 87% Pass@3, yet Caveman consumed 25% more tokens per resolved trial and cost 17% more.\n\nThis comparison changes the practical recommendation. Caveman can reduce resource use, but it does so by indirectly altering the agent’s trajectory. When Claude already provides a direct control for effort, using a global style instruction to reach a similar operating point is difficult to justify.\n\n## Why Codex Saved Less\n\nCaveman reduced Claude token usage by 18%, but reduced Codex usage by only 3.9%. One possible explanation is that Codex had much less waste available to remove.\n\nBefore applying Caveman, the Codex and Terra combination already used substantially shorter and cheaper trajectories:\n\n| Baseline metric | Claude Code with Sonnet 5 | Codex with Terra |\n|---|---|---|\n| Tokens per trial | 1,595,199 | 418,852 |\n| Steps per trial | 25.5 | 17.8 |\n| Tokens per step | 62,600 | 23,500 |\n| Pass@1 | 81.1% | 89.4% |\n| Pass@3 | 91.7% | 93.3% |\n\nCodex reported around 74% fewer tokens while obtaining slightly better resolution rates. This difference appeared at two levels: Codex took around 30% fewer steps, and each step processed around 62% fewer tokens.\n\nThat doesn’t prove that GPT 5.6 Terra is intrinsically more efficient than Sonnet 5. The comparison mixes the model with the scaffold, the tokenizer, and everything else that differs between two harnesses.\n\nThe defensible conclusion is narrower:\n\nIn this harness, Codex with Terra started from a much more token efficient trajectory than Claude Code with Sonnet 5.\n\nThis provides a plausible explanation for the different Caveman results: Claude offered a long trajectory that the skill could truncate, while Codex already operated closer to a floor.\n\n### Caveman Needs Waste to Remove\n\nAn exploratory analysis inside the Codex results supports this interpretation.\n\nI divided the tasks into four groups according to their baseline token consumption. Caveman increased median consumption among the cheaper tasks and only produced savings among the more expensive ones:\n\n| Baseline consumption group | Median token change with Caveman |\n|---|---|\n| First quartile | 27% higher |\n| Second quartile | 11% higher |\n| Third quartile | 13% lower |\n| Fourth quartile | 9.5% lower |\n\nThis analysis shouldn’t be treated as causal. The groups mix datasets, contain relatively few tasks, and are defined using observed baseline consumption. Regression to the mean may explain part of the pattern.\n\nStill, the result is consistent with a floor effect. When the original trajectory is already compact, Caveman has little redundant work to remove. Its additional prompt and behavioral changes may cost more than they save. When a trajectory becomes unusually expensive, Caveman has more room to terminate it early.\n\nThe 3.9% aggregate reduction in Codex was therefore driven by the expensive tail. For the median Codex task, Caveman didn’t save tokens.\n\nThis suggests a better mental model:\n\nCaveman behaves like an indirect effort limiter whose effectiveness depends on how inefficient the baseline agent already is.\n\n## Quality\n\nCaveman didn’t improve aggregate Pass@1 or Pass@3 in any of the main comparisons.\n\nFor Claude:\n\n```\nPass@1: 81.1% → 80.0%\nPass@3: 91.7% → 86.7%\n```\n\nFor Codex:\n\n```\nPass@1: 89.4% → 85.0%\nPass@3: 93.3% → 90.0%\n```\n\nThe direction is consistent, but the experiment isn’t large enough to establish the magnitude precisely.\n\nBootstrap confidence intervals computed at the task level all crossed zero:\n\n| Comparison | Pass@3 change | 95% confidence interval |\n|---|---|---|\n| Claude baseline to Caveman | 5.0 points lower | 11.7 lower to 1.7 higher |\n| Claude baseline to medium | 5.0 points lower | 11.7 lower to 1.7 higher |\n| Claude medium to Caveman | no change | 8.3 lower to 8.3 higher |\n| Codex baseline to Caveman | 3.3 points lower | 10.0 lower to 3.3 higher |\n\nWe found no evidence that Caveman improves task resolution. Every aggregate quality comparison moved in the negative direction, but the experiment was not large enough to estimate effects of a few percentage points precisely.\n\nThere is one statistically detectable behavioral effect. Codex reasoning tokens fell by 14%, with a 95% confidence interval from 4.6% to 20% lower. Caveman doesn’t leave reasoning untouched, whatever the README says.\n\n## Other Evaluations\n\nA [JetBrains evaluation](https://blog.jetbrains.com/ai/2026/07/speak-to-ai-agents-like-cavemen-tosave-tokens/) tested Caveman on Claude Code using Harbor and SkillsBench.\n\nIt measured an 8.5% reduction in output tokens, far below the advertised 65%, with no detectable quality difference. Its first ten task evaluation had suggested savings near 30%; the number fell after expanding the benchmark. Its explanation was structural: coding agent output is dominated by code, diffs, tool invocations, and exact strings that Caveman can’t safely compress.\n\nThat result is compatible with this experiment, but the medium effort arm adds an important control. Even when Caveman reduces a trajectory without producing a detectable quality loss, it may still be an inefficient way to reach that operating point.\n\nOther evaluations tested an even simpler question: does Caveman outperform asking the model to “be brief”?\n\n[Max Taylor](https://www.maxtaylor.me/articles/i-benchmarked-caveman-against-two-words) evaluated 24 prompts with prompt specific quality rubrics and found Caveman in roughly the same token and quality range as a two word concision instruction.\n\n[Alex Rios](https://alexriosme.substack.com/p/there-is-a-reason-that-caveman-is) described the problem as the *Terse Control Fallacy*: comparing a sophisticated compression technique against silence instead of comparing it against the trivial alternative.\n\n## The Brevity Preprint\n\nThe Caveman README references the preprint [Brevity Constraints Reverse Performance Hierarchies in Language Models](https://arxiv.org/abs/2604.00025) as evidence that shorter answers can sometimes improve accuracy.\n\nThe paper evaluated 31 models on 1,485 questions and identified 115 cases where smaller models systematically outperformed larger ones. The authors associate part of this behavior with overthinking: larger models sometimes elaborate beyond what’s useful and introduce additional opportunities for error. On this selected subset, constraining responses improved large model accuracy from 40% to 67%.\n\nThis is a meaningful result, but also a targeted one. The brevity intervention was evaluated on problems already exhibiting unusual scaling behavior, rather than across the benchmark as a whole.\n\nThere’s also an important difference in settings. The study evaluates standalone questions: no tools, no repository, no decision about whether to keep exploring. A coding agent must repeatedly decide what to do next.\n\nThe paper doesn’t validate Caveman as an agent optimization. At most, it shows that excessive elaboration can hurt on a selected group of standalone reasoning questions.\n\n## Per-Dataset Breakdown\n\nThe aggregate results also hide substantial variation across task types:\n\n| Dataset and agent | Token change | Pass@1 change | Pass@3 change |\n|---|---|---|---|\n| SWE Bench with Claude | 28.8% lower | no change | 5.0 points lower |\n| USACO with Claude | 14.9% lower | no change | 3.3 points lower |\n| Terminal Bench with Claude | 4.1% higher | 6.7 points lower | 10.0 points lower |\n| SWE Bench with Codex | 5.2% lower | 11.7 points lower | 5.0 points lower |\n| Terminal Bench with Codex | 4.0% lower | 3.3 points lower | 10.0 points lower |\n| USACO with Codex | 1.5% higher | no change | no change |\n\nEach row compares that agent’s Caveman run against its own baseline, broken out by dataset.\n\nSWE Bench with Claude was Caveman’s strongest case. It preserved Pass@1 while reducing token usage by almost 29%.\n\nTerminal Bench with Claude was the opposite. Token usage increased while both success metrics fell, but that subset contained only 10 tasks, so a difference of 10 Pass@3 points represents a single task. One contrast in that subset produced a token ratio above 100% because of one unusual task.\n\nUSACO with Codex showed no quality degradation, but the baseline already reached 100% Pass@3. That subset had little room to distinguish the configurations.\n\nThe result depends on the agent, the model, and the dataset. Probably on the individual task too.\n\n## Limitations\n\nThis experiment has several limitations:\n\n-\nThe runs used Harbor as the evaluation harness. The reported cost is modeled rather than billed because they used subscription authentication rather than API keys.\n\n-\nTimed out runs didn’t report token or cost metrics. The estimated savings should therefore be treated as approximate rather than exact.\n\n-\nClaude and Codex receive Caveman through different integration mechanisms. That reflects how the skill is distributed for those tools, but it means cross agent comparisons mix the skill with its delivery path.\n\n-\nThe experiment needs more tasks to estimate small quality effects. Most tasks were solved in all three attempts or failed in all three, and only about ten of the sixty produced mixed results within a configuration, so three trials don’t provide three independent observations. With sixty tasks and few discordant pairs, differences around five percentage points remain uncertain.\n\n-\nThe medium arm controls for cost and effort, but not for style. A stronger future design would include a plain instruction such as “communicate briefly, but do not change your exploration or verification behavior.”\n\n## Should You Use Caveman?\n\nI’d use Caveman primarily as a style preference. If you find terse status updates and compact final answers easier to read, that alone may be a good reason to enable it.\n\nEnabling it, though, does more than change the wording. Caveman reaches into the trajectory itself: how much the agent reasons, how many tools it calls, and when it decides the task is done. If you’re worried about quality loss, raising reasoning effort may compensate, but that spends back the tokens Caveman saved.\n\nIf the goal is reducing cost, I’d prefer explicit controls such as reasoning effort or context management. These mechanisms target resource usage directly and are easier to evaluate.\n\nCaveman makes an agent more concise. Making it more efficient is a different claim, and this experiment doesn’t support it.\n\n## The Broader Lesson\n\nThe most interesting result isn’t specific to Caveman.\n\nA system prompt is part of an agent’s policy.\n\nInstructions about tone, confidence, or verbosity may look cosmetic, but they can influence more than the text an agent produces. Because the agent repeatedly conditions its next action on those instructions, a request for concision can also affect how long it explores, how many tools it uses, and when it decides that the task is finished.\n\nThat’s what happened here. Caveman did save tokens in Claude Code, but most of the reduction came from shorter trajectories. Tokens per step barely moved. Codex, which already started from a shorter and cheaper trajectory, had much less room for that effect.\n\nThis is also why explicit controls matter. When the goal is to trade reasoning effort for cost, a native effort setting targets that tradeoff directly. In this experiment, medium reasoning reached the same Pass@3 as Caveman while using fewer tokens and costing less.\n\nThe broader lesson is simple: instructions that look like presentation choices can become behavioral interventions once they’re placed inside an agent loop.\n\n## Future Work\n\nThere are several ways to extend this experiment.\n\nThe first is simply scale. Sixty tasks were enough to reveal clear behavioral differences, but not enough to estimate small changes in resolution rate precisely. A larger evaluation would reduce that uncertainty and make dataset specific effects easier to distinguish. It would also be interesting to include newer and harder benchmarks such as DeepSWE.\n\nThe second is model diversity. This experiment only covers Claude Code and Codex with one model family each. Repeating it with models from other families, such as DeepSeek, Kimi, or Muse Spark, would help determine whether Caveman’s effects depend on the underlying model. The large difference observed between Claude Code and Codex already suggests that its impact may depend strongly on both the model and the agent scaffold.\n\nThe most important control, however, would be a minimal terse control. Just something like `be concise`\n\nor `be brief`\n\nadded to the system prompt. A stronger follow up experiment would therefore compare baseline, a minimal terse control, Caveman, and native effort controls across more tasks and model families. That would make it possible to separate style, effort, and agent behavior much more cleanly.\n\n## References\n\nCaveman repository: [https://github.com/JuliusBrussee/caveman](https://github.com/JuliusBrussee/caveman)\n\nCaveman evaluation directory: [https://github.com/JuliusBrussee/caveman/tree/main/evals](https://github.com/JuliusBrussee/caveman/tree/main/evals)\n\nJetBrains evaluation: [https://blog.jetbrains.com/ai/2026/07/speak-to-ai-agents-like-cavemen-tosave-tokens/](https://blog.jetbrains.com/ai/2026/07/speak-to-ai-agents-like-cavemen-tosave-tokens/)\n\nMax Taylor, *I benchmarked caveman against two words*: [https://www.maxtaylor.me/articles/i-benchmarked-caveman-against-two-words](https://www.maxtaylor.me/articles/i-benchmarked-caveman-against-two-words)\n\nAlex Rios, *There is a reason that caveman is a character from the past*: [https://alexriosme.substack.com/p/there-is-a-reason-that-caveman-is](https://alexriosme.substack.com/p/there-is-a-reason-that-caveman-is)\n\n*Brevity Constraints Reverse Performance Hierarchies in Language Models*: [https://arxiv.org/abs/2604.00025](https://arxiv.org/abs/2604.00025)\n\n*How Do AI Agents Spend Your Money? Analyzing and Predicting Token Consumption in Agentic Coding Tasks*: [https://arxiv.org/abs/2604.22750](https://arxiv.org/abs/2604.22750)\n\n-\nThe prose estimate converts visible agent text into an approximate token count.\n\n[↩](#fnref:prose)\n\n## Enjoy Reading This Article?\n\nHere are some more articles you might like to read next:\n\n[Training with scikit-learn, Deploying with the Right Runtime](/blog/2026/tree-compiler/)\n\n[Amazon S3 Vectors: What It Is, Where It Fits, and the Gotchas Nobody Tells You](/blog/2025/s3-vectors/)\n\n[From scikit-learn to Faiss: Migrating PCA for Scalable Vector Search](/blog/2025/sklearn-faiss/)\n\n[How to Start a Machine Learning Project Before Starting a Machine Learning Project](/blog/2024/start-ml-project/)\n\n[DVC + Many Files: A Strategy for Efficient Large Dataset Management](/blog/2024/dvc-fix/)", "url": "https://wpnews.pro/news/caveman-saves-tokens-by-doing-less-not-just-saying-less", "canonical_source": "https://stuckinalocalminima.com/blog/2026/caveman/", "published_at": "2026-08-18 20:46:20+00:00", "updated_at": "2026-08-18 21:11:40.425476+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-tools", "large-language-models", "developer-tools"], "entities": ["Caveman", "Claude Code", "Codex CLI", "SWE Bench Verified", "Terminal Bench 2", "USACO", "Julius Brussee", "Sonnet 5"], "alternates": {"html": "https://wpnews.pro/news/caveman-saves-tokens-by-doing-less-not-just-saying-less", "markdown": "https://wpnews.pro/news/caveman-saves-tokens-by-doing-less-not-just-saying-less.md", "text": "https://wpnews.pro/news/caveman-saves-tokens-by-doing-less-not-just-saying-less.txt", "jsonld": "https://wpnews.pro/news/caveman-saves-tokens-by-doing-less-not-just-saying-less.jsonld"}}