Caveman Saves Tokens by Doing Less, Not Just Saying Less

wpnews.pro

Caveman promises an appealing optimization for coding agents:

Same answers, 65% fewer output tokens. Brain still big. Mouth small.

The idea is simple. Models often produce unnecessary explanations, narrate their tool calls, and pad conclusions with filler. Remove that verbosity and the agent should become faster and cheaper without becoming less capable.

Coding agents can consume a surprising number of tokens while inspecting repositories, running commands, and correcting their own mistakes. A small skill that removes wasted language sounds like an easy efficiency win. But a coding agent operates through a trajectory: call a tool, interpret the result, and decide whether to continue. An instruction that changes how the agent communicates can also change how it acts.

To measure that effect, I ran 60 tasks across Claude Code and Codex CLI using SWE Bench Verified, Terminal Bench 2, and USACO tasks. Caveman reduced Claude Code token usage by 18% and its reported cost by 25%, but in Codex token usage fell by only 3.9%.

The most important result was not how many tokens disappeared, but where they disappeared from. With Caveman enabled, Claude Code performed 17% fewer steps and made almost 20% fewer tool calls, and in Codex hidden reasoning tokens fell by 14%. Caveman didn’t merely make the agents speak less. It changed how much work they performed.

And if the skill mostly works by making the agent do less, adjusting the reasoning effort might be a better choice. Claude Code exposes reasoning effort as a native setting, and in this experiment medium effort reached roughly the same Pass@3 as Caveman with high effort while costing about 20% less. A style instruction that arrives at a similar operating point indirectly is harder to justify when a direct knob is one flag away.

What Caveman Promises #

Caveman is a skill for Claude Code, Codex, Gemini, and other coding agents. It instructs the model to avoid narration, drop hedging, and communicate in compact fragments.

Its README summarizes the intended behavior with a memorable distinction:

Brain still big. Mouth small.

The main benchmark compares ten standalone answers, including prompts such as explaining Git rebase, React rerenders, and Docker multistage builds. The reported average falls from about 1,200 output tokens to under 300, the headline reduction of 65%.

That result is plausible for those prompts. They ask for prose, and Caveman aggressively compresses prose. The problem is the extrapolation.

A single conversational answer and a repository level coding task have very different token distributions. In a coding agent, useful output is source code, patches, and exact error messages. Caveman leaves much of that unchanged.

The project itself acknowledges this distinction. Its documentation warns that Caveman only targets output tokens, adds input tokens of its own, and may become net negative when the original model is already concise. It also describes the honest evaluation control as Caveman versus a simple terse instruction, rather than Caveman versus an unconstrained baseline.

Those caveats are good. Caveman is free, open source, and doesn’t collect telemetry. The issue is that the headline claims are much broader than the evidence supporting them.

The Experiment #

I evaluated five agent configurations:

Configuration	Agent	Treatment
`claude-baseline`	Claude Code with Sonnet 5	High effort
`claude-caveman`	Claude Code with Sonnet 5	Caveman injected at session start (high effort)
`claude-medium`	Claude Code with Sonnet 5	Medium reasoning effort
`codex-baseline`	Codex CLI with GPT 5.6 Terra	High effort
`codex-caveman`	Codex CLI with GPT 5.6 Terra	Caveman forcibly activated (high effort)

The benchmark contained 60 tasks: 20 from SWE Bench Verified, 30 from USACO, and 10 from Terminal Bench 2. Each task was executed three times under every configuration, for 900 trials in total.

The claude-medium

configuration is particularly important. Comparing only baseline against Caveman mixes together two possible effects:

Caveman may compress communication.
Caveman may cause the model to spend less effort.

The medium effort arm provides a native way to reduce model effort without installing Caveman. It lets us ask whether the skill offers something beyond indirectly making the agent do less work.

Results #

Claude Code showed substantial savings:

Metric	Baseline	Caveman	Change
Total tokens	1,595,199	1,302,271	18.4% lower
Output tokens	19,860	16,059	19.1% lower
Reported cost	$1.003	$0.758	24.5% lower
Steps	25.5	21.2	17.0% lower
Tool calls	24.6	19.7	19.7% lower
Agent prose characters	2,956	1,741	41.1% lower
Pass@1	81.1%	80.0%	1.1 points lower
Pass@3	91.7%	86.7%	5.0 points lower

Codex showed a much smaller reduction:

Metric	Baseline	Caveman	Change
Total tokens	418,852	402,402	3.9% lower
Output tokens	5,252	4,757	9.4% lower
Reasoning tokens	2,542	2,200	13.5% lower
Reported cost	$0.257	$0.249	3.2% lower
Steps	17.8	17.5	1.8% lower
Tool calls	11.8	11.5	2.7% lower
Pass@1	89.4%	85.0%	4.4 points lower
Pass@3	93.3%	90.0%	3.3 points lower

If we stop here, Caveman looks like a reasonable tradeoff for Claude and a poor one for Codex. Claude saves approximately one quarter of its modeled cost while losing little Pass@1, while Codex gives up considerably more performance for a 3.2% cost reduction.

But these aggregated numbers hide the most important part of the experiment.

The Wrong Denominator #

Caveman promotes a reduction in output tokens. Coding agent cost, however, isn’t dominated by visible prose.

For Claude Code, the baseline token distribution looked like this:

Quantity	Share
Cache hit rate	97.8%
Output tokens as a share of all tokens	1.24%
Visible prose as a share of output tokens	3.7%
Visible prose as a share of all tokens	0.046%

Directly generated prose accounts for only about 0.05% of Claude’s reported session token volume. 1 A 41% reduction therefore removes only about 0.02% of the total tokens directly.

Shorter replies also make subsequent contexts slightly smaller, so the full mechanical saving from prose compression is larger than 0.02%. But even a deliberately generous fixed trajectory estimate puts that effect well below 1% of total token volume. Most output tokens are code, diffs, and commands that must remain exact. That’s nowhere near the observed 18% reduction.

The missing savings have to come from somewhere else. The trajectory data provides the answer: with Caveman, Claude took 17% fewer steps and made almost 20% fewer tool calls. Each removed turn avoided processing a large context again.

The expensive part of an agent is taking another turn.

Decomposing the Reduction #

The total token count can be decomposed approximately into two factors:

total tokens = number of steps × tokens per step

Applying that decomposition gives the following:

Comparison	Total tokens	Steps	Tokens per step
Codex baseline to Caveman	3.9% lower	1.8% lower	2.2% lower
Claude baseline to Caveman	18.4% lower	17.0% lower	1.7% lower
Claude baseline to medium	37.1% lower	20.7% lower	20.6% lower

For Claude, approximately 92% of the token reduction is explained by taking fewer steps. Tokens per step barely changed.

Caveman certainly changed the writing style. Visible prose fell by 41%, and prose per step also decreased. But this compressed prose was too small to explain the session level savings. The savings appeared because the model ended its trajectory earlier.

That interpretation is also visible in the tool usage:

Metric	Claude baseline to Caveman
Steps	17.0% lower
Tool calls	19.7% lower
Shell calls	19.0% lower
Agent execution time	15.6% lower

A style instruction propagated into the action policy of the agent. This doesn’t automatically mean that the removed steps were useful. Agents can overinspect repositories, repeat tests unnecessarily, and continue reasoning after they already have a valid solution. Some trajectories should be shorter.

But it does mean that Caveman shrinks the brain along with the mouth.

The Control That Changes the Conclusion #

The strongest argument against using Caveman isn’t the small drop in quality. It’s the claude-medium

control.

Both configurations reached exactly the same Pass@3:

Claude Caveman: 86.7%
Claude medium:  86.7%

But Caveman used 30% more tokens and cost 23% more than medium.

The chart makes the relevant comparison visible. On this plane, a configuration is dominated when another configuration achieves at least the same quality at a lower cost. Claude medium sits directly to the left of Claude Caveman: same Pass@3, lower cost.

The difference remains after accounting for successful trials:

Metric	Medium	Caveman	Caveman overhead
Pass@1	77.2%	80.0%	2.8 points higher
Pass@3	86.7%	86.7%	no change
Tokens per resolved trial	1.30 million	1.63 million	25.3% more
Cost per resolved trial	$0.749	$0.874	16.6% more

The quality comparison between Caveman and medium is inconclusive at Pass@1: Caveman was 2.8 points higher, but the difference was not statistically significant. The efficiency comparison is much clearer. Both configurations reached the same 87% Pass@3, yet Caveman consumed 25% more tokens per resolved trial and cost 17% more.

This comparison changes the practical recommendation. Caveman can reduce resource use, but it does so by indirectly altering the agent’s trajectory. When Claude already provides a direct control for effort, using a global style instruction to reach a similar operating point is difficult to justify.

Why Codex Saved Less #

Caveman reduced Claude token usage by 18%, but reduced Codex usage by only 3.9%. One possible explanation is that Codex had much less waste available to remove.

Before applying Caveman, the Codex and Terra combination already used substantially shorter and cheaper trajectories:

Baseline metric	Claude Code with Sonnet 5	Codex with Terra
Tokens per trial	1,595,199	418,852
Steps per trial	25.5	17.8
Tokens per step	62,600	23,500
Pass@1	81.1%	89.4%
Pass@3	91.7%	93.3%

Codex reported around 74% fewer tokens while obtaining slightly better resolution rates. This difference appeared at two levels: Codex took around 30% fewer steps, and each step processed around 62% fewer tokens.

That doesn’t prove that GPT 5.6 Terra is intrinsically more efficient than Sonnet 5. The comparison mixes the model with the scaffold, the tokenizer, and everything else that differs between two harnesses.

The defensible conclusion is narrower:

In this harness, Codex with Terra started from a much more token efficient trajectory than Claude Code with Sonnet 5.

This provides a plausible explanation for the different Caveman results: Claude offered a long trajectory that the skill could truncate, while Codex already operated closer to a floor.

Caveman Needs Waste to Remove

An exploratory analysis inside the Codex results supports this interpretation.

I divided the tasks into four groups according to their baseline token consumption. Caveman increased median consumption among the cheaper tasks and only produced savings among the more expensive ones:

Baseline consumption group	Median token change with Caveman
First quartile	27% higher
Second quartile	11% higher
Third quartile	13% lower
Fourth quartile	9.5% lower

This analysis shouldn’t be treated as causal. The groups mix datasets, contain relatively few tasks, and are defined using observed baseline consumption. Regression to the mean may explain part of the pattern.

Still, the result is consistent with a floor effect. When the original trajectory is already compact, Caveman has little redundant work to remove. Its additional prompt and behavioral changes may cost more than they save. When a trajectory becomes unusually expensive, Caveman has more room to terminate it early.

The 3.9% aggregate reduction in Codex was therefore driven by the expensive tail. For the median Codex task, Caveman didn’t save tokens.

This suggests a better mental model:

Caveman behaves like an indirect effort limiter whose effectiveness depends on how inefficient the baseline agent already is.

Quality #

Caveman didn’t improve aggregate Pass@1 or Pass@3 in any of the main comparisons.

For Claude:

Pass@1: 81.1% → 80.0%
Pass@3: 91.7% → 86.7%

For Codex:

Pass@1: 89.4% → 85.0%
Pass@3: 93.3% → 90.0%

The direction is consistent, but the experiment isn’t large enough to establish the magnitude precisely.

Bootstrap confidence intervals computed at the task level all crossed zero:

Comparison	Pass@3 change	95% confidence interval
Claude baseline to Caveman	5.0 points lower	11.7 lower to 1.7 higher
Claude baseline to medium	5.0 points lower	11.7 lower to 1.7 higher
Claude medium to Caveman	no change	8.3 lower to 8.3 higher
Codex baseline to Caveman	3.3 points lower	10.0 lower to 3.3 higher

We found no evidence that Caveman improves task resolution. Every aggregate quality comparison moved in the negative direction, but the experiment was not large enough to estimate effects of a few percentage points precisely.

There is one statistically detectable behavioral effect. Codex reasoning tokens fell by 14%, with a 95% confidence interval from 4.6% to 20% lower. Caveman doesn’t leave reasoning untouched, whatever the README says.

Other Evaluations #

A JetBrains evaluation tested Caveman on Claude Code using Harbor and SkillsBench.

It measured an 8.5% reduction in output tokens, far below the advertised 65%, with no detectable quality difference. Its first ten task evaluation had suggested savings near 30%; the number fell after expanding the benchmark. Its explanation was structural: coding agent output is dominated by code, diffs, tool invocations, and exact strings that Caveman can’t safely compress.

That result is compatible with this experiment, but the medium effort arm adds an important control. Even when Caveman reduces a trajectory without producing a detectable quality loss, it may still be an inefficient way to reach that operating point.

Other evaluations tested an even simpler question: does Caveman outperform asking the model to “be brief”?

Max Taylor evaluated 24 prompts with prompt specific quality rubrics and found Caveman in roughly the same token and quality range as a two word concision instruction.

Alex Rios described the problem as the Terse Control Fallacy: comparing a sophisticated compression technique against silence instead of comparing it against the trivial alternative.

The Brevity Preprint #

The Caveman README references the preprint Brevity Constraints Reverse Performance Hierarchies in Language Models as evidence that shorter answers can sometimes improve accuracy.

The paper evaluated 31 models on 1,485 questions and identified 115 cases where smaller models systematically outperformed larger ones. The authors associate part of this behavior with overthinking: larger models sometimes elaborate beyond what’s useful and introduce additional opportunities for error. On this selected subset, constraining responses improved large model accuracy from 40% to 67%.

This is a meaningful result, but also a targeted one. The brevity intervention was evaluated on problems already exhibiting unusual scaling behavior, rather than across the benchmark as a whole.

There’s also an important difference in settings. The study evaluates standalone questions: no tools, no repository, no decision about whether to keep exploring. A coding agent must repeatedly decide what to do next.

The paper doesn’t validate Caveman as an agent optimization. At most, it shows that excessive elaboration can hurt on a selected group of standalone reasoning questions.

Per-Dataset Breakdown #

The aggregate results also hide substantial variation across task types:

Dataset and agent	Token change	Pass@1 change	Pass@3 change
SWE Bench with Claude	28.8% lower	no change	5.0 points lower
USACO with Claude	14.9% lower	no change	3.3 points lower
Terminal Bench with Claude	4.1% higher	6.7 points lower	10.0 points lower
SWE Bench with Codex	5.2% lower	11.7 points lower	5.0 points lower
Terminal Bench with Codex	4.0% lower	3.3 points lower	10.0 points lower
USACO with Codex	1.5% higher	no change	no change

Each row compares that agent’s Caveman run against its own baseline, broken out by dataset.

SWE Bench with Claude was Caveman’s strongest case. It preserved Pass@1 while reducing token usage by almost 29%.

Terminal Bench with Claude was the opposite. Token usage increased while both success metrics fell, but that subset contained only 10 tasks, so a difference of 10 Pass@3 points represents a single task. One contrast in that subset produced a token ratio above 100% because of one unusual task.

USACO with Codex showed no quality degradation, but the baseline already reached 100% Pass@3. That subset had little room to distinguish the configurations.

The result depends on the agent, the model, and the dataset. Probably on the individual task too.

Limitations #

This experiment has several limitations:

The runs used Harbor as the evaluation harness. The reported cost is modeled rather than billed because they used subscription authentication rather than API keys.

Timed out runs didn’t report token or cost metrics. The estimated savings should therefore be treated as approximate rather than exact.

Claude and Codex receive Caveman through different integration mechanisms. That reflects how the skill is distributed for those tools, but it means cross agent comparisons mix the skill with its delivery path.

The experiment needs more tasks to estimate small quality effects. Most tasks were solved in all three attempts or failed in all three, and only about ten of the sixty produced mixed results within a configuration, so three trials don’t provide three independent observations. With sixty tasks and few discordant pairs, differences around five percentage points remain uncertain.

The medium arm controls for cost and effort, but not for style. A stronger future design would include a plain instruction such as “communicate briefly, but do not change your exploration or verification behavior.”

Should You Use Caveman? #

I’d use Caveman primarily as a style preference. If you find terse status updates and compact final answers easier to read, that alone may be a good reason to enable it.

Enabling it, though, does more than change the wording. Caveman reaches into the trajectory itself: how much the agent reasons, how many tools it calls, and when it decides the task is done. If you’re worried about quality loss, raising reasoning effort may compensate, but that spends back the tokens Caveman saved.

If the goal is reducing cost, I’d prefer explicit controls such as reasoning effort or context management. These mechanisms target resource usage directly and are easier to evaluate.

Caveman makes an agent more concise. Making it more efficient is a different claim, and this experiment doesn’t support it.

The Broader Lesson #

The most interesting result isn’t specific to Caveman.

A system prompt is part of an agent’s policy.

Instructions about tone, confidence, or verbosity may look cosmetic, but they can influence more than the text an agent produces. Because the agent repeatedly conditions its next action on those instructions, a request for concision can also affect how long it explores, how many tools it uses, and when it decides that the task is finished.

That’s what happened here. Caveman did save tokens in Claude Code, but most of the reduction came from shorter trajectories. Tokens per step barely moved. Codex, which already started from a shorter and cheaper trajectory, had much less room for that effect.

This is also why explicit controls matter. When the goal is to trade reasoning effort for cost, a native effort setting targets that tradeoff directly. In this experiment, medium reasoning reached the same Pass@3 as Caveman while using fewer tokens and costing less.

The broader lesson is simple: instructions that look like presentation choices can become behavioral interventions once they’re placed inside an agent loop.

Future Work #

There are several ways to extend this experiment.

The first is simply scale. Sixty tasks were enough to reveal clear behavioral differences, but not enough to estimate small changes in resolution rate precisely. A larger evaluation would reduce that uncertainty and make dataset specific effects easier to distinguish. It would also be interesting to include newer and harder benchmarks such as DeepSWE.

The second is model diversity. This experiment only covers Claude Code and Codex with one model family each. Repeating it with models from other families, such as DeepSeek, Kimi, or Muse Spark, would help determine whether Caveman’s effects depend on the underlying model. The large difference observed between Claude Code and Codex already suggests that its impact may depend strongly on both the model and the agent scaffold.

The most important control, however, would be a minimal terse control. Just something like be concise

or be brief

added to the system prompt. A stronger follow up experiment would therefore compare baseline, a minimal terse control, Caveman, and native effort controls across more tasks and model families. That would make it possible to separate style, effort, and agent behavior much more cleanly.