A pilot benchmark, a $5.60 receipt, and a 97%-savings number that was actually a silent failure.
This started from a plain problem: I kept hitting token limits at work. I was using Claude Code for real engineering work, and I was burning through budget faster than I wanted. The obvious question was: can I cut that down without hurting the quality of what the agent produces? Not "just use a cheaper model and hope." Something more deliberate — a set of tools that each attack a different part of the token bill. How much context gets read. How much gets re-read. How verbose the agent's own output is. How it finds its way around a codebase in the first place.
That question turned into a side project: [ token-optimization-stack](https://github.com/shreyasht/token-optimization-stack), a public repo with setup docs for tools that reduce token spend. And
`token-stack-benchmarks`
I'm writing this up because the project ended, a few weeks in, in a place I didn't expect. Not with a working stack and a savings number. Instead, with proof that the token-savings numbers I was looking at were actively misleading — and a cost problem that made the whole thing stop making sense before I could even publish a result. I think both of those are more useful to write about than a clean win would have been.
The first version of the stack had five tools in it:
Two of those five didn't survive contact with a real benchmark.
Headroom didn't do what its own docs implied. The only way to register it without wrapping the whole claude
command in a separate launcher is headroom init claude
. That just adds an on-demand MCP tool — something the agent can call, not something that compresses context automatically. Running headroom doctor
confirmed nothing was actually being routed through it unless you also ran a separate proxy process with an ANTHROPIC_BASE_URL
override. That's a much heavier setup than "no behavioral changes" suggested. On top of that, its mcp serve
command crashed against a current MCP SDK — it needed an old, pinned mcp<2
dependency just to start.
LiteLLM had a different problem. Its usage-based routing is a load-balancing strategy across provider endpoints, not the complexity-based, per-task routing I actually wanted. And mechanically, Claude Code sends one fixed model for an entire session — there's no way to swap models mid-task based on how hard a step is. The tool I wanted didn't exist yet, at least not in this shape.
I removed both rather than keep them in as unverified claims. What was left — Graphify, Serena, a compression/caching layer called LeanCTX, and Caveman — became the actual stack I tested.
Not because the idea was wrong. That came later. The experiment itself stopped making financial sense.
The rigorous version of this test — real tasks from SWE-bench Verified and Multi-SWE-bench, sixteen repos, five versions of the stack, three repeats each — works out to about 4,800 agent runs. I never got close to that. Instead I ran a much cheaper pilot: 31 tasks, 2 versions of the stack, one repeat, on the cheapest model I had (claude-haiku-4-5
, medium effort). Even that only partly finished — 11 of 31 task pairs — and it already cost about $5.60 in raw API spend. That's before EC2 costs, Docker builds, or the multi-day slog of getting this running cleanly on both EC2 and an Apple Silicon Mac.
Scale that same per-run cost up to the full test and you pass $1,200 in API spend — on the cheapest model available, before a single result is even trustworthy. Sonnet costs 2x what Haiku does, on both input and output tokens ($2/$10 per million tokens vs. Haiku's $1/$5). So switching to it to get a trustworthy result would push the same test past $2,400. And Haiku wasn't trustworthy: it got zero correct fixes on the Java tasks, and it broke two of the three Python tasks the plain baseline had already solved.
Is this savings number — or this correctness failure — actually about the stack? Or is it about the fact that I'm running everything on the cheapest model I could afford to run 4,800 times of?
I didn't have a good answer. That's where I stopped.
Even the partial pilot data was worth sharing, because it directly contradicts what a token-savings-only view would have told me. On the three Python tasks the plain baseline agent solved correctly, adding the full stack did this:
num_turns: 1
, no error, nothing left to score. The plain baseline took 30 turns on the exact same prompt and fixed the bug. Read only off the token dashboard, this run showed I'm not treating "2 of 3" as a rate. Three tasks is too small a sample to turn into a percentage. But something else holds, even at this size: the token numbers and the correctness numbers pointed in opposite directions, and the worst result in the batch produced the best-looking number. That doesn't need a bigger sample to be true — it happened, on a real task, and it's exactly the kind of failure a token-savings-only report can't catch.
I'd also expect this to get worse on a weak model, not better. Haiku has less room to recover once a terser style takes away its ability to push back or think through whether a task is really ambiguous. A stronger model might ask the same question but keep working anyway — or not need to ask at all. I didn't get to test that. It's a specific, checkable prediction for whoever picks this up next, not just a guess.
Token and cost savings numbers, without a real correctness check against the actual test suite, aren't just incomplete — they can point in exactly the wrong direction. And the biggest, flashiest savings number is a plausible place for that to happen, not an unlikely one.
The fix is simple: report cost per solved task, not cost per task. Under that measure, the 97%-savings run isn't a win with an asterisk. It's infinitely expensive, because it solved zero tasks. That one change closes the trap — a dashboard built around it can't turn a silent failure into a headline number.
Pair that with something even cheaper to check: turn count. A run that takes 1 turn when the baseline took 30 is a giant red flag, one that no token dashboard shows on its own. And unlike correctness scoring, checking it costs nothing — no test suite, no scoring setup, no Docker. It's already sitting in the same log that produced the token numbers.
I'm not going to keep running this. Not because I think the question is answered — I just can't afford to answer it properly right now. If you want to take it further, both repos are public:
Contributions and forks are welcome. So are "here's why your pilot was wrong" pull requests. A few concrete places to start: