# Why your AI bill is bigger than it should be > Source: > Published: 2026-07-01 10:54:07+00:00 You have **1** article left to read this month before you need to [register](/register) a free LeadDev.com account. Estimated reading time: 13 minutes **Key takeaways:** - Most of what you send to LLMs is **unnecessary**, and you’re** paying for all of it**. One $287 AI bill led to a tool that saved users $700,000 in five months. **Token hygiene is the next engineering discipline**. Treat token budgets like compute credits and measure what a task actually needs, not what it consumes.- Providers **compress your data** but don’t pass**the savings on**: compressing context before it reaches them gives teams visibility into AI spend that providers have no incentive to offer. A $287 debugging session prompted one engineer to rethink how we feed data to [large language models (LLMs)](https://leaddev.com/leadership/llms-an-operators-view). The result has saved users an estimated $700,000 in five months. [Tejas Chopra](https://www.linkedin.com/in/chopratejas/) was debugging a Graphics Processing Unit (GPU) failure. Routine procedure for a [senior engineer](https://leaddev.com/management/addressing-skills-gaps-senior-engineers): pull the logs, ask Claude to identify the problem, get on with your day. When the answer came back, he noticed something odd. That single prompt had consumed his entire context window twice over. “I spent a lot of money just asking that one question,” he recalls. He wondered why. It turned out that the model had read the entire log file multiple times, processing everything before extracting the three lines that actually mattered. By the time Chopra added up his monthly bill, he was looking at $287 for personal project work. The fix was to rewrite the prompt to ignore INFO lines, and focus only on warnings and alerts. Response time improved and token cost dropped, but Chopra remained perturbed. “You cannot expect every developer to open their window and curate the prompts to match what they’re looking for,” he says. “People – or models – will blindly say, ‘I need to look at logs, I need to grab the logs.’” To address this, he wondered whether the process could be [automated](https://leaddev.com/velocity/automate-tasks-reclaim-your-time-2025). The result is [Headroom](https://github.com/chopratejas/headroom), an open-source context optimization layer for the LLM. On presenting the project at Linux Open Source Summit, Chopra found that the idea really resonated. “Simply put, many companies are struggling firstly to understand where the token spend is, and then optimize for it. Headroom, as an open-source project that can live on your machine, helps with both.” Before they stopped collecting the stats, Headroom had saved its users an estimated $700,000, and reclaimed 200 billion tokens, in just five months. This early success prompted Chopra to leave his senior engineering job and found [Headroom Labs](https://headroomlabs.ai), to explore the idea that most of what we’re sending to LLMs isn’t necessary. ## Your inbox, upgraded. Receive weekly engineering insights to level up your leadership approach. ## How Headroom’s compression works Chopra describes the compression pipeline as having evolved through three distinct stages, each building on the previous one. The first target was JavaScript Object Notation (JSON), since it is widely used and wasteful when tokenized naively. Whitespace, commas, quotation marks, and nested indentation all cost tokens, without adding semantic meaning. Headroom strips it and converts it to a compact representation that results in “30% savings instantly, without dropping any [data](https://leaddev.com/technical-direction/whatever-happened-big-data),” Chopra says. Headroom next looks for statistical similarity across values, and compresses accordingly. If 88 out of 90 values in an array fall between 0 and 1, and two are outliers at 99 and 100, you don’t need to transmit all 90 values. You transmit the outliers and a summary: “88 entries between 0 and 1.” The outliers are preserved exactly; the common cases become a single annotation. “That itself is valuable,” Chopra says. “You just need to keep one copy of statistically similar things and the delta.” Every compressed payload in Headroom is backed by a cache entry with a key, which is a composite of the session ID and a hash of the original data. Since the hash is based on content rather than context, Hash collisions won’t produce cross-session contamination. The full original payload lives in a local Redis or SQLite instance. Since context that was valid half an hour ago may not be valid now, the cache has a configurable Time To Live (TTL), defaulting to between five and 30 minutes for an individual developer. The expiry forces freshness, without requiring the developer to think about cache invalidation manually. For enterprise deployments, instead of a local Redis instance, the cache can live in a database such as RDS on AWS, Bigtable on GCP, or Postgres in a private cloud or local data center, depending on which service the organization already uses. Multiple developers working across multiple sessions can benefit from shared cache entries. A fetched Application Programming Interface (API) response that ten engineers hit on the same afternoon gets compressed and stored once, not ten times. The TTL settings become an organizational decision, configurable centrally. A risk with compression is that the model may need what you threw away. Chopra’s answer is to leave a tool call in the compressed output. When Headroom compresses a payload, it hashes the original and stores it locally. It then inserts a breadcrumb into the compressed version, which provides a tool definition that the model can call to retrieve the full original data, if it decides it needs it. If the model is intelligent enough to request more context, the mechanism exists; if it isn’t, nothing is wasted on sending data that the model would have ignored. “I rely on the intelligence of the models to do that,” Chopra says, “plus my own statistical analysis to compress the right stuff out.” Of course, if the retrieval step fires, it is a full extra tool call with its own latency, although Chopra says that happens in less than 1% of cases. The intent is that it should never be needed: the statistical compression should be conservative enough, and the models sufficiently intelligent, that the compressed version contains everything required to answer the prompt. There’s also a second-order latency effect: passing in fewer tokens means faster processing and a shorter response. On high-throughput workloads, the input compression savings partially offset the first-call overhead. ## More like this ## A different compressor for every context type Chopra suggests that the numbers work out more favorably than you might expect. This is because while the compression overhead is around 50 milliseconds, Time-To-First-Token (TTFT) from a cold [LLM](https://leaddev.com/velocity/rethinking-collaboration-llms-teams-and-cognitive-load) call is typically two to three seconds. Even so, Headroom is currently mid-migration from Python to Rust, specifically to save on latency. The compress-cache-retrieve approach is used for various other pieces of input context such as code, lock files, web pages, or plain text, but each requires a different compression strategy, so Headroom has a distinct compressor for each. Code compression uses the abstract syntax tree. Headroom can reason about the [code’s structure](https://leaddev.com/velocity/writing-code-was-never-the-bottleneck), and understand which functions are called and which aren’t. Lock files, which can be enormous and almost irrelevant to any given prompt, get their own treatment. Web pages such as documentation, API references, or Stack Overflow answers that may be fetched for context, are processed differently again. Then there’s unstructured flat text that doesn’t conform to any parseable structure. For this, Chopra trained a small open-source model from scratch. “It looks at every token in the text and either keeps it or drops it,” he notes. The training signal is based on determining, for each word in a document, whether removing it changes the semantic meaning of the surrounding text. Run that repeatedly across a large corpus and you can train a model that’s essentially learning a compression grammar for natural language. The model, called [Kompress Base](https://huggingface.co/chopratejas/kompress-v2-base), is open source and available on Hugging Face. Pass it a financial document today and it will compress it meaningfully, and the model can be further fine-tuned for a given domain. ## What Headroom doesn’t compress Since output tokens are typically priced at five times the input, the cost savings available for output compression are higher than for input. Headroom currently only compresses inputs, but output token compression is in active development, with pull requests (PRs) open at time of writing. Local file reads, which account for around 60% of the context in typical coding agent flows, are not compressed. Consequently, when an agent reads a source file, it may be looking for a specific line. Compress that file, and you risk dropping that line. The model then falls back to the retrieval tool, adds a round-trip, and the exercise has cost more than it saved. Instead, Headroom tries to reduce the surface area of what needs to be read in the first place. Tools like [Serena](https://github.com/oraios/serena) or [CodeMCP](https://github.com/ezyang/codemcp) build symbolic indexes and dependency graphs of a codebase. By integrating with them, Headroom can steer an agent toward reading the right five lines in a 100-line file. ## Learning from failure Another interesting feature of Headroom, called ‘learn,’ is a mechanism that mines historical agent sessions for repeated failures and writes corrections back into your CLAUDE.md, or equivalent, files. Since every developer interacts with AI agents differently, Chopra argues that systems should be curated per individual. Headroom reads the historical session data that [coding agents](https://leaddev.com/ai/best-ai-coding-assistants) leave behind, uses the model to extract recurring failure patterns, and proposes a correction. “You can only do that by learning from patterns of usage – from their system, via all your historical data.” The pattern it targets is common. An agent looks for Python at /usr/local/bin/python when the developer’s environment has it at /opt/homebrew/bin/python. It fails. The next session, it tries the same wrong path and fails again. Across ten sessions, a thousand tokens are spent on a mistake that could be fixed with one line in a config file. **Berlin** • **November 9 & 10, 2026** **AI. Burnout. Big decisions.** The pressure is real. Find what works at LeadDev Berlin. ## The integration problem The compression system in Headroom is technically impressive, but when I asked Chopra what the main challenge with building Headroom is, he cited integration. Every LLM provider has a different API dialect. Claude’s API differs from OpenAI’s. When you add routing layers (Bedrock, Vertex AI, Azure) those introduce their own variants on top. Furthermore, within a single model family, the API can change between versions in ways that aren’t always clearly documented. Running Claude directly, or via Bedrock or Vertex, each requires a substantially different integration path for what is notionally the same underlying model. On top of this, the plethora of [coding agents and tools](https://leaddev.com/ai/your-ai-coding-tools-buying-checklist-for-2026) makes the compatibility matrix even harder. “You have a multiplicative effect,” Chopra says. “We are now trying to balance that with open source.” Headroom claims first-class support for Claude and Codex; everything else is marked experimental, with the community filing tickets and contributing fixes as they find edge cases. ## Managing your AI bill with token hygiene There was a brief, deeply embarrassing fashion amongst the silicon valley tech bros for ‘[tokenmaxxing](https://leaddev.com/ai/tokenmaxxing-and-the-search-for-ai-metrics-that-matter),’ the deliberately wasteful practice of maximizing AI token consumption to inflate usage metrics or climb corporate leaderboards, rather than prioritizing business value. When management ties performance to raw token usage, employees game the system. Tech firms like Meta and Amazon, which [has now deprecated its KiroRank leaderboard](https://www.ft.com/content/b1a62a7f-6df5-4c90-94ce-64ce9c9961b6?syn-25a6b1a6=1), introduced internal dashboards tracking token consumption. Employees racked up billions of tokens by running agents or redundant tasks on loop, purely to secure titles like Token Legend, while ignoring both the carbon and financial costs. Even as this horrifying trend fades, something that remains common is context dumping – i.e. the practice of dropping entire codebases, full database schemas, complete API response payloads, and verbose log files into context windows, because the model can handle it and it saves the developer having to think. It is another architectural smell that will seem embarrassing in hindsight. We have conventions around code hygiene, database normalization, and bundle optimization. Token hygiene, which Chopra calls “being tokenwise,” is at an earlier stage, but he imagines a near-future where token budgets are a real resource, allocated per engineer and per project like compute credits: “Here is your 100K worth of tokens as a perk. Now as an engineer, you have to be tokenwise with that.” The longer-term version of this would be “outcome-based tokenization:” measuring not how many tokens a task consumed, but how many tokens were required to reach a correct and useful outcome, encouraging engineers to optimize for minimal token use. As open-source models close the gap on closed-source frontier models, intelligent routing between them – sending routine tasks to cheaper models, escalating to frontier models only when the task warrants it – becomes feasible. **What comes next?** Headroom Labs has closed a pre-seed round. Chopra’s CTO, [Devanshi Vyas](https://www.linkedin.com/in/devanshivyas/), is a two-time founder, and together they’re building the enterprise layer on top of the open-source core: a control plane that handles org-level cost attribution, shared caching infrastructure, configurable governance policies, and the observability tooling that engineering leaders need to understand where their AI spend is going. An obvious question here is whether Headroom Labs gets ‘[Sherlocked](https://techcrunch.com/2024/06/12/the-apps-that-apple-sherlocked-at-wwdc/)’ – i.e. made obsolete by an operating system or platform creator like Apple, who creates software that does the same thing. When pushed, Chopra’s view was: “I’m pretty sure they’re doing it, but will they pass the savings to you?” His analogy is that when you upload two identical photos to a cloud storage provider, you’re charged for two. Deduplication happens on the provider’s side, but billing doesn’t reflect it. The same logic applies to LLM providers. They almost certainly have KV cache optimizations, token-level deduplication, and infrastructure-level compression running internally. “They can compress it to death,” Chopra acknowledges, “but they have margins to keep, and lofty IPOs.” Headroom compresses the [data](https://leaddev.com/technical-direction/data-science-demystified) before it reaches the provider, regardless of what the provider does internally. It’s a neutral intervention that doesn’t depend on any provider choosing to behave differently. It also produces something providers can’t easily replicate: per-developer, per-session visibility into exactly where token spend is going and why. Aiming at coding agents is an obvious go-to-market strategy for Headroom Labs, since coding is somewhere that LLMs appear to have some early product market fit. The same compression logic extends to other AI modalities. Image compression is already partially implemented in Headroom, for instance. When a user sends an image to Claude with a question like, “Is there a television in this room?,” by default Claude processes the image at full resolution, paying full token cost for every pixel. However, answering whether a television is present doesn’t require a 4K image, as a substantially downsampled version conveys the same answer. Headroom can compress the image intelligently based on the accompanying prompt before it reaches the model, paying only for the resolution the question actually needs. Most voice agents run on a three-step pipeline: speech to text, LLM processing, text to speech. The middle step (the LLM interaction) is where token costs accumulate. For voice, the interesting variable isn’t cost, it’s latency. A voice agent that takes 300 milliseconds to respond sounds acceptable; one that takes 800 milliseconds sounds broken. Chopra argues that input compression can shave meaningful time off that middle step, because fewer tokens in means faster TTFT out. “A matter of 50 milliseconds can tell you whether it’s a human or an agent,” he says. ## The long-term vision The longer-term vision is larger still. Chopra describes Headroom’s ambition as becoming “the IO substrate for [agents](https://leaddev.com/technical-direction/how-to-prepare-for-ai-agents)” – a context intelligence layer that handles not just compression, but attribution, memory, observability, and security. There are other players in agent middleware, attacking reliability, identity, and governance from different angles, but Chopra thinks that starting with token compression gives Headroom an early advantage. The open-core model means that a single developer running headroom wrap claude on their laptop is fully supported. A company with 100 developers can’t manage that individually; they need the control plane outside, with centralized visibility and policy. Those are different products with different buyers, and Headroom is trying to serve both without compromising. Whether the future of AI infrastructure belongs to the monolithic providers, or neutral middleware like Headroom, remains to be seen. For now, Chopra’s upstream approach is proving that the best way to [manage soaring AI costs](https://leaddev.com/ai/how-to-justify-ai-investments) is to stop over-feeding the models. Just as web developers learned to minify JavaScript, AI engineers will need to adopt token hygiene as a core discipline. Chopra’s proxy is a pragmatic first step toward that future.