How I Cut My Anthropic API Bill by 50% With a Local Python Tool

A developer built a local Python CLI tool called `ai-cost-optimizer` that reduced their Anthropic API bill by 50% by implementing a semantic cache, prompt compressor, and model router. The tool intercepts API requests to store and retrieve cached responses via vector embeddings, compress bloated prompts by discarding irrelevant sentences, and automatically route queries to the cheapest suitable model (e.g., Haiku over Opus). After two weeks of development use, the tool saved an estimated $0.000300 from cache hits alone across 23 API calls totaling $0.16.

My Anthropic bill doubled two months in a row. Not because I was building something bigger — because I kept asking the same questions, sending bloated prompts, and defaulting to Sonnet for tasks that Haiku could handle. I built a tool to fix it. Here's how it works. AI API costs compound fast for three reasons. First, if you're iterating on a project, you ask similar questions repeatedly — "how does X work," "what's wrong with this code" — and pay full price every time. Second, prompts accumulate context: documentation snippets, error traces, boilerplate instructions that add hundreds of tokens but contribute nothing to the answer. Third, most people just use whatever model they defaulted to first. Claude Opus at $15/1M input tokens for a query that Haiku could answer for $1/1M is a 15x cost multiplier on every single call. I built ai-cost-optimizer — a local CLI that sits between your terminal and the Anthropic API. It runs a semantic cache, a prompt compressor, and a model router on every request before anything hits the network. No cloud, no subscription, no data leaving your machine. Just a Python package you install once. The cache stores every response as a vector embedding. On each new request, it computes the embedding for your prompt and checks cosine similarity against everything stored. If similarity is above the threshold default: 0.80 , it returns the cached answer — no API call, zero cost. bash $ aiproxy ask "What is the capital of France?" Model claude-haiku-4-5-20251001 Cached No Input tokens 18 Output tokens 9 Cost $0.000063 Cache saved $0.000000 cached for next time Total saved $0.000000 $ aiproxy ask "Capital of France?" Model claude-haiku-4-5-20251001 Cached Yes Input tokens 0 Output tokens 0 Cost $0.000000 Cache saved $0.000063 Total saved $0.000063 "Capital of France?" is semantically identical to the first query. Cache hit. The API never sees it. The cache uses sentence-transformers/all-MiniLM-L6-v2 for embeddings 80 MB, runs entirely in-process and usearch for fast ANN lookup. Cold load is ~1.5 seconds on first run; subsequent queries are sub-100ms. Long prompts are expensive not because they're long, but because most of that length is filler. The compressor uses BM25 to score each sentence by relevance to the query, keeps the top-scoring sentences, and discards the rest. In plain terms: it reads your prompt, figures out which sentences actually relate to what you're asking, and throws out the ones that don't. No summarization, no LLM — pure lexical scoring, deterministic, fast. Real example from a documentation query: Original prompt: 370 tokens $0.001110 at Sonnet pricing Compressed: 61 tokens $0.000183 at Sonnet pricing Tokens saved: 309 tokens 83% reduction The threshold for compression is configurable MAX PROMPT TOKENS=500 in .env . Prompts under that limit are sent as-is. The router classifies each prompt and picks the cheapest model that can handle it. The logic is rule-based: token count, keyword signals for complexity, and a few heuristics for code vs. prose vs. reasoning tasks. | Query | Routed to | Input cost per 1M tokens | |---|---|---| | "What is 2+2?" | Haiku | $1.00 | | "Explain binary search trees" | Sonnet | $3.00 | | "Review this system architecture" | Opus | $15.00 | You can override the router by passing --model explicitly. But if you don't, it defaults to the cheapest model that fits the task, and in practice that means Haiku handles the majority of short factual queries. After two weeks of normal development use — asking questions about code, debugging errors, generating short snippets: bash $ aiproxy stats Cache entries 14 Cache hits 3 Estimated savings $0.000300 Total API calls 23 Total cost $0.160000 The compression savings don't show in stats yet — that's a known gap I'm fixing next. git clone https://github.com/desaikat/ai-cost-optimizer.git cd ai-cost-optimizer python -m venv .venv source .venv/Scripts/activate Windows: .venv\Scripts\activate pip install -e . cp .env.example .env Add your ANTHROPIC API KEY to .env aiproxy ask "What is the difference between a list and a tuple in Python?" There's also a Streamlit dashboard aiproxy-dashboard that shows cumulative spend, cache hit rate, model distribution, and compression savings over time. .exe Repo: github.com/desaikat/ai-cost-optimizer https://github.com/desaikat/ai-cost-optimizer Two things I'm actively unsure about and would value input on: Open issues and PRs welcome.