Kotro – I cut my Cursor API bill by 68% with a 15MB local proxy

Developer Kotro released a local proxy that cuts AI coding-agent API costs by up to 99% by intercepting LLM traffic, blocking prompt injection, and preserving upstream prefix caching. The single-binary tool sits between the agent and cloud providers, ensuring no third party sees the traffic, and reduced billed tokens by 68% in real-world use.

The local security and efficiency layer for MCP-native agentic AI — intercept streaming LLM traffic from OpenAI and Anthropic SDKs, block prompt injection from tool results, keep secrets off the wire, and cut token waste. One binary, no SaaS, no third party ever in the request path. As MCP-native agents Cursor, Claude Code, custom SDK clients pull in tool results from files, git, search, and external APIs, those results become the largest new attack surface in the stack — a single malicious tool result can hijack the agent's next action. Kotro sits between your agent and the cloud provider and enforces a security policy your LLM provider cannot: it sees every message, every tool result, and every secret before they leave your machine. Kotro is deliberately narrow: a single-binary, zero-dependency proxy for one developer's coding-agent traffic. It is not trying to be a team-wide LLM gateway — for that, the tools below are more mature and better fits. | Tool | Deployment | Best fit | |---|---|---| Kotro | Self-hosted, single binary, zero external services no Redis, no vector DB, no Postgres | One developer's machine running MCP-native agents — you want a local security layer injection scanner, secret redaction, circuit breaker and efficiency layer cache, compressor, budget without any third party in the request path. | | Self-hosted Python proxy | A team or org routing to 100+ providers behind one OpenAI-compatible API, with a large ecosystem and community behind it. | | Self-hosted Apache 2.0 or managed cloud | A team that needs production guardrails PII/jailbreak/prompt-injection detection and real embedding-based semantic caching out of the box, at the cost of a heavier deployment. | TokenShift and similar hosted gateways | Managed SaaS | Teams that want zero infrastructure to run themselves and are comfortable with a third-party operator seeing 100% of their traffic in exchange for that convenience. | If you're evaluating infrastructure that will see your API keys and your code, that "no third party ever sees your traffic" property is the one thing here that's structural, not a feature checkbox — it's true of Kotro by construction there's nothing else in the request path and isn't something a hosted gateway can offer without changing its own business model. By preserving request-shape stability for upstream prefix caching DeepSeek V4, Qwen, and similar and layering a local prompt-state cache on top for exact repeats, Kotro reduces inference costs for heavy agent loops. In a standard 3-turn codebase benchmark full data in benchmarks/eval-suite/RESULTS.md /kotro-labs/kotro-proxy-engine/blob/main/benchmarks/eval-suite/RESULTS.md : Turn 1 : 2042 tokens sent. Turn 2 : Local proxy cache miss new turn content → forwarded upstream → DeepSeek's own prefix cache hit 1920 tokens cached server-side . Only 141 tokens billed. Turn 3 : Local proxy cache miss → DeepSeek prefix cache hit again. Only 159 tokens billed. Total upstream billed tokens: ~99.3% reduction in this benchmark. Read this number precisely: in every recorded turn above, Kotro's own local cache missed — each turn had new content, so there was nothing to replay locally. The reduction shown here is upstream provider prefix caching doing the work; Kotro's contribution in this specific benchmark is keeping the request shape stable so that upstream caching can fire cleanly, not a local cache hit. Kotro's local cache adds a second, independent savings layer on genuinely repeated prompts retries, shared fixtures, parallel agent runs hitting the same turn with zero upstream round-trip — that scenario isn't yet represented in the published eval suite, and we're adding a repeated-prompt fixture so the local-cache contribution can be measured and reported on its own. | Feature | Description | |---|---| MCP prompt injection scanner | Scans every tool result and user message for prompt injection patterns 14 regex rules before forwarding to the LLM. Warn-by-default; set KOTRO INJECTION BLOCK=true to hard-block. | Secret redaction | Strips API keys, DB URLs, passwords, and PII from requests before they leave your machine; restores placeholders in streaming responses so your agent still sees the right values. | Agent loop circuit breaker | Detects 3+ identical tool calls in one conversation window and opens the circuit X-Kotro-Circuit-Open header , preventing runaway billing from stuck agent loops. | Reasoning model budget controller | Caps thinking.budget tokens Anthropic or max completion tokens OpenAI o1/o3 per request. Set KOTRO MAX THINKING TOKENS ; set KOTRO REASONING BLOCK=true to block reasoning models entirely. | Streaming prompt-state cache | Captures complete SSE streams on miss; replays on exact-match prompt state system + latest user + model . Zero upstream round-trip on repeated prompts. | Local semantic cache Rust engine | Layers embedding-based fuzzy matching on top of the exact-match cache — all-MiniLM-L6-v2 | candle . Paraphrased prompts hit the cache with no embedding API call. Adds ~26-28ms per request. MCP tool result cache read file , git status , search results by scope, tool name, args with per-category TTLs read=30s / status=5m / search=1h . Write operations auto-invalidate stale reads for the same path. Context compressor Per-session token budget KOTRO SESSION TOKEN BUDGET . Returns HTTP 429 with X-Kotro-Budget-Remaining header before the LLM call is made. Universal provider support POST /v1/messages . Offline test harness Load benchmarks | Channel | Command | |---|---| 1-Click Install macOS/Linux | curl -sL https://raw.githubusercontent.com/kotro-labs/kotro-proxy-engine/main/scripts/install.sh | bash | Docker | docker run -p 3000:3000 kotrolabs/kotro-proxy | npm | npm install -g @kotro-labs/proxy-engine → kotro-proxy | Homebrew | brew install kotro-labs/tap/kotro | VS Code / Cursor | | GitHub Release Download binary https://github.com/kotro-labs/kotro-proxy-engine/releases From source cargo install --path rust/kotro-proxy Registry publish runs automatically on v tags when NPM TOKEN and VSCE PAT secrets are configured. Marketplace uses marketplace-publish.yml /kotro-labs/kotro-proxy-engine/blob/main/.github/workflows/marketplace-publish.yml see distributions/MARKETPLACE-AUTOMATION.md /kotro-labs/kotro-proxy-engine/blob/main/distributions/MARKETPLACE-AUTOMATION.md . Every GitHub Release binary is signed keylessly via cosign https://github.com/sigstore/cosign — the workflow proves it built the artifact using its own GitHub Actions identity via Sigstore's public Fulcio/Rekor infrastructure , with no private key for anyone to manage or leak. Each release also ships an SPDX SBOM. To verify a downloaded binary before running it: Download the binary, its .sig, and its .pem alongside it all in the same GitHub Release cosign verify-blob \ --certificate kotro-proxy-x86 64-apple-darwin.tar.gz.pem \ --signature kotro-proxy-x86 64-apple-darwin.tar.gz.sig \ --certificate-identity-regexp 'https://github.com/kotro-labs/kotro-proxy-engine/. ' \ --certificate-oidc-issuer https://token.actions.githubusercontent.com \ kotro-proxy-x86 64-apple-darwin.tar.gz A successful verification confirms the binary was built by this repository's release workflow and hasn't been modified since. The curl | bash and Homebrew/npm channels above don't currently run this check automatically — if you need that guarantee, verify the GitHub Release asset directly rather than one of the wrapped installers, or open an issue if automatic verification in those channels matters for your use case. - In Cursor, open Settings → Models . - Set the OpenAI Base URL to http://localhost:3000/v1 . - Set your OpenAI/Anthropic API Key. - Enjoy prompt-state caching and AST pruning out of the box Kotro automatically translates protocols. You can use Anthropic-native tools with local OpenAI-compatible models - Start your local Ollama: ollama run llama3 . - Start Kotro, pointing upstream to Ollama: KOTRO UPSTREAM URL=http://localhost:11434/v1 kotro - Run Aider: export ANTHROPIC API KEY="dummy" aider --model anthropic/claude-3-5-sonnet-20241022 --openai-api-base http://localhost:3000/v1 Terminal A: Start Proxy kotro Point your IDE or SDK at http://localhost:3000/v1 . View your savings dashboard at http://localhost:3000/ . curl -N http://127.0.0.1:8080/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $OPENAI API KEY" \ -d '{"model":"gpt-4","stream":true,"messages": {"role":"user","content":"hello"} }' curl -N http://127.0.0.1:8080/v1/messages \ -H "Content-Type: application/json" \ -H "x-api-key: $ANTHROPIC API KEY" \ -H "anthropic-version: 2023-06-01" \ -d '{"model":"claude-3-5-sonnet-20241022","max tokens":256,"stream":true,"messages": {"role":"user","content":"hello"} }' Cache hits return X-Kotro-Cache: HIT . Local dashboard: http://127.0.0.1:9090/dashboard http://127.0.0.1:9090/dashboard requires KOTRO ENABLE METRICS=true . | Variable | Default | Purpose | |---|---|---| KOTRO LISTEN ADDR | :8080 | Proxy bind address | KOTRO UPSTREAM URL | http://127.0.0.1:9000 | Provider base URL | KOTRO ENABLE CACHE | true | Prompt-state SSE cache | KOTRO ENABLE VECTOR CACHE | true | Local semantic embedding cache layer — Rust engine only | KOTRO ENABLE REDACTION | true | Local PII guardrail | KOTRO ENABLE COMPRESSION | true | Context deduplication | KOTRO CACHE HIT DELAY MS | 2 | Replay pacing on cache hits | KOTRO CACHE TTL | 24h | Cache entry lifetime 0 disables expiry | KOTRO EVICTION INTERVAL | 10m | Background sweep for expired keys | KOTRO ENABLE PPROF | false | Expose /debug/pprof for leak audits | KOTRO ENABLE METRICS | true | Expose /metrics and /dashboard on KOTRO METRICS ADDR default 127.0.0.1:9090 | KOTRO METRICS ADDR | 127.0.0.1:9090 | Isolated telemetry bind address | KOTRO CACHE KEY STRATEGY | window n | Cache key material: latest only , window n , full digest | KOTRO CACHE WINDOW SIZE | 4 | Trailing non-system turns hashed when strategy is window n | | Strategy | What is hashed | Recommended for | |---|---|---| default window n | System prompt + last N user/assistant/tool turns | Production agent loops — balances hit rate and correctness | full digest | Entire conversation JSON | Shared multi-tenant or strict deterministic pipelines | latest only | System + latest user text only | Legacy compatibility only — risky for multi-turn agents | latest only can return a cache hit when two agent sessions share the same final user phrase but different tool outputs in between silent state corruption . Prefer window n or full digest in production. Prometheus exposes the active strategy as kotro cache key strategy{strategy,window size} . You can use the KOTRO PROFILE environment variable for zero-friction setup: | Profile | Listen | Cache strategy | Recommended IDE | |---|---|---|---| cursor | :8080 | window n | Cursor Automatically handles Composer context | copilot | :8080 | full digest | GitHub Copilot strict full-context cache keys | continue | :8080 | window n | Continue.dev | For advanced control: | Profile | Listen | Cache strategy | Scope / trust | |---|---|---|---| Local dev | :8080 | window n | Default credential-derived scope | Trusted gateway | 0.0.0.0:8080 | window n | KOTRO TRUST UPSTREAM GATEWAY=true + KOTRO TRUSTED PROXY CIDRS | Shared multi-tenant | 0.0.0.0:8080 | full digest | Gateway headers + trusted proxy CIDRs; telemetry on loopback only | Verifies zero goroutine leak after mass mid-stream client disconnects. brew install k6 make cancel-audit Full storm: 500 parallel agents for 30s K6 VUS=500 K6 DURATION=30s make cancel-audit Requires KOTRO ENABLE PPROF=true set automatically by run audit.sh . Pass criteria: post-stress goroutine count within ±5 of baseline. The internal/ directory contains the original Go implementation, tagged v0.1.0-go and frozen. It is preserved as a behavioral reference — the Rust implementation is what you should run. No new features will land in Go.Bug reports and PRs for internal/ will not be accepted. If you're reading the Go code, treat it as an annotated spec for the Rust port. cd rust && cargo test && cargo run -p kotro-proxy Architecture map: docs/RUST-ARCHITECTURE.md /kotro-labs/kotro-proxy-engine/blob/main/docs/RUST-ARCHITECTURE.md Install k6 https://k6.io/ : brew install k6 chmod +x scripts/bench/run.sh make load-test all scenarios make load-test SCENARIO=hit make eval-suite full ROI dashboard → benchmarks/eval-suite/RESULTS.md Scenarios: miss , hit , anthropic , mixed , all . Eval suite results and methodology: benchmarks/eval-suite/RESULTS.md /kotro-labs/kotro-proxy-engine/blob/main/benchmarks/eval-suite/RESULTS.md . Roadmap and security docs: docs/roadmap/90-DAY-ROADMAP.md /kotro-labs/kotro-proxy-engine/blob/main/docs/roadmap/90-DAY-ROADMAP.md , docs/security/THREAT-MODEL.md /kotro-labs/kotro-proxy-engine/blob/main/docs/security/THREAT-MODEL.md . Vegeta alternative: go install github.com/tsenart/vegeta@latest bash scripts/bench/vegeta.sh Go micro-benchmarks: make bench IDE / SDK → kotro-proxy :8080 ├─ /v1/chat/completions intercept: cache · redact · compress ├─ /v1/messages intercept: cache · redact · compress └─ /v1/ passthrough ↓ upstream provider OpenAI, Anthropic, mock, … rust/kotro-proxy/ Active Rust implementation run this cmd/proxy/ Go proxy binary frozen — reference only cmd/mockupstream/ Offline OpenAI + Anthropic SSE server internal/ Go reference implementation frozen at v0.1.0-go cache/ bbolt prompt-state cache compressor/ Context block dedup guardrail/ Secret redaction models/ OpenAI + Anthropic request types proxy/ Handlers, SSE interceptor pipeline sse/ Frame parser OpenAI data: + Anthropic event: scripts/bench/ k6 / vegeta load tests MIT /kotro-labs/kotro-proxy-engine/blob/main/LICENSE — contributions welcome.