Show HN: Prefixwise – names why each LLM request missed the prompt cache

A developer released Prefixwise, a proxy that sits between an AI agent and a model provider and names the specific cause of each prompt-cache miss, with a `pip install prefixwise` command and `--optimize` mode that repairs requests before they are sent. In a real run against the Anthropic API with Claude Sonnet 4.6, an eight-turn agent that reordered its tool list on turn 4 and again on turn 6 logged 26,932 prompt tokens, a 74.6% cache hit rate and $0.0356 in cost, of which 3,375 tokens ($0.0116, 12.5% of prompt tokens) were avoidable; turn 4 alone re-paid for 3,329 tokens ($0.0115) after a `tool_reserialize` change at `tools[0]`. Prefixwise also offers `prefixwise check` to fail CI when a code change would break the cache, auto-detects vLLM and SGLang behind a URL, and stores only hashes and counts rather than prompts.

Your agent resends its whole conversation every turn, and one changed byte near the top means the provider charges full price for everything after it. Prefixwise tells you which byte, what it cost, and fixes the common causes for you. Providers and engines report how many tokens missed the prompt cache, not why . Prefixwise sits between your agent and the model and does four things: - Identify. One named cause for every request, with its evidence, its cost and its fix. - Fix. --optimize repairs the request before it is sent. Only fixes that passed a replay against live models. - Prevent. prefixwise check fails CI when a code change would break the cache. - Auto-detect. It recognises vLLM and SGLang behind a URL and, when the engine publishes KV events, subscribes to them on its own. It stores hashes and counts, never your prompts. pip install prefixwise prefixwise run -- claude any agent with an API key: claude, codex, opencode, your own prefixwise report --session latest the session turn by turn prefixwise run --optimize -- claude the same agent, with the proven fixes applied No API key Claude Code or Codex on a subscription ? prefixwise import claude-code reads the logs already on your machine. A real run against the Anthropic API Claude Sonnet 4.6 : an eight-turn agent whose tool list comes back in another order on turn 4 and returns on turn 6 the run, with the agent https://github.com/tejachavali/prefixwise/blob/main/experiments/demo/2026-11-10-optimize-real-run.md . bash $ prefixwise run --ledger before.db -- python agent.py 8 requests, 26,932 prompt tokens, 75% served from cache. Cost $0.0356. Avoidable: 3,375 tokens $0.0116 , 12.5% of prompt tokens. Fix these, biggest first: 1. The same tools were sent in a different order or format 2 requests, 3,375 tokens, $0.0116 . Fix: Send tools in a fixed order with deterministic serialization or let the proxy do it: prefixwise serve --fix canonical tools . Prefixwise can do it for you: prefixwise run --fix canonical tools proven Expected misses not waste : 1 x first request of a session: nothing to reuse yet. Four lines of the output are left out here: the proxy's address, the -- prefixwise: this run -- header, a note on 46 tokens written and never read, and the pointer to the ledger. The record has all of it. prefixwise report --ledger before.db --session latest shows where it happened: Session run-20261110-010636: 8 requests, 26,932 prompt tokens, cache hit rate 74.6%, 3,375 avoidable, cost $0.0356 claude-sonnet-4-6 turn prompt cached hit cause first changed cost avoidable avoidable $ ---- ------ ------ ----- ---------------- ------------- ------- --------- ----------- 1 3,286 0 0.0% first write - $0.0125 0 $0.0000 2 3,309 3,283 99.2% none messages 8 $0.0017 0 $0.0000 3 3,332 3,306 99.2% none messages 10 $0.0016 0 $0.0000 4 3,355 0 0.0% tool reserialize tools 0 $0.0132 3,329 $0.0115 5 3,378 3,352 99.2% none messages 14 $0.0019 0 $0.0000 6 3,401 3,329 97.9% tool reserialize tools 0 $0.0016 46 $0.0002 7 3,424 3,398 99.2% none messages 18 $0.0020 0 $0.0000 8 3,447 3,421 99.2% none messages 20 $0.0013 0 $0.0000 "first changed" is the block where the prompt stopped matching the previous request, counted over the whole prompt: tools, then system, then messages. Turn 4 paid for the whole prompt again. Turn 6 went back to the first order and read 3,329 tokens, the prefix turn 3 had written under that order: the older entry was still cached. With --optimize , same agent and same prompts: | | before | with --optimize | |---|---|---| | Served from cache | 74.6% | 87.1% | | Cost of the run | $0.0356 | $0.0250 | | Avoidable tokens | 3,375 | 0 | Every number is from a live run against real models, and each links to its record. The full table is in the guide https://github.com/tejachavali/prefixwise/blob/main/docs/GUIDE.md results . | Claim | Result | |---|---| | Agrees with Anthropic's own miss reason | 32 of 32 misses Anthropic named 1 https://github.com/tejachavali/prefixwise/blob/main/experiments/e1 hosted/runs/2026-11-06-anthropic-claude-haiku-5-5.md , 2 https://github.com/tejachavali/prefixwise/blob/main/experiments/e1 hosted/runs/2026-11-06-round4.md | | Names injected causes on open models | 120 of 120 on vLLM, vLLM + LMCache and SGLang vLLM https://github.com/tejachavali/prefixwise/blob/main/experiments/e1 open/runs/2026-11-06-colab-l4-qwen3-4b.md , LMCache and SGLang https://github.com/tejachavali/prefixwise/blob/main/experiments/e evict/runs/2026-11-07-colab-l4.md | | Tells an engine eviction from everything else | 5 of 5 , confirmed from KV events vLLM https://github.com/tejachavali/prefixwise/blob/main/experiments/e evict/runs/2026-11-07-colab-l4.md , SGLang https://github.com/tejachavali/prefixwise/blob/main/experiments/e evict/runs/2026-11-07-colab-l4-sglang-events.md ; a prefix LMCache or HiCache restored counted as a hit 5 of 5 , through the proxy and inside vLLM record https://github.com/tejachavali/prefixwise/blob/main/experiments/e evict/runs/2026-11-08-colab-l4-host-restored.md , inside vLLM https://github.com/tejachavali/prefixwise/blob/main/experiments/e evict/runs/2026-11-10-colab-l4-middleware-lmcache.md | | The tool-order fix saves what was predicted | 99.9% on Claude, 97.5% on vLLM, task success flat one of 60 turns changed on Qwen3-4B record https://github.com/tejachavali/prefixwise/blob/main/experiments/e4/runs/2026-11-06-live.md | | The cache-marker fix pays for itself | 106.8% of predicted savings, prompt cost down 85.6% with the write surcharge included, task success flat record https://github.com/tejachavali/prefixwise/blob/main/experiments/e4/runs/2026-11-09-auto-breakpoints-live.md | | Works with real agents | Claude Code, Codex, opencode, the Anthropic and OpenAI SDKs record https://github.com/tejachavali/prefixwise/blob/main/experiments/compat/2026-11-07.md , opencode https://github.com/tejachavali/prefixwise/blob/main/experiments/compat/2026-11-09-opencode-run.md | | Costs almost nothing | 3 ms per request at 2k tokens, 13 ms at 120k record https://github.com/tejachavali/prefixwise/blob/main/benchmarks/runs/2026-11-07-vm.md | Each takes about thirty seconds. All write the same ledger and print the same report. | You have | Do this | |---|---| | An agent you run from a terminal | prefixwise run -- <your command | | A service, several agents, or a gateway | prefixwise serve --upstream https://api.anthropic.com , and point clients at it | | Your own vLLM server | vllm serve <model ... --middleware prefixwise.PrefixwiseMiddleware : no proxy, no port change docs/VLLM.md https://github.com/tejachavali/prefixwise/blob/main/docs/VLLM.md | | vLLM with LMCache | the same --middleware flag, or prefixwise serve --upstream <url in front, which detects the engine docs/LMCACHE.md https://github.com/tejachavali/prefixwise/blob/main/docs/LMCACHE.md | | SGLang, with or without HiCache | prefixwise serve --upstream <url in front; it detects the engine docs/SGLANG.md https://github.com/tejachavali/prefixwise/blob/main/docs/SGLANG.md | | Your own Python code | client = prefixwise.wrap Anthropic or OpenAI | | LiteLLM | litellm.callbacks = PrefixwiseCallback | | Claude Code or Codex on a subscription | prefixwise import claude-code / prefixwise import codex | Nineteen causes, each marked as waste or as a legitimate miss. The ones people meet most: | Cause | What happened | Fix | |---|---|---| | tool reserialize | The same tools, in another order or with other bytes | --optimize does it, or sort them yourself | | volatile system | A timestamp or id in the system prompt changed | Move it to the latest user message | | history edit | Earlier messages were rewritten | Append, do not rewrite; compact at task boundaries | | breakpoint misplaced | Anthropic: no cache control , or one that stays too early | --optimize places them, or add one on the last message | | ttl expired | The gap since the last turn outlived the cache | A longer TTL, if the reads pay for it | | evicted | A self-hosted engine dropped the session's blocks | More KV memory, or an offload tier | Beyond the report: prefixwise explain --diff a.json b.json shows what changed between two requests, prefixwise fixes ranks fixes by dollars, and every non-streaming response carries an X-Prefixwise-Cause header for a stream, look the request up by its id . All causes and details: guide https://github.com/tejachavali/prefixwise/blob/main/docs/GUIDE.md what-it-tells-you . --optimize on serve and run , or PREFIXWISE OPTIMIZE=1 for the vLLM middleware applies every fix that has passed a replay against a live model. A fix earns its place by realising at least half of what Prefixwise predicted with task success flat. - canonical tools sorts the tool list and each tool's keys, so a list that drifts between turns stops breaking the cache. - auto breakpoints Anthropic's API only adds cache control markers to a request that has none, once the same prefix was seen within the cache window. A one-shot request never pays for a write nobody reads, and a client's own markers are never touched. Off by default. If the upstream rejects a request carrying markers Prefixwise placed, the client's original is sent instead; canonical tools changes only the order of tools and keys. prefixwise fixes says whether a fix paid off on your traffic. Fixes that would change what the model reads, such as moving a timestamp, stay advice. prefixwise check -- pytest tests/agent scenarios exit 2 on a cache regression The costly regressions pass every functional test: a tool list built from a set, a timestamp added to the system prompt, a marker moved. prefixwise check runs your agent's scenarios against a local mock provider, with no API calls, and fails on four: tools in a changed order, a volatile system prompt, a misplaced marker, a mutated tool result. As a GitHub Action: - uses: tejachavali/prefixwise@v0.11 with: command: pytest tests/agent scenarios Details, allow-lists and baselines: guide https://github.com/tejachavali/prefixwise/blob/main/docs/GUIDE.md gate-it-in-ci . On vLLM and SGLang, with pip install "prefixwise engines " and the engine publishing its KV-cache events, Prefixwise reads them, so an eviction is confirmed and not inferred, and a prefix that LMCache or HiCache restored from CPU memory is labelled host restored : a hit the GPU alone would have missed. It also names what a chat template costs, such as a reasoning model recomputing its previous answer every turn. Step by step: vLLM https://github.com/tejachavali/prefixwise/blob/main/docs/VLLM.md , LMCache https://github.com/tejachavali/prefixwise/blob/main/docs/LMCACHE.md , SGLang https://github.com/tejachavali/prefixwise/blob/main/docs/SGLANG.md , how the engine join works https://github.com/tejachavali/prefixwise/blob/main/docs/ENGINES.md . Prometheus metrics with an alert rule and a Grafana dashboard docs https://github.com/tejachavali/prefixwise/blob/main/docs/prometheus.md , OpenTelemetry spans docs https://github.com/tejachavali/prefixwise/blob/main/docs/opentelemetry.md , an HTTP API docs https://github.com/tejachavali/prefixwise/blob/main/docs/API.md , a collector for several proxies, and sinks to JSONL or ClickHouse architecture https://github.com/tejachavali/prefixwise/blob/main/docs/ARCHITECTURE.md . Measured overhead is 3 ms per request at 2k tokens. - One traffic shape per proof. canonical tools was proven on Claude Haiku 5.5 and Qwen3-4B, auto breakpoints on Haiku 5.5 only; both on scripted multi-turn sessions seconds apart, not an agent's own traffic. - OpenAI's own verdict is unchecked. Its diagnostics answered unavailable on every call we made. Prefixwise still classifies OpenAI traffic. - A change is located per block, not per token. An edit at the end of a long system prompt is reported for the whole block. - Long sessions and many agents at once are less tested than short ones. - Dollars need a price. Models in the dated table are priced; others show a dash until you add rates. - Ollama and llama.cpp report no cached tokens most builds : Prefixwise names every change to the prompt, but cannot tell a hit from a miss there, and says so. - API keys only. An agent signed in with a subscription cannot be routed through a proxy; use prefixwise import for those. The full list, with what was and was not verified live: guide https://github.com/tejachavali/prefixwise/blob/main/docs/GUIDE.md limits . - The full guide https://github.com/tejachavali/prefixwise/blob/main/docs/GUIDE.md : every way in, every cause, the command and environment reference, production notes. - Use cases https://github.com/tejachavali/prefixwise/blob/main/docs/USE CASES.md , data and metrics https://github.com/tejachavali/prefixwise/blob/main/docs/DATA.md , architecture https://github.com/tejachavali/prefixwise/blob/main/docs/ARCHITECTURE.md , changelog https://github.com/tejachavali/prefixwise/blob/main/CHANGELOG.md , security https://github.com/tejachavali/prefixwise/blob/main/SECURITY.md . - Working on Prefixwise: pip install -e ". dev " && python -m pytest tests -q . Read WORKLOG.md https://github.com/tejachavali/prefixwise/blob/main/WORKLOG.md first. MIT. See LICENSE.