{"slug": "kvdiff-local-proxy-showing-the-byte-breaking-your-llm-prompt-cache", "title": "Kvdiff – Local proxy showing the byte breaking your LLM prompt cache", "summary": "A developer released kvdiff v0.1.0, a zero-code local proxy that identifies the exact UTF-8 byte offset breaking an LLM prompt cache across Anthropic, OpenAI, Google Gemini/Antigravity, llama.cpp and vLLM, and tallies the wasted spend. In its bundled demo, a Claude Code session with 15 MCP tools runs at 94% cache hit until Turn 14, when mcp-github reconnects and reorders tool schemas; kvdiff flags the break at $.tools[9].name, byte 1,670, calculates $0.38 in wasted cost, and shows Turn 15 returning to 97% cache hit under the opt-in --heal flag. The tool runs via npx @drod915/kvdiff and classifies breaks into six kinds, including TOOL_REORDER, VOLATILE_SYSTEM and TTL_EXPIRED.", "body_md": "**Zero-code local proxy that pinpoints the exact UTF-8 byte breaking your LLM prompt cache (Anthropic, OpenAI, Google Gemini / Antigravity, `llama.cpp`/vLLM), tallies wasted dollars, and optionally auto-heals prefixes live.**\n\n[Watch the 15-second kvdiff demo (MP4)](https://github.com/flipcyde101/kvdiff/releases/download/v0.1.0/kvdiff-scoped-demo.mp4)\n\n**Try the demo:** `npx @drod915/kvdiff --demo`\n\n```\nnpx @drod915/kvdiff --demo\n```\n\nOpens `http://127.0.0.1:4318` pre-seeded with two realistic sessions:\n\n1. **Claude Code + 15 MCP Tools (16 turns):** Turn 1 warms the prompt cache at 0%, and Turns 2-13 run at 94% cache hit. On**Turn 14** ,`mcp-github` reconnects and reorders tool schemas:`kvdiff` highlights the exact red byte offset (`$.tools[9].name` , byte 1,670), calculates the wasted cost ($0.38), and shows**Turn 15** snapping back to**97% cache hit** under`--heal` .\n2. **Antigravity / Gemini 2.5 Pro (5 turns):** Catches a volatile timestamp injected into`$.systemInstruction.parts[0].text` on Turn 4.\n\nRecord the demo in a terminal with `vhs demo/kvdiff.tape`.\n\nWrap any coding agent CLI with zero code changes:\n\n```\n# Claude Code (Anthropic Messages API)\nnpx @drod915/kvdiff -- claude\n\n# Codex CLI / OpenCode (OpenAI Responses & Chat APIs)\nnpx @drod915/kvdiff -- codex\n\n# Gemini CLI / Antigravity API workflows (Google Gemini API)\nnpx @drod915/kvdiff -- gemini\n\n# Opt-in deterministic tool-schema healing\nnpx @drod915/kvdiff --heal -- claude\n```\n\nOr run as a persistent local proxy for Antigravity, Cursor, IDE plugins, or local `llama.cpp`/vLLM:\n\n```\n# Cloud providers (Anthropic, OpenAI, Google Gemini / Antigravity)\nnpx @drod915/kvdiff serve --port 4318\n\n# Local llama.cpp / vLLM KV-cache prefix reuse debugging\nnpx @drod915/kvdiff serve --port 4318 --upstream-local http://127.0.0.1:8080\n```\n\nThen point your client base URL:\n\n- `ANTHROPIC_BASE_URL=http://127.0.0.1:4318`\n- `OPENAI_BASE_URL=http://127.0.0.1:4318/v1`\n- `GOOGLE_GEMINI_BASE_URL=http://127.0.0.1:4318`\n\n| Break Kind | What Happened on the Wire | Fix | \n|---|---|---|\n| `TOOL_REORDER` | MCP servers reconnected or registered tools in non-deterministic order | Pass `--heal` to sort`tools[]` (and Gemini`functionDeclarations[]` ) deterministically | \n| `TOOL_MUTATED` | A dynamic tool description or schema field changed mid-session | Pin static tool descriptions outside the cached prefix | \n| `VOLATILE_SYSTEM` | A timestamp, UUID, or mutable status string landed inside `$.system` /`$.systemInstruction` | Move dynamic metadata into the trailing user turn | \n| `TTL_EXPIRED` | Bytes matched 100%, but `>5m` elapsed between turns | Keep interactive sessions warm or opt into extended TTL | \n| `MODEL_CHANGED` | Model name switched mid-session | Keep one model per cache prefix | \n| `PREFIX_TOO_SHORT` | Prompt fell under OpenAI ( `1,024` ) or Gemini (`1,024` /`2,048` ) implicit-cache floor | Expected until context grows past threshold | \n\n| Feature | `kvdiff` | `ccusage` /`codexbar` | `ContextSpy` | Langfuse / Phoenix | \n|---|---|---|---|---|\n| **Install & Infra** | 1 cmd ( `npx` ), built-in`node:sqlite` | 1 cmd ( `npx` /binary) | 1 cmd ( `npx` ) | Docker + Postgres/ClickHouse | \n| **Wire-level UTF-8 byte LCP + JSONPath diff** | **Yes** | No (reads post-hoc logs) | No (token category pies) | No | \n| **Opt-in live prefix healing (`--heal`)** | **Yes** | No | No | No | \n| **Multi-provider (Anthropic + OpenAI + Gemini/Antigravity + `llama.cpp`)** | **Yes** | Single-provider focus | Anthropic/OpenAI | Yes | \n| **Zero-mutation default passthrough** | **Yes** (OAuth/beta headers intact) | N/A (offline log reader) | Yes | SDK instrumentation | \n| **Where competitors win** | - | Zero proxy setup; great historical billing totals across past months | High-level token breakdown visualizations | Production team evals, prompt versioning, multi-user cloud traces | \n\n```\nflowchart LR\n    Agent[\"Coding Agent / IDE\\n(Claude Code · Codex · Antigravity · Gemini · llama.cpp)\"] -->|\"HTTP (127.0.0.1:4318)\"| Proxy[\"kvdiff Proxy\\nZero-mutation default\\n(opt-in --heal)\"]\n    Proxy -->|\"Immediate chunked SSE flush\"| Upstream[\"Upstream Provider\\nAnthropic · OpenAI · Gemini · Local\"]\n    Proxy -.->|\"Non-blocking SSE tee\"| Engine[\"UTF-8 Byte LCP +\\nJSONPath Mapper +\\nBreak Classifier\"]\n    Engine --> DB[(\"~/.kvdiff/kvdiff.db\\nNode 24 node:sqlite WAL\")]\n    DB --> UI[\"Localhost Waterfall UI\\n127.0.0.1:4318\"]\n```\n\n- **100% Local:** Binds to`127.0.0.1` by default. Stores traffic only in`~/.kvdiff/kvdiff.db` . Zero telemetry, zero cloud accounts.\n- **Strict Zero-Mutation Default:** Without`--heal` ,`kvdiff` forwards your exact raw request bytes and every non-hop-by-hop header (`Authorization` ,`x-api-key` ,`x-goog-api-key` ,`anthropic-beta` ,`openai-beta` ) unmodified.\n- **Zero-Buffering SSE:** Streaming responses flush downstream chunk-by-chunk as they arrive from upstream.\n- **Safe `--heal` Invariant:**`--heal` sorts tools and tool-definition keys, normalizes JSON formatting, and keeps the trailing Anthropic cache breakpoint on the final tool after sorting. It**never** reorders`messages[]` ,`contents[]` ,`parts[]` ,`content[]` ,`input[]` , or JSON Schema`enum[]` .\n\n```\ngit clone https://github.com/flipcyde101/kvdiff.git\ncd kvdiff\nnpm install\nnpm run typecheck\nnpm run build\nnpm test\n```\n\nFound a cache-break pattern or provider edge case? Open an issue at [github.com/flipcyde101/kvdiff/issues](https://github.com/flipcyde101/kvdiff/issues).", "url": "https://wpnews.pro/news/kvdiff-local-proxy-showing-the-byte-breaking-your-llm-prompt-cache", "canonical_source": "https://github.com/flipcyde101/kvdiff", "published_at": "2026-11-21 22:38:12+00:00", "updated_at": "2026-11-21 23:13:11.651972+00:00", "lang": "en", "topics": ["ai-tools", "large-language-models", "ai-agents", "developer-tools", "mlops"], "entities": ["kvdiff", "Anthropic", "OpenAI", "Google Gemini", "Antigravity", "llama.cpp", "vLLM", "Claude Code"], "also_reported_by": [], "takeaway": "Run kvdiff as a local proxy to see the exact UTF-8 byte and JSONPath where a prompt prefix diverges, and pass --heal to sort tools[] deterministically so MCP reconnects stop breaking the cache.", "recommendation": null, "alternates": {"html": "https://wpnews.pro/news/kvdiff-local-proxy-showing-the-byte-breaking-your-llm-prompt-cache", "markdown": "https://wpnews.pro/news/kvdiff-local-proxy-showing-the-byte-breaking-your-llm-prompt-cache.md", "text": "https://wpnews.pro/news/kvdiff-local-proxy-showing-the-byte-breaking-your-llm-prompt-cache.txt", "jsonld": "https://wpnews.pro/news/kvdiff-local-proxy-showing-the-byte-breaking-your-llm-prompt-cache.jsonld"}}