{"slug": "how-to-measure-prompt-cache-hit-ratio-in-your-hermes-agent", "title": "How to measure prompt cache hit ratio in your Hermes Agent", "summary": "To measure and monitor prompt cache hit ratios in Hermes Agent to reduce costs, as cache-breaking significantly increases expenses. It introduces a Python library called cachebench that wraps model client calls to track cache metadata from providers like Anthropic, OpenAI, and Bedrock, providing real-time hit ratios and cost savings. The tool offers features like per-prefix breakdowns, miss alerts, and automatic retries to help users identify and fix cache-breaking patterns.", "body_md": "If you run Hermes Agent for any real workload, the single biggest knob on your bill is the prompt cache. The Hermes developer notes say it plainly: cache-breaking forces dramatically higher costs. The fix is simple. Do not break the cache. The hard part is knowing whether your cache is actually working.\n\nHermes hides the model call behind a clean tool-and-skill loop, which is great for everything except observability. A cold prompt and a warm one look the same from the agent side. You only see the difference in the provider's invoice. This post shows how to attach a cache meter to your Hermes Agent and read the hit ratio in real time.\n\n## What you can measure\n\nAnthropic, OpenAI, and Bedrock all return cache metadata in the response payload. The fields are named differently per vendor. If you wrap the model client before Hermes hands it a request, you can compute:\n\n- per-call cache hit ratio\n- cumulative ratio for the session\n- USD saved versus an uncached baseline\n- which prefix is performing well and which is not\n\nHermes does not expose this out of the box. You add it in the provider plugin.\n\n## The library\n\nI built `cachebench`\n\nfor exactly this problem. It is a small Python library that you wrap around the SDK call your provider plugin already uses. Every request and response passes through it, the cache metadata gets parsed, and you get a running tally.\n\n```\npip install cachebench\n```\n\nIt reads cache fields for Anthropic, OpenAI, and Bedrock today. About a hundred lines of vendor-neutral code, no provider lock-in.\n\n## Plugging it into Hermes\n\nHermes model-provider plugins use lazy discovery via `providers.register_provider(ProviderProfile(...))`\n\n. The cleanest place to add a meter is wherever your provider builds its client. You wrap the `messages.create`\n\nmethod (Anthropic) or `chat.completions.create`\n\n(OpenAI) with `CacheTracker.wrap`\n\n, and you are done.\n\n``` python\nfrom anthropic import Anthropic\nfrom cachebench import CacheTracker, Provider\n\nclient = Anthropic()\ntracker = CacheTracker(provider=Provider.ANTHROPIC)\n\n# Replace the bare method with the tracked one once at startup.\nclient.messages.create = tracker.wrap(client.messages.create)\n```\n\nHand the wrapped client to your provider profile and Hermes will use it for every model call. The wrapper is sync-and-async aware, so the same line works whether your profile uses the sync or async client.\n\n## Reading the meter\n\n`CacheTracker`\n\nkeeps the rolling counts in memory. The simplest report is `tracker.aggregate()`\n\n:\n\n```\nprint(tracker.aggregate())\n# {'calls': 41, 'hit_ratio': 0.805, 'tokens_read_from_cache': 184320,\n#  'tokens_written_to_cache': 14210, 'cost_usd': 0.94, 'cost_saved_usd': 1.21}\n```\n\nA good place to call this is the `on_session_end`\n\nhook. That keeps your turn-by-turn experience clean and gives you a single summary line when the agent finishes.\n\nFor a per-prefix breakdown:\n\n```\nfor prefix_id, stats in tracker.by_prefix().items():\n    print(prefix_id, f\"{stats['hit_ratio']:.2f}\", stats['calls'])\n```\n\n`prefix_id`\n\nis a stable hash of the cacheable portion of the request: system message, tools, and all prior turns. When the ratio drops you can see which prefix changed, then go fix it.\n\n## What kills your hit ratio\n\nThree common patterns explain most of the bad runs I have seen.\n\n- A system message that includes the current time. One timestamp at the top of the prompt invalidates everything below it. Move time-sensitive context into a tool call instead.\n- A dynamic tool list. If your toolset changes between turns, the cached prefix is gone. Settle on a stable toolset for the session and re-register on demand.\n- Skill loading that drops or reorders skills mid-conversation. Hermes defers some of this for you, but custom plugins can still break it. Use\n`--now`\n\nonly when you really mean it.\n\n`tracker.by_prefix()`\n\nwill surface the offending prefix the moment it appears.\n\n## Catching regressions\n\nThe richer use is automated. `CacheTracker`\n\naccepts a `miss_alert_threshold`\n\nand an `on_miss_alert`\n\ncallback. Wire it to your logging once and any plugin or skill that drops the ratio below the floor produces a structured warning.\n\n``` python\ndef alert(metrics):\n    print(f\"[CACHE-MISS] prefix={metrics.prefix_id} \"\n          f\"ratio={metrics.hit_ratio:.2f} cost=${metrics.cost_usd(tracker.pricing):.4f}\")\n\ntracker = CacheTracker(\n    provider=Provider.ANTHROPIC,\n    miss_alert_threshold=0.6,\n    on_miss_alert=alert,\n)\n```\n\nYou can also opt into miss-aware retry. When `CachePolicy.miss_aware()`\n\nis enabled and a call returns a cache-eligible prefix but a zero hit ratio (a silent provider miss), `CacheTracker`\n\nwaits two seconds and retries once. Useful right after a system message change, when the cache is still propagating.\n\n``` python\nfrom cachebench import CachePolicy\ntracker = CacheTracker(provider=Provider.ANTHROPIC, policy=CachePolicy.miss_aware())\n```\n\n## What to do with the numbers\n\nOnce the meter is running, two small habits pay back fast.\n\n- Check\n`tracker.aggregate()`\n\nat session end. If the hit ratio is under fifty percent, something is invalidating the cache. Run`tracker.by_prefix()`\n\nand look for the prefix that ballooned recently. - Add a CI test that runs a short scripted conversation and asserts the hit ratio is above a floor. If a future plugin starts breaking the cache, the test catches it before your bill does.\n\n## Closing\n\nYou do not need a vendor dashboard to know how well your Hermes Agent is caching. About fifteen lines of wrapper code and a small Python library are enough. The biggest cost optimization for an agent that talks a lot is the one that already shipped, you just have to confirm it is working.\n\nRepo: [https://github.com/MukundaKatta/cachebench](https://github.com/MukundaKatta/cachebench)\n\nPyPI: [https://pypi.org/project/cachebench/](https://pypi.org/project/cachebench/)\n\nHermes Agent: [https://hermes-agent.nousresearch.com/](https://hermes-agent.nousresearch.com/)", "url": "https://wpnews.pro/news/how-to-measure-prompt-cache-hit-ratio-in-your-hermes-agent", "canonical_source": "https://dev.to/mukundakatta/how-to-measure-prompt-cache-hit-ratio-in-your-hermes-agent-2k25", "published_at": "2026-05-19 06:48:36+00:00", "updated_at": "2026-05-19 07:05:28.116827+00:00", "lang": "en", "topics": ["large-language-models", "developer-tools", "artificial-intelligence", "cloud-computing", "open-source"], "entities": ["Hermes Agent", "Anthropic", "OpenAI", "Bedrock", "cachebench"], "alternates": {"html": "https://wpnews.pro/news/how-to-measure-prompt-cache-hit-ratio-in-your-hermes-agent", "markdown": "https://wpnews.pro/news/how-to-measure-prompt-cache-hit-ratio-in-your-hermes-agent.md", "text": "https://wpnews.pro/news/how-to-measure-prompt-cache-hit-ratio-in-your-hermes-agent.txt", "jsonld": "https://wpnews.pro/news/how-to-measure-prompt-cache-hit-ratio-in-your-hermes-agent.jsonld"}}