How to measure prompt cache hit ratio in your Hermes Agent

To measure and monitor prompt cache hit ratios in Hermes Agent to reduce costs, as cache-breaking significantly increases expenses. It introduces a Python library called cachebench that wraps model client calls to track cache metadata from providers like Anthropic, OpenAI, and Bedrock, providing real-time hit ratios and cost savings. The tool offers features like per-prefix breakdowns, miss alerts, and automatic retries to help users identify and fix cache-breaking patterns.

If you run Hermes Agent for any real workload, the single biggest knob on your bill is the prompt cache. The Hermes developer notes say it plainly: cache-breaking forces dramatically higher costs. The fix is simple. Do not break the cache. The hard part is knowing whether your cache is actually working. Hermes hides the model call behind a clean tool-and-skill loop, which is great for everything except observability. A cold prompt and a warm one look the same from the agent side. You only see the difference in the provider's invoice. This post shows how to attach a cache meter to your Hermes Agent and read the hit ratio in real time. What you can measure Anthropic, OpenAI, and Bedrock all return cache metadata in the response payload. The fields are named differently per vendor. If you wrap the model client before Hermes hands it a request, you can compute: - per-call cache hit ratio - cumulative ratio for the session - USD saved versus an uncached baseline - which prefix is performing well and which is not Hermes does not expose this out of the box. You add it in the provider plugin. The library I built cachebench for exactly this problem. It is a small Python library that you wrap around the SDK call your provider plugin already uses. Every request and response passes through it, the cache metadata gets parsed, and you get a running tally. pip install cachebench It reads cache fields for Anthropic, OpenAI, and Bedrock today. About a hundred lines of vendor-neutral code, no provider lock-in. Plugging it into Hermes Hermes model-provider plugins use lazy discovery via providers.register provider ProviderProfile ... . The cleanest place to add a meter is wherever your provider builds its client. You wrap the messages.create method Anthropic or chat.completions.create OpenAI with CacheTracker.wrap , and you are done. python from anthropic import Anthropic from cachebench import CacheTracker, Provider client = Anthropic tracker = CacheTracker provider=Provider.ANTHROPIC Replace the bare method with the tracked one once at startup. client.messages.create = tracker.wrap client.messages.create Hand the wrapped client to your provider profile and Hermes will use it for every model call. The wrapper is sync-and-async aware, so the same line works whether your profile uses the sync or async client. Reading the meter CacheTracker keeps the rolling counts in memory. The simplest report is tracker.aggregate : print tracker.aggregate {'calls': 41, 'hit ratio': 0.805, 'tokens read from cache': 184320, 'tokens written to cache': 14210, 'cost usd': 0.94, 'cost saved usd': 1.21} A good place to call this is the on session end hook. That keeps your turn-by-turn experience clean and gives you a single summary line when the agent finishes. For a per-prefix breakdown: for prefix id, stats in tracker.by prefix .items : print prefix id, f"{stats 'hit ratio' :.2f}", stats 'calls' prefix id is a stable hash of the cacheable portion of the request: system message, tools, and all prior turns. When the ratio drops you can see which prefix changed, then go fix it. What kills your hit ratio Three common patterns explain most of the bad runs I have seen. - A system message that includes the current time. One timestamp at the top of the prompt invalidates everything below it. Move time-sensitive context into a tool call instead. - A dynamic tool list. If your toolset changes between turns, the cached prefix is gone. Settle on a stable toolset for the session and re-register on demand. - Skill loading that drops or reorders skills mid-conversation. Hermes defers some of this for you, but custom plugins can still break it. Use --now only when you really mean it. tracker.by prefix will surface the offending prefix the moment it appears. Catching regressions The richer use is automated. CacheTracker accepts a miss alert threshold and an on miss alert callback. Wire it to your logging once and any plugin or skill that drops the ratio below the floor produces a structured warning. python def alert metrics : print f" CACHE-MISS prefix={metrics.prefix id} " f"ratio={metrics.hit ratio:.2f} cost=${metrics.cost usd tracker.pricing :.4f}" tracker = CacheTracker provider=Provider.ANTHROPIC, miss alert threshold=0.6, on miss alert=alert, You can also opt into miss-aware retry. When CachePolicy.miss aware is enabled and a call returns a cache-eligible prefix but a zero hit ratio a silent provider miss , CacheTracker waits two seconds and retries once. Useful right after a system message change, when the cache is still propagating. python from cachebench import CachePolicy tracker = CacheTracker provider=Provider.ANTHROPIC, policy=CachePolicy.miss aware What to do with the numbers Once the meter is running, two small habits pay back fast. - Check tracker.aggregate at session end. If the hit ratio is under fifty percent, something is invalidating the cache. Run tracker.by prefix and look for the prefix that ballooned recently. - Add a CI test that runs a short scripted conversation and asserts the hit ratio is above a floor. If a future plugin starts breaking the cache, the test catches it before your bill does. Closing You do not need a vendor dashboard to know how well your Hermes Agent is caching. About fifteen lines of wrapper code and a small Python library are enough. The biggest cost optimization for an agent that talks a lot is the one that already shipped, you just have to confirm it is working. Repo: https://github.com/MukundaKatta/cachebench https://github.com/MukundaKatta/cachebench PyPI: https://pypi.org/project/cachebench/ https://pypi.org/project/cachebench/ Hermes Agent: https://hermes-agent.nousresearch.com/ https://hermes-agent.nousresearch.com/