If you run Hermes Agent for any real workload, the single biggest knob on your bill is the prompt cache. The Hermes developer notes say it plainly: cache-breaking forces dramatically higher costs. The fix is simple. Do not break the cache. The hard part is knowing whether your cache is actually working.
Hermes hides the model call behind a clean tool-and-skill loop, which is great for everything except observability. A cold prompt and a warm one look the same from the agent side. You only see the difference in the provider's invoice. This post shows how to attach a cache meter to your Hermes Agent and read the hit ratio in real time.
What you can measure #
Anthropic, OpenAI, and Bedrock all return cache metadata in the response payload. The fields are named differently per vendor. If you wrap the model client before Hermes hands it a request, you can compute:
- per-call cache hit ratio
- cumulative ratio for the session
- USD saved versus an uncached baseline
- which prefix is performing well and which is not
Hermes does not expose this out of the box. You add it in the provider plugin.
The library #
I built cachebench
for exactly this problem. It is a small Python library that you wrap around the SDK call your provider plugin already uses. Every request and response passes through it, the cache metadata gets parsed, and you get a running tally.
pip install cachebench
It reads cache fields for Anthropic, OpenAI, and Bedrock today. About a hundred lines of vendor-neutral code, no provider lock-in.
Plugging it into Hermes #
Hermes model-provider plugins use lazy discovery via providers.register_provider(ProviderProfile(...))
. The cleanest place to add a meter is wherever your provider builds its client. You wrap the messages.create
method (Anthropic) or chat.completions.create
(OpenAI) with CacheTracker.wrap
, and you are done.
from anthropic import Anthropic
from cachebench import CacheTracker, Provider
client = Anthropic()
tracker = CacheTracker(provider=Provider.ANTHROPIC)
client.messages.create = tracker.wrap(client.messages.create)
Hand the wrapped client to your provider profile and Hermes will use it for every model call. The wrapper is sync-and-async aware, so the same line works whether your profile uses the sync or async client.
Reading the meter #
CacheTracker
keeps the rolling counts in memory. The simplest report is tracker.aggregate()
:
print(tracker.aggregate())
A good place to call this is the on_session_end
hook. That keeps your turn-by-turn experience clean and gives you a single summary line when the agent finishes.
For a per-prefix breakdown:
for prefix_id, stats in tracker.by_prefix().items():
print(prefix_id, f"{stats['hit_ratio']:.2f}", stats['calls'])
prefix_id
is a stable hash of the cacheable portion of the request: system message, tools, and all prior turns. When the ratio drops you can see which prefix changed, then go fix it.
What kills your hit ratio #
Three common patterns explain most of the bad runs I have seen.
- A system message that includes the current time. One timestamp at the top of the prompt invalidates everything below it. Move time-sensitive context into a tool call instead.
- A dynamic tool list. If your toolset changes between turns, the cached prefix is gone. Settle on a stable toolset for the session and re-register on demand.
- Skill that drops or reorders skills mid-conversation. Hermes defers some of this for you, but custom plugins can still break it. Use
--now
only when you really mean it.
tracker.by_prefix()
will surface the offending prefix the moment it appears.
Catching regressions #
The richer use is automated. CacheTracker
accepts a miss_alert_threshold
and an on_miss_alert
callback. Wire it to your logging once and any plugin or skill that drops the ratio below the floor produces a structured warning.
def alert(metrics):
print(f"[CACHE-MISS] prefix={metrics.prefix_id} "
f"ratio={metrics.hit_ratio:.2f} cost=${metrics.cost_usd(tracker.pricing):.4f}")
tracker = CacheTracker(
provider=Provider.ANTHROPIC,
miss_alert_threshold=0.6,
on_miss_alert=alert,
)
You can also opt into miss-aware retry. When CachePolicy.miss_aware()
is enabled and a call returns a cache-eligible prefix but a zero hit ratio (a silent provider miss), CacheTracker
waits two seconds and retries once. Useful right after a system message change, when the cache is still propagating.
from cachebench import CachePolicy
tracker = CacheTracker(provider=Provider.ANTHROPIC, policy=CachePolicy.miss_aware())
What to do with the numbers #
Once the meter is running, two small habits pay back fast.
- Check
tracker.aggregate()
at session end. If the hit ratio is under fifty percent, something is invalidating the cache. Runtracker.by_prefix()
and look for the prefix that ballooned recently. - Add a CI test that runs a short scripted conversation and asserts the hit ratio is above a floor. If a future plugin starts breaking the cache, the test catches it before your bill does.
Closing #
You do not need a vendor dashboard to know how well your Hermes Agent is caching. About fifteen lines of wrapper code and a small Python library are enough. The biggest cost optimization for an agent that talks a lot is the one that already shipped, you just have to confirm it is working.
Repo: https://github.com/MukundaKatta/cachebench
PyPI: https://pypi.org/project/cachebench/
Hermes Agent: https://hermes-agent.nousresearch.com/