Tradeoff considerations while running LLM models locally

A developer who ran the Hermes personal agent for three months found that agent frameworks consume an order of magnitude more tokens than plain chatbots, with a simple "hi" message burning 17,292 tokens because large system prompts, skills overviews, and tool definitions are injected into every call. The overhead forced Hermes to reject local models with context windows under 64k, ruling out lightweight models on an 8GB Ubuntu machine running Ollama and pushing the developer toward closed frontier models instead.

My last post https://swapniltalekar.substack.com/p/three-months-of-running-personal?r=1v4ea was about my experience of running personal Hermes agents for the last couple of months. One of the things I mentioned there was to avoid running models locally. This post goes into the details of my experiments with local models, how agents really use LLMs and when it actually makes sense to run the models locally. A lot of the stuff being discussed on the internet about local models is either a benchmark comparison or a homelab enthusiast bragging about squeezing a 70B model onto a single GPU. Very few talk about the specific way an agent framework, as opposed to a plain chatbot, interacts with the models. I ran Hermes as my personal agent for three months, on real hardware, doing real tasks, checking my calendar, drafting messages, running small scripts, and it was quite a ride understanding it’s working with LLMs. Hermes is picky Like I said before, since running Hermes was a personal hobby project, I wanted to keep the cost of my agent setup to a minimum. I tried to locally run the models, which were small enough to run on my 8GB Ubuntu machine along with Ollama as the local inference provider. These are obviously not SOTA-level models, but they are good enough for experimentation and tinkering. To my surprise, Hermes won’t let me hook any of those models with it. Whenever I tried, it errored out, complaining that the model’s context window is less than 64k. I tried out a few small models, but it kept erroring out and asking me to choose another model. This restriction ruled out a lot of lightweight models I had on my list to cheaply run locally. The intriguing question was: Why does Hermes do that? Why would it insist that we only hook models with 64k context? How many tokens does a “hi” cost? The answer lies in how Hermes uses the model’s context window. Do you think you know enough about LLMs and Agents? Alright then, let’s do a quick quiz. How many tokens would it cost to send a simple “hi” message to an LLM? If you answered 1 token, well, you’re more or less correct. Now how about you send the same message via an AI agent? It could be absolutely any agent, not just Hermes. It could be OpenClaw, Claude Code, Codex, Cursor, just about anything else. Any guess? Well, the answer is: it could be anything between 10k - 40k+ tokens Yup, that many tokens for a simple “hi” message to the LLM when you’re using your favourite agent. Don’t believe me? Here’s a snapshot of the actual usage stats from my Hermes agent. To send a “hi” message to the LLM, it burnt 17,292 tokens The truth is that AI agents are giant token guzzlers. The reason they need so many tokens is that the agents need to inject giant system prompts and other information, such as Skills overview, tools information, useful persistent info, etc., into every single message to the LLM, even before your “hi” gets to have a place there. For example, here’s a breakdown of my Hermes’ system prompt: Agent loops don’t burn just a few more tokens than a direct LLM call; THEY BURN AN ORDER OF MAGNITUDE MORE While this token usage seems excessive, it’s this same overhead that lets the framework behave like an agent at all, deciding whether to call a tool, check a calendar, or just chat, instead of being locked into one fixed behaviour. If you strip that overhead out, you’d also strip out the thing that makes it an agent instead of a chatbot with extra steps. Because of this, the minimum token usage for every message in an agent framework sits far above what the same message would cost as a bare chat completion. This is also why a lot of top-tier models get way more expensive for you when you use them with agents. Using Closed Frontier models Once it was clear that I won’t be able to run small models locally, I tried looking at a few big models but the amount of GPU horsepower they needed exceeded the cloud spend budget I had in mind. So I gave up on running the models locally and tried hooking in some popular frontier models, but there were a few issues with that. Anthropic has already banned 3<sup rd</sup party harnesses https://www.mindstudio.ai/blog/anthropic-openclaw-ban-oauth-authentication since OpenClaw showed up on the scene I feel this was a really bad move on their part . It is technically possible to use Claude with Hermes, but you’ll have to pay their API pricing, which is much higher than their subscription cost, and it would be in addition to the subscription cost you’re already paying, which I wasn’t interested in. So I played around with Gemini and GPT models. I ran into another weird issue. It looked like Hermes was not compatible with certain models, and it threw an “Encrypted content is not supported with this model” error. I should have debugged this more, but I didn’t give it much thought and assumed that it was a problem with the models. I tried out a few other models, and finally I managed to get Hermes working with a few frontier models. The only problem after that was that those models were quite expensive, and given the heavy token usage of Hermes as shown above, cost was again shooting up quickly. I was once again forced to look for an alternative. Later, I found that the “Encrypted content is not supported with this model” error was actually a Hermes bug t https://github.com/NousResearch/hermes-agent/issues/23450 hat can be fixed by changing its config file Set include encrypted content: false in ~/.hermes/config.yaml Enter OpenRouter I finally switched to OpenRouter. If you’re not already familiar with it, OpenRouter is a gateway and proxy that lets you access hundreds of AI models. You integrate it once, and you can easily switch between just about any models at any time while keeping a single API key and billing. With OpenRouter, I was freely able to experiment with dozens of cheap models. I found a few models that work quite well for my use and cost me almost nothing. I’ve been using mainly Qwen-3.6-35B-A3B and DeepSeek-v4-Flash for all my Hermes profiles. The only critical feature I rely on Qwen-3.6 for is its multimodal capability, which DeepSeek-v4 lacks. I also tried out a few models that are listed as free, such as NVIDIA’s Nemotron, but I found that most of the time, the free models don’t work due to rate limiting. Incidentally, just recently I also found out that some time back Julien Chaumond had praised Qwen3.5-35B-A3B for its agentic capabilities. One interesting thing about OpenRouter which I admittedly did not know before using it is that OpenRouter does not own, run or host any AI models on its own servers. Instead, OpenRouter is more like a marketplace of different model inference providers. There are 3<sup rd</sup party interference providers such as Together AI, Fireworks AI, etc which actually own and manage all the infrastructure and run the models. OpenRouter simply routes your request to these providers as per your configuration. Apart from the flexibility it provides, there are a few more advantages of using OpenRouter. The OpenRouter Advantage One of the biggest advantages of OpenRouter is that it automatically manages failover at provider-level and model-level. For a model served by multiple upstream hosts, OpenRouter automatically retries the next available provider if the primary one throws an error or rate-limits your request. Failed provider attempts are not billed; you only pay for the execution path that completes. If an entire model or all its respective providers go down, you can also configure it to fall back to a different model. A second pretty neat thing about OpenRouter is that it does multiple levels of caching, which helps in reducing your usage bill. Firstly, it caches responses by storing them against a hash of your complete request. Any duplicate requests fetch the stored response immediately and do not hit the upstream LLMs at all. The second type of caching is especially useful for agent loops that use the large opening blocks of system prompts, tool definitions, etc., as shown above. Model providers such as Anthropic and OpenAI don’t charge for this repeated block of input tokens even if they’re sent each time in a multi-turn chat interaction. This is because they’re able to cache the precomputed values of those blocks once and use them on the subsequent turns within the same session. OpenRouter allows having sticky sessions where all messages of a single agent session can be routed to the same upstream provider so that you can leverage this discount on the heavy agent loops from the providers. If your requests from your agent session were routed to different providers, you’d be paying an order of magnitude more in your token bills. Hermes + OpenRouter = perfect match Hermes leverages all of the advantages of OpenRouter pretty neatly. It automatically generates and appends a session ID and sends it in every request of a multi-turn session. This allows OpenRouter to do sticky routing for every Hermes session. Hermes also automatically attaches OpenRouter-compatible caching controls to your system prompt, rules, and skill blocks, which allows OpenRouter and upstream providers to do prompt caching. Conclusion Although this post turned out to be a crash course on OpenRouter obviously not sponsored , none of this is to say that local models are pointless. Like I said before, if you already have a beefy machine and enough GPU/TPU power, do not use 4-5 different models at once and do not frequently change or upgrade your models that’d be super weird though , it might make sense to run models locally. But at all other times, I believe OpenRouter is your best bet.