I spent weeks assuming that picking the biggest model was the safe move. Then I actually measured it.
I run a single OpenAI-compatible endpoint that routes across 50+ models with auto/eco/premium tiers, and the first thing the logs showed me was that agents don't need the biggest model — they need the right one. A classification task that cost $0.08 on a frontier model cost $0.002 on a mid-tier one with identical accuracy on my eval set. The frontier model wasn't better; it was just more expensive at being the same.
The second lesson: latency matters more than agents admit. A 4-second response on a premium model breaks a tool-calling loop that a 900ms eco-tier response sails through. I started tagging every call with a tier in my traces and found that roughly 70% of my traffic could run on eco without any user-visible difference. The remaining 30% — long-context synthesis, multi-step reasoning — genuinely needed the big guns.
The third lesson was the uncomfortable one: model choice is a product decision, not a technical one. I let agents pick freely at first and the bills got weird fast. Now the default is eco, escalation is explicit, and premium is reserved for tasks where I've measured that smaller models actually fail. 'Use the best model' is how you burn budget. 'Use the smallest model that passes my evals' is how you ship.
If you're building agent tooling, instrument before you optimize. I packaged the endpoint at https://x402.freeq.one/tools/llm_chat.html so the routing tiers are a config change, not a rewrite — and the model list at /tools/llm_chat/models is worth reading before you hardcode anything.