How Intelligent Query Planning Can Reduce LLM Costs in Production AI Agents

A senior AI engineer argues that the biggest source of unnecessary LLM cost in production AI agents is not the model choice but the retrieval strategy, citing that a smarter query planning approach can save more tokens than switching to a smaller model. The engineer illustrates this with an example of an AI engineering assistant that queries multiple sources like GitHub, Jira, and Confluence, where a typical agent retrieves excessive information before invoking the LLM.

Member-only story How Intelligent Query Planning Can Reduce LLM Costs in Production AI Agents Most AI agents don’t waste money because of the model they use. They waste money because they retrieve too much information before asking the model to think. Everyone talks about choosing cheaper models, writing better prompts, or reducing context size to cut LLM costs. But after working on enterprise AI agents, I noticed something else. non members can read here The biggest source of unnecessary cost wasn’t the LLM. It was how the agent decided what to retrieve before calling the LLM. A smarter retrieval strategy can often save more tokens than switching to a smaller model. A Real Example Imagine you’re building an AI engineering assistant connected to: - GitHub - Jira - Confluence - Logs - Knowledge Graph - Internal documentation A developer asks: “Why is the checkout service failing after yesterday’s deployment?” A typical agent might do this: Search GitHub↓Search Jira↓Search…