# Agent cost management is about more than the model > Source: > Published: 2026-09-01 15:00:44+00:00 Every LLM call your application makes costs money, and agentic applications make a lot of LLM calls. A single user request fans out into planning, tool selection, retrieval, and synthesis steps, each one a model invocation, and the bill for all of it lands as one undifferentiated line item at the end of the month. While you’re prototyping, this doesn’t matter. The moment you have real traffic, it becomes one of the larger numbers on your infrastructure bill, and it has a nasty property that ordinary compute doesn’t: agent cost scales non-linearly with usage. Because context accumulates across turns, a tool that returns a large payload doesn’t cost you once, it costs you on every subsequent step in the same trace, because that payload is now part of the conversation being re-sent to the model. So you can see traffic double, but your inference costs will more than double as the shape of interactions changes. This makes cost a key part of your [observability story](https://arize.com/blog/how-to-reduce-llm-costs-without-sacrificing-quality/), and cost control a critical part of your scaling. It’s important that your observability tell you not just what’s expensive, but how to bring those costs down. **Cost optimization often loses priority to feature work** A common problem with optimizing production agent costs is that it’s nobody’s specific job, and it’s also lengthy and tedious work, so nobody volunteers to pick it up. The work isn’t rocket science: read the telemetry, rank spans by spend, then [correlate cost against quality scores](https://arize.com/blog/how-to-reduce-llm-costs-without-sacrificing-quality/) to find the steps where the spend isn’t worth it. Finally, trace the expensive spans back to the specific function that produced them and draft a small, well-scoped diff. This is a shape of problem that agents are good at solving. The evidence is all available as easily-accessed, structured data, the analysis is mechanical, and the decisions are evidence driven. The natural answer to expensive agents is another agent whose job is to control their spend. Even better, the process is a loop, so you can run it continuously and autonomously. Diagnosis is a mechanical process, the fixes are relatively small and quickly reviewed: trim a payload, shorten a prompt, add caching, collapse a redundant call, downgrade an over-provisioned model. Importantly, it’s also trivially verifiable: once you ship a change, exactly the same evidence will tell you if the cost really went down or not. Tedious to do, mechanical to solve, cheap to verify: this is ideal agent territory. **Let the agent cook** Arize AX now ships cost control as a [managed agent](https://arize.com/docs/ax/agents/cost-agent) that runs directly inside our platform. In the New Agent picker it sits alongside [Signal](https://arize.com/docs/ax/observe/signal) and the debugging agents. Configuring the cost agent is only a handful of choices. You choose which integration runs the agent. You pick the tracing project it’s allowed to read: here, live-fin-langgraph, a multi-agent financial assistant built with LangGraph, where a supervisor routes questions between agents that retrieve financial data, do web research, and produce the final summary. And you decide whether it runs once or on a schedule. There are some important choices here, though. The first is the repo connection: with a [GitHub skill](https://arize.com/docs/ax/agents/skills-and-permissions) attached, the agent can read the implementation, not just the telemetry. This allows it to not just diagnose the problem but create a PR which directly addresses it. The second is the automation mode: you can either try it out in a one-off session, or run it automatically on a schedule, allowing the loop to run on its own with minimal human intervention. Before launching you see the task prompt, and you can edit it. This is a great way to add additional domain context like constraints that aren’t obvious or strategies that have been previously tried. **Real-world results** The agent runs in a sandbox, with a live transcript you can watch and interrupt while it works. In this example, the run took 16 minutes and 62 steps. The first thing it reported was that there was nothing to report on the most obvious question: **30-day baseline:** 500 LLM spans, 100% gpt-4o-mini. Good news first: there’s no frontier model misuse. The model choice is already optimal. The most obvious thing to change when optimizing agents for cost is to [downgrade to a cheaper model](https://arize.com/blog/how-cheap-models-changed-multi-agent-economics/). Here, the tool has already gone past this naive strategy and started looking deeper. It broke spend down by node: the agent node doing financial data and web research accounted for 72% of cost, the supervisor 15%, the summarizer 12%. So it went looking at what the agent was doing. The agent node’s token distribution was (to use the agent’s own choice of words) brutal: p50 of 616 tokens, p95 of 10,240, max of 11,983. A 20x spread between the median call and the tail. It traced the tail to a single function: `read_webpage` in `tools.py` , which was returning up to 50,000 characters, roughly 12,500 tokens, per call. It also found the mechanism: that webpage content lands in LangGraph’s shared AgentState.messages and is re-sent to the model on every subsequent step in the same trace. One large page fetch inflates three to four downstream LLM calls. There were 71 of those calls in the sample. Three more findings followed. First, there were zero [prompt cache](https://arize.com/blog/prompt-caching-analysis) hits across 631,654 prompt tokens. The supervisor has only two unique system prompts, called 184 times, but at an average of 688 tokens they sit below OpenAI’s 1,024-token auto-cache floor, so none of that repetition was being cached. The summarizing node was using the wrong message role and re-ingesting the full conversation history, raw API payloads included. And two financial tools were returning multi-year time series when only the most recent period was needed. These are context management challenges, and the agent is smart enough to solve them directly. **Direct fixes** Because the repo was connected, the investigation ended in a [pull request](https://arize.com/blog/from-signal-to-pr/). Six changes, each with its own estimated saving: `read_webpage` truncated from 50K to 6K characters, the summarizer’s role and message filtering fixed, an `InMemoryCache` wired up via `set_llm_cache()` , the two financial tools capped at `limit=2` , the supervisor’s context trimmed to the last four messages, and the supervisor’s LLM split into its own instance so that per-role model routing becomes possible later. Combined, roughly 35-40% of the monthly bill. The last change is particularly interesting, since it’s not directly cost-related. It’s a refactor that makes a future optimization available. That’s the kind of thing a thorough engineer does while they’re already in the file, and not the kind of thing you’d expect from something pattern-matching on cost. The agent then closes the loop, by telling you what to watch after merging: the agent node’s p95 should drop from around 10,240 tokens to 2,500-3,000. It also finds where the next win is: the supervisor’s system prompt needs about 340 more tokens to cross the auto-cache threshold, which would make all 184 monthly routing calls eligible for cache hits. This is what you want from an agent doing this work. A ranked set of findings with the mechanism explained, a diff, a falsifiable prediction about what the telemetry should look like afterward, and a pointer to the next thing. **The human is still the one who ships** Nothing here involves an agent pushing to production on its own authority. The Cost Agent proposes; a pull request is a proposal. Your engineers read the diff and decide, and some of these they’ll reject: truncating `read_webpage` to 6K characters, for instance, is a judgment call about how much of a page the research agent needs, and that’s a product decision, not purely a cost decision. This is an agent doing what an agent should do: reading 500 spans, finding the 71 calls that mattered, and working out that the real problem was a tool payload being silently re-sent four times. The final calls on product changes remain reserved to human judgement. **Getting started** [Cost tracking](https://arize.com/docs/ax/instrument/track-costs) is generally available in AX today: it prices every LLM call at both span and trace level, with default configurations for common models and support for custom rates, cache tokens, reasoning tokens, and tiered pricing. Turn it on, confirm your models match the defaults, and you have the substrate the whole loop depends on. [Managed agents](https://arize.com/docs/ax/agents/agent-studio), including the [Cost Agent](https://arize.com/docs/ax/agents/cost-agent), are in Enterprise beta. If you’d like an agent auditing your traces every week instead of an engineer auditing them every quarter, contact your Arize account team.