How to reduce LLM costs without sacrificing quality

Arize AI outlines a cost-optimization workflow for LLM applications, emphasizing that reducing spend requires tracing complete workflows, comparing cost per validated outcome rather than cost per token, and validating changes against a fixed quality bar. The guide advises adding cost data to traces, investigating model choice, context growth, retries, tool output, and evaluator usage before making broad changes, and warns that a cheaper model can become more expensive overall if it requires more context, retries, or agent turns.

Key takeaways - Add cost to your traces so you can connect spend to the request, model call, tool, or agent step that produced it. - Compare cost and quality for the same traffic before deciding what to cut. - Investigate model choice, context growth, retries, tool output, and evaluator usage before making broad changes. - Validate every optimization against a fixed dataset and the same quality bar. Your LLM bill went up. The invoice tells you by how much, but not which behavior caused the increase or whether the additional spend improved the result. A spike might come from a harder class of request, a retry loop, a bloated tool response, or a more expensive model that produces no measurable improvement. Some of that spend is waste; some of it is doing valuable work. A cheaper model is not automatically a cheaper system, a pattern we also saw when cheap models changed multi-agent economics https://arize.com/blog/how-cheap-models-changed-multi-agent-economics/ . When it needs more context, retries, agent turns, or human rework to finish the task, the total can climb past that of a more capable model. The useful unit is not cost per token https://arize.com/blog/why-ai-token-costs-dont-tell-you-if-your-ai-is-working/ but cost per validated outcome https://arize.com/blog/cost-per-successful-task-ai-model-benchmark . Reducing that cost while protecting the spend that improves the application takes cost data, traces, and evaluation results in one workflow. Together they let you find the expensive pattern, test one change, and confirm that the savings do not come at the expense of quality. Why LLM cost optimization starts with traces Coding agents and production applications may land on the same AI bill, but they generate spend in different ways. A coding agent might repeatedly inspect files, carry a growing repository context, or explore several approaches before opening a pull request. A production agent might retrieve customer data, route between subagents, call tools, and generate a final response. In both cases, cost comes down to two inputs: Model price: The rate for input, output, cached, or reasoning tokens. Token usage: Everything processed across prompts, history, tool results, retries, and responses. At a high level: Total request cost = Σ tokens processed by type × model rate for that token type A lower-priced model does not guarantee a cheaper workflow, though. When that model needs more context, retries, or agent turns to complete the task, the total cost can climb past that of a more capable model. Model price therefore explains only part of the bill. You also need to know how the workflow behaved and whether the result passed its quality checks. Before changing anything, define what a successful result looks like. For a coding agent, that might be a merged pull request with fewer review cycles. For a support agent, it might be a correct resolution without escalation. Choose one primary quality metric and a small set of supporting metrics, and make those become the guardrails for every optimization that follows. Step 1: Trace the complete workflow Start by instrumenting the full path https://arize.com/docs/ax/get-started/get-started-tracing to the answer, including model calls, retrieval steps, tools, routing decisions, and agent steps. A useful trace should capture: - Model and provider - Input and output token counts - Tool calls and responses - Retries and repeated model calls - Latency - Final output - Relevant evaluation results For production applications, use OpenTelemetry-compatible instrumentation to send traces to an Arize AX project. For coding agents, install the appropriate tracing integration https://arize.com/docs/ax/set-up-with-ai-assistants , connect it to the project, and run a test interaction. Confirm that the resulting trace includes the agent’s model calls, tool activity, and intermediate steps. Instrument your agent in one command The AX auto-instrumentation https://arize.com/docs/ax/instrument/set-up-tracing docs describe the npx evals flow. Run this from your project root to launch a supported coding agent, connect to or create an Arize AX account, install the needed tracing tooling, instrument your app or agent workflow, and confirm that your first traces land. npx evals That visibility exposes behavior an invoice cannot, such as repeated file reads, runaway context growth, stalled tools, unnecessary retries https://arize.com/blog/how-signal-found-two-hidden-retry-loops-in-alyx/ , or an expensive session that never produces a usable change. Step 2: Attach pricing and find the expensive spans Token counts show how much work the system performed, but to understand spend, those tokens need the correct rates. In AX, configure the applicable input, output, cached-token, and reasoning-token prices for each model. This also lets teams use negotiated rates when they differ from public pricing. The AX cost tracking docs https://arize.com/docs/ax/instrument/track-costs cover this behavior: when an incoming span already contains cost attributes, AX uses them. Otherwise, it matches the model and provider to the configured rates. AX includes default cost configurations for common models, and cost is not retroactive, so configure pricing before ingesting the traces you want to analyze. The main cost attributes are llm.cost.prompt , llm.cost.completion , and llm.cost.total . AX adds the cost of the LLM spans in a trace to show the total cost of the request. Next, open a representative trace and work backward from total cost to the span responsible for it. Inspect: - Total trace cost - Cost by span - Prompt and completion tokens - Cached-token usage - Tool-response size - Retry count - Model and provider - Evaluation results Then ask three questions: Which span created most of the cost? Why did it process so many tokens or require so many calls? Did the request still meet its quality requirement? A tool that returns five years of data to answer a question about the most recent quarter needs a different fix from a difficult request that genuinely benefits from a stronger model. The trace tells you which case you are looking at. Step 3: Monitor cost beside quality Individual traces explain a request, but monitors reveal whether the pattern is spreading. Track metrics such as: - Total LLM cost - Cost per request - Prompt tokens per request - Number of model calls - Retry count - Tool-call count - Latency - Evaluation pass rate When a monitor crosses its threshold, open the traces that contributed to the change. A tracing project monitor can use a span attribute or a custom Arize Query Language AQL metric; the custom metrics docs https://arize.com/docs/ax/observe/custom-metrics show how to define AQL metrics when you need to filter or aggregate the source data first. This gives you a direct path from a production spike to the model, tool, context, or retry pattern behind it. Cost should then be compared with quality for the same traffic and time window, the same shift from tokens to outcomes described in agent evaluation metrics https://arize.com/guides/ai-agent-handbook/agent-evaluation-metrics/ : Cost | Quality | Recommended action | |---|---|---| | High | Weak | Cut or simplify the pattern | | High | Strong | Investigate carefully and protect valuable spend | | Low | Weak | Fix the workflow or upgrade the model, context, or tools | | Low | Strong | Reproduce the pattern across similar workloads | This framework keeps teams from treating every expensive request as waste. The target is the lowest-cost pattern that continues to meet the application’s quality threshold. Step 4: Investigate recurring waste with managed agents A monitor can flag the problem, but someone still has to inspect the traces, find the recurring behavior, and propose a change. Managed agents can automate much of that investigation. In the webinar workflow, the agent runs in an isolated sandbox with access to the traces and evaluations from an AX project. It can also receive additional context from a GitHub repository or relevant skills. The managed agent harness docs https://arize.com/docs/ax/agents/connect-your-harness describe how the harness, sandbox, optional skills, and repository context are configured. Two templates are especially useful. Cost Agent Use Cost Agent https://arize.com/docs/ax/agents/cost-agent when the investigation is primarily about spend. It can identify: - Expensive models handling low-complexity work - Oversized prompts and tool responses - Redundant model or tool calls - High-cost tasks or time windows - Potential routing changes - Estimated savings from proposed fixes Signal Agent Use Signal Agent https://arize.com/docs/ax/observe/signal for broader behavioral analysis. It can surface: - Runaway context accumulation - Repeated exploration - Slow or stalled tools - Inefficient routing - Sessions that fail to compact or restart - Recurring failures that increase cost or latency Cost Agent focuses on the bill, and Signal Agent looks more broadly at the behavior producing it. To run that analysis on your own traces, follow the Signal tutorial https://arize.com/blog/debug-production-ai-agents-with-signal-tutorial/ . Example: Context bloat hiding inside a small model In the webinar demonstration, Cost Agent analyzed more than 500 spans. The application was already running on a relatively small model, so switching models was not the biggest opportunity. The agent instead found a financial-data tool returning more than five years of history. Twenty-eight spans contained more than 5,000 prompt tokens, cost more than seven times the average, and represented roughly 24% of the measured bill. It also found an oversized supervisor prompt and a redundant tool whose information was already available elsewhere. The proposed fix was small: return only the two most recent financial periods, shorten the supervisor prompt, and remove the duplicate tool. The agent estimated that those changes could reduce the measured bill by approximately 34%. That percentage came from a demonstration workload, rather than a universal benchmark. The broader lesson is more useful: context and tool design can create a larger cost problem than the model itself. Step 5: Fix the pattern the trace reveals Most LLM cost problems fall into a handful of categories. Route by measured difficulty Use the smallest model that continues to pass the quality bar. Simple classification or extraction may work well on a smaller model, while complex reasoning or difficult refactors may still justify a frontier model. Test that assumption with data instead of routing solely by model price. The same evals-first approach applies when switching to a cheaper model as inference costs become visible https://arize.com/blog/ai-model-subsidies-ending-llm-inference-costs/ . Reduce unnecessary context Look for old conversation history, repeated system instructions, oversized tool responses, irrelevant repository files, and information another tool has already returned. Fetch context when the task needs it instead of carrying every possible input through the workflow. Shorten agent loops Repeated searches, file reads, retries, and subagent turns can increase spend without improving the answer. Use the traces to set clearer stopping conditions, retry limits, context ceilings, or compaction rules. Cache stable prompt content Separate stable instructions and reference material from request-specific input. When the provider supports prompt caching https://arize.com/blog/prompt-caching-analysis , this can reduce the cost of repeatedly processing the same context. Right-size your evaluations Evaluation spend can become material, especially when every request triggers an LLM judge. Start with the least expensive evaluator that reliably measures the behavior. The evaluator type https://arize.com/docs/ax/concepts/evaluators/evaluator-types docs cover code, human, and LLM-as-judge evaluators, and the results and costs docs https://arize.com/docs/ax/evaluate/results-and-costs cover evaluation cost, sampling, and filtering. - Use deterministic checks for requirements that can be expressed in code. - Combine code and an LLM when only part of the task requires semantic judgment. - Use LLM judges for subjective or open-ended criteria. - Reserve agent-as-a-judge https://arize.com/blog/agent-as-a-judge-agentic-evaluation/ workflows for evaluations that require multi-step investigation. Sampling and filtering can further limit expensive evaluations to the traffic where they provide the most value. Arize AX measures the effect of these changes. Routing, caching, and workflow modifications still happen in your application, gateway, or agent harness. Step 6: Validate the optimization before shipping A promising trace is enough to form a hypothesis. It is not enough to deploy the change. Create a dataset that includes: - Common requests - Expensive requests - Known failures - Difficult edge cases - Requests the current system handles well Run the current workflow as the baseline. Then create a candidate that changes one variable, such as the model, prompt, context size, tool output, retry policy, routing logic, or evaluator. The AX datasets docs https://arize.com/docs/ax/develop/datasets describe the dataset side of this workflow. Compare both versions using the experiment workflow https://arize.com/docs/ax/develop/tutorial/iteration-workflow-experiments . Evaluate both versions against the same dataset and the same quality criteria. That is the same loop used in evaluation-driven development https://arize.com/blog/evaluation-driven-development-ai-agents-production/ : change one variable, measure cost and quality, then decide. Candidate change | Cost measurement | Quality guardrail | |---|---|---| | Use a smaller model | Cost per request | Task success | | Trim context | Prompt tokens | Answer correctness | | Limit tool output | Tool and prompt tokens | Completeness | | Reduce retries | Calls and latency | Successful completion | | Sample an evaluator | Evaluation cost | Regression coverage | Accept the candidate only when it reduces cost and continues to meet the required quality threshold. Then follow the normal delivery process: - Review the proposed code, prompt, tool, or configuration change. - Run CI and the relevant evaluation dataset. - Confirm that cost fell without a meaningful quality regression. - Require human approval. - Deploy and continue monitoring production traffic. When a cheaper pattern works consistently, encode it as a shared skill, routing rule, prompt template, tool-response limit, or agent configuration. That lets the entire team benefit from the optimization instead of rediscovering it one session at a time. A repeatable LLM cost optimization workflow The steps below distill the entire approach into a single loop you can run whenever costs climb: A lower bill only counts when the application still produces the result users need. Once traces, evaluations, and experiments are connected, cost spikes become debuggable production behavior. You can see where the money went, determine whether it created value, and ship a fix with evidence behind it. See this whole workflow in action in our webinar on reducing LLM costs without sacrificing quality https://www.youtube.com/watch?v=Pg5PaM7xa8o&feature=youtu.be . Then bring the same cost, trace, and evaluation visibility to your own application with Arize AX https://arize.com/products/ax/ .