{"slug": "how-to-reduce-llm-costs-without-sacrificing-quality", "title": "How to reduce LLM costs without sacrificing quality", "summary": "Arize AI outlines a cost-optimization workflow for LLM applications, emphasizing that reducing spend requires tracing complete workflows, comparing cost per validated outcome rather than cost per token, and validating changes against a fixed quality bar. The guide advises adding cost data to traces, investigating model choice, context growth, retries, tool output, and evaluator usage before making broad changes, and warns that a cheaper model can become more expensive overall if it requires more context, retries, or agent turns.", "body_md": "**Key takeaways**\n\n- Add cost to your traces so you can connect spend to the request, model call, tool, or agent step that produced it.\n- Compare cost and quality for the same traffic before deciding what to cut.\n- Investigate model choice, context growth, retries, tool output, and evaluator usage before making broad changes.\n- Validate every optimization against a fixed dataset and the same quality bar.\n\nYour LLM bill went up. The invoice tells you by how much, but not which behavior caused the increase or whether the additional spend improved the result.\n\nA spike might come from a harder class of request, a retry loop, a bloated tool response, or a more expensive model that produces no measurable improvement. Some of that spend is waste; some of it is doing valuable work.\n\nA cheaper model is not automatically a cheaper system, a pattern we also saw when [cheap models changed multi-agent economics](https://arize.com/blog/how-cheap-models-changed-multi-agent-economics/). When it needs more context, retries, agent turns, or human rework to finish the task, the total can climb past that of a more capable model. The useful unit is not [cost per token](https://arize.com/blog/why-ai-token-costs-dont-tell-you-if-your-ai-is-working/) but [cost per validated outcome](https://arize.com/blog/cost-per-successful-task-ai-model-benchmark).\n\nReducing that cost while protecting the spend that improves the application takes cost data, traces, and evaluation results in one workflow. Together they let you find the expensive pattern, test one change, and confirm that the savings do not come at the expense of quality.\n\n**Why LLM cost optimization starts with traces**\n\nCoding agents and production applications may land on the same AI bill, but they generate spend in different ways.\n\nA coding agent might repeatedly inspect files, carry a growing repository context, or explore several approaches before opening a pull request. A production agent might retrieve customer data, route between subagents, call tools, and generate a final response.\n\nIn both cases, cost comes down to two inputs:\n\n**Model price:** The rate for input, output, cached, or reasoning tokens.**Token usage:** Everything processed across prompts, history, tool results, retries, and responses.\n\nAt a high level:\n\n```\nTotal request cost =\nΣ(tokens processed by type × model rate for that token type)\n```\n\nA lower-priced model does not guarantee a cheaper workflow, though. When that model needs more context, retries, or agent turns to complete the task, the total cost can climb past that of a more capable model.\n\nModel price therefore explains only part of the bill. You also need to know how the workflow behaved and whether the result passed its quality checks.\n\nBefore changing anything, define what a successful result looks like. For a coding agent, that might be a merged pull request with fewer review cycles. For a support agent, it might be a correct resolution without escalation.\n\nChoose one primary quality metric and a small set of supporting metrics, and make those become the guardrails for every optimization that follows.\n\n**Step 1: Trace the complete workflow**\n\nStart by [instrumenting the full path](https://arize.com/docs/ax/get-started/get-started-tracing) to the answer, including model calls, retrieval steps, tools, routing decisions, and agent steps.\n\nA useful trace should capture:\n\n- Model and provider\n- Input and output token counts\n- Tool calls and responses\n- Retries and repeated model calls\n- Latency\n- Final output\n- Relevant evaluation results\n\nFor production applications, use OpenTelemetry-compatible instrumentation to send traces to an Arize AX project.\n\nFor coding agents, install the appropriate [tracing integration](https://arize.com/docs/ax/set-up-with-ai-assistants), connect it to the project, and run a test interaction. Confirm that the resulting trace includes the agent’s model calls, tool activity, and intermediate steps.\n\n**Instrument your agent in one command**\n\nThe AX [auto-instrumentation](https://arize.com/docs/ax/instrument/set-up-tracing) docs describe the `npx evals`\n\nflow. Run this from your project root to launch a supported coding agent, connect to or create an Arize AX account, install the needed tracing tooling, instrument your app or agent workflow, and confirm that your first traces land.\n\n```\nnpx evals\n```\n\nThat visibility exposes behavior an invoice cannot, such as repeated file reads, runaway context growth, stalled tools, [unnecessary retries](https://arize.com/blog/how-signal-found-two-hidden-retry-loops-in-alyx/), or an expensive session that never produces a usable change.\n\n**Step 2: Attach pricing and find the expensive spans**\n\nToken counts show how much work the system performed, but to understand spend, those tokens need the correct rates.\n\nIn AX, configure the applicable input, output, cached-token, and reasoning-token prices for each model. This also lets teams use negotiated rates when they differ from public pricing.\n\nThe [AX cost tracking docs](https://arize.com/docs/ax/instrument/track-costs) cover this behavior: when an incoming span already contains cost attributes, AX uses them. Otherwise, it matches the model and provider to the configured rates. AX includes default cost configurations for common models, and cost is not retroactive, so configure pricing before ingesting the traces you want to analyze.\n\nThe main cost attributes are `llm.cost.prompt`\n\n, `llm.cost.completion`\n\n, and `llm.cost.total`\n\n. AX adds the cost of the LLM spans in a trace to show the total cost of the request.\n\nNext, open a representative trace and work backward from total cost to the span responsible for it.\n\nInspect:\n\n- Total trace cost\n- Cost by span\n- Prompt and completion tokens\n- Cached-token usage\n- Tool-response size\n- Retry count\n- Model and provider\n- Evaluation results\n\nThen ask three questions:\n\n**Which span created most of the cost?****Why did it process so many tokens or require so many calls?****Did the request still meet its quality requirement?**\n\nA tool that returns five years of data to answer a question about the most recent quarter needs a different fix from a difficult request that genuinely benefits from a stronger model. The trace tells you which case you are looking at.\n\n**Step 3: Monitor cost beside quality**\n\nIndividual traces explain a request, but monitors reveal whether the pattern is spreading.\n\nTrack metrics such as:\n\n- Total LLM cost\n- Cost per request\n- Prompt tokens per request\n- Number of model calls\n- Retry count\n- Tool-call count\n- Latency\n- Evaluation pass rate\n\nWhen a monitor crosses its threshold, open the traces that contributed to the change.\n\nA tracing project monitor can use a span attribute or a custom Arize Query Language (AQL) metric; the [custom metrics docs](https://arize.com/docs/ax/observe/custom-metrics) show how to define AQL metrics when you need to filter or aggregate the source data first. This gives you a direct path from a production spike to the model, tool, context, or retry pattern behind it.\n\nCost should then be compared with quality for the same traffic and time window, the same shift from tokens to outcomes described in [agent evaluation metrics](https://arize.com/guides/ai-agent-handbook/agent-evaluation-metrics/):\n\nCost |\nQuality |\nRecommended action |\n|---|---|---|\n| High | Weak | Cut or simplify the pattern |\n| High | Strong | Investigate carefully and protect valuable spend |\n| Low | Weak | Fix the workflow or upgrade the model, context, or tools |\n| Low | Strong | Reproduce the pattern across similar workloads |\n\nThis framework keeps teams from treating every expensive request as waste. The target is the lowest-cost pattern that continues to meet the application’s quality threshold.\n\n**Step 4: Investigate recurring waste with managed agents**\n\nA monitor can flag the problem, but someone still has to inspect the traces, find the recurring behavior, and propose a change.\n\nManaged agents can automate much of that investigation. In the webinar workflow, the agent runs in an isolated sandbox with access to the traces and evaluations from an AX project. It can also receive additional context from a GitHub repository or relevant skills. The [managed agent harness docs](https://arize.com/docs/ax/agents/connect-your-harness) describe how the harness, sandbox, optional skills, and repository context are configured.\n\nTwo templates are especially useful.\n\n**Cost Agent**\n\nUse [Cost Agent](https://arize.com/docs/ax/agents/cost-agent) when the investigation is primarily about spend. It can identify:\n\n- Expensive models handling low-complexity work\n- Oversized prompts and tool responses\n- Redundant model or tool calls\n- High-cost tasks or time windows\n- Potential routing changes\n- Estimated savings from proposed fixes\n\n**Signal Agent**\n\nUse [Signal Agent](https://arize.com/docs/ax/observe/signal) for broader behavioral analysis. It can surface:\n\n- Runaway context accumulation\n- Repeated exploration\n- Slow or stalled tools\n- Inefficient routing\n- Sessions that fail to compact or restart\n- Recurring failures that increase cost or latency\n\nCost Agent focuses on the bill, and Signal Agent looks more broadly at the behavior producing it. To run that analysis on your own traces, follow the [Signal tutorial](https://arize.com/blog/debug-production-ai-agents-with-signal-tutorial/).\n\n**Example: Context bloat hiding inside a small model**\n\nIn the webinar demonstration, Cost Agent analyzed more than 500 spans. The application was already running on a relatively small model, so switching models was not the biggest opportunity.\n\nThe agent instead found a financial-data tool returning more than five years of history. Twenty-eight spans contained more than 5,000 prompt tokens, cost more than seven times the average, and represented roughly 24% of the measured bill.\n\nIt also found an oversized supervisor prompt and a redundant tool whose information was already available elsewhere.\n\nThe proposed fix was small: return only the two most recent financial periods, shorten the supervisor prompt, and remove the duplicate tool. The agent estimated that those changes could reduce the measured bill by approximately 34%.\n\nThat percentage came from a demonstration workload, rather than a universal benchmark. The broader lesson is more useful: context and tool design can create a larger cost problem than the model itself.\n\n**Step 5: Fix the pattern the trace reveals**\n\nMost LLM cost problems fall into a handful of categories.\n\n**Route by measured difficulty**\n\nUse the smallest model that continues to pass the quality bar. Simple classification or extraction may work well on a smaller model, while complex reasoning or difficult refactors may still justify a frontier model.\n\nTest that assumption with data instead of routing solely by model price. The same evals-first approach applies when [switching to a cheaper model as inference costs become visible](https://arize.com/blog/ai-model-subsidies-ending-llm-inference-costs/).\n\n**Reduce unnecessary context**\n\nLook for old conversation history, repeated system instructions, oversized tool responses, irrelevant repository files, and information another tool has already returned.\n\nFetch context when the task needs it instead of carrying every possible input through the workflow.\n\n**Shorten agent loops**\n\nRepeated searches, file reads, retries, and subagent turns can increase spend without improving the answer.\n\nUse the traces to set clearer stopping conditions, retry limits, context ceilings, or compaction rules.\n\n**Cache stable prompt content**\n\nSeparate stable instructions and reference material from request-specific input. When the provider supports [prompt caching](https://arize.com/blog/prompt-caching-analysis), this can reduce the cost of repeatedly processing the same context.\n\n**Right-size your evaluations**\n\nEvaluation spend can become material, especially when every request triggers an LLM judge.\n\nStart with the least expensive evaluator that reliably measures the behavior. The [evaluator type](https://arize.com/docs/ax/concepts/evaluators/evaluator-types) docs cover code, human, and LLM-as-judge evaluators, and the [results and costs docs](https://arize.com/docs/ax/evaluate/results-and-costs) cover evaluation cost, sampling, and filtering.\n\n- Use deterministic checks for requirements that can be expressed in code.\n- Combine code and an LLM when only part of the task requires semantic judgment.\n- Use LLM judges for subjective or open-ended criteria.\n- Reserve\n[agent-as-a-judge](https://arize.com/blog/agent-as-a-judge-agentic-evaluation/)workflows for evaluations that require multi-step investigation.\n\nSampling and filtering can further limit expensive evaluations to the traffic where they provide the most value.\n\nArize AX measures the effect of these changes. Routing, caching, and workflow modifications still happen in your application, gateway, or agent harness.\n\n**Step 6: Validate the optimization before shipping**\n\nA promising trace is enough to form a hypothesis. It is not enough to deploy the change.\n\nCreate a dataset that includes:\n\n- Common requests\n- Expensive requests\n- Known failures\n- Difficult edge cases\n- Requests the current system handles well\n\nRun the current workflow as the baseline. Then create a candidate that changes one variable, such as the model, prompt, context size, tool output, retry policy, routing logic, or evaluator. The [AX datasets docs](https://arize.com/docs/ax/develop/datasets) describe the dataset side of this workflow. Compare both versions using the [experiment workflow](https://arize.com/docs/ax/develop/tutorial/iteration-workflow-experiments).\n\nEvaluate both versions against the same dataset and the same quality criteria. That is the same loop used in [evaluation-driven development](https://arize.com/blog/evaluation-driven-development-ai-agents-production/): change one variable, measure cost and quality, then decide.\n\nCandidate change |\nCost measurement |\nQuality guardrail |\n|---|---|---|\n| Use a smaller model | Cost per request | Task success |\n| Trim context | Prompt tokens | Answer correctness |\n| Limit tool output | Tool and prompt tokens | Completeness |\n| Reduce retries | Calls and latency | Successful completion |\n| Sample an evaluator | Evaluation cost | Regression coverage |\n\nAccept the candidate only when it reduces cost and continues to meet the required quality threshold.\n\nThen follow the normal delivery process:\n\n- Review the proposed code, prompt, tool, or configuration change.\n- Run CI and the relevant evaluation dataset.\n- Confirm that cost fell without a meaningful quality regression.\n- Require human approval.\n- Deploy and continue monitoring production traffic.\n\nWhen a cheaper pattern works consistently, encode it as a shared skill, routing rule, prompt template, tool-response limit, or agent configuration. That lets the entire team benefit from the optimization instead of rediscovering it one session at a time.\n\n**A repeatable LLM cost optimization workflow**\n\nThe steps below distill the entire approach into a single loop you can run whenever costs climb:\n\nA lower bill only counts when the application still produces the result users need.\n\nOnce traces, evaluations, and experiments are connected, cost spikes become debuggable production behavior. You can see where the money went, determine whether it created value, and ship a fix with evidence behind it.\n\nSee this whole workflow in action in our [webinar on reducing LLM costs without sacrificing quality](https://www.youtube.com/watch?v=Pg5PaM7xa8o&feature=youtu.be). Then bring the same cost, trace, and evaluation visibility to your own application with [Arize AX](https://arize.com/products/ax/).", "url": "https://wpnews.pro/news/how-to-reduce-llm-costs-without-sacrificing-quality", "canonical_source": "https://arize.com/blog/how-to-reduce-llm-costs-without-sacrificing-quality/", "published_at": "2026-08-31 16:23:20+00:00", "updated_at": "2026-08-31 16:54:19.968400+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "mlops", "ai-tools"], "entities": ["Arize AI", "Arize AX", "OpenTelemetry"], "alternates": {"html": "https://wpnews.pro/news/how-to-reduce-llm-costs-without-sacrificing-quality", "markdown": "https://wpnews.pro/news/how-to-reduce-llm-costs-without-sacrificing-quality.md", "text": "https://wpnews.pro/news/how-to-reduce-llm-costs-without-sacrificing-quality.txt", "jsonld": "https://wpnews.pro/news/how-to-reduce-llm-costs-without-sacrificing-quality.jsonld"}}