Start by measuring cost per successful task. Then remove duplicate work and unnecessary retries, reduce avoidable input and output tokens, and test whether a less expensive model meets your quality requirements. Evaluate caching and batching for suitable workloads. Check for unauthorized usage separately: a large bill is not proof of abuse.
What is actually driving my inference cost? #
Break usage down by workload, model, input tokens, output tokens, and attempts per completed task. Distinguish recorded estimates from the provider invoice. Check for tool charges, cache charges, infrastructure, and other billed items where applicable. If the bill changed suddenly, begin with why your AI bill jumped before changing the whole architecture.
For simple token-priced requests, the starting calculation is input tokens multiplied by the input rate plus output tokens multiplied by the output rate. Use matching units. Separate cached-token categories and other charges when your provider bills them differently; one blended token rate can hide the cause of a change.
Which cost changes should I try first? #
| What you find | Change to evaluate | What to measure alongside cost |
|---|---|---|
| The same task is submitted repeatedly | Deduplicate jobs and fix retry or timeout behavior | Missing work, retry success, and duplicate side effects |
| Long context dominates input usage | Remove irrelevant context; test targeted retrieval or summarization | Answer accuracy and information lost during compression |
| Responses are longer than the product needs | Request an appropriate response length and test output limits | Truncation and task completion |
| A costly model handles simple tasks | Evaluate a smaller model or route tasks by difficulty | Failure rate, fallback rate, and latency |
| A large prompt prefix repeats | Evaluate the provider's prompt or context cache | Hit rate, write/storage charges, and expiration |
| Jobs are not time-sensitive | Compare eligible batch processing with interactive processing | Turnaround time, failures, and actual billed rates |
Will a smaller model always save money? #
No. A lower request price can be offset by retries, fallback to a larger model, or manual review. Compare the total cost of completing the same task at an acceptable quality level. Include the cost of all attempts, not only the successful response.
Synthetic example: a baseline processes 10,000 tasks at $0.02 per task, totaling $200. A smaller model costs $0.005 per initial attempt, totaling $50. If 20% of tasks require one $0.02 fallback, the model-call total becomes $90. That is $110 less under these assumptions, before evaluation, review, and infrastructure costs. It is not a prediction of savings for your workload.
Does caching mean I can reuse every answer? #
No. Prompt-prefix caching and application response caching solve different problems. Provider caching can reduce repeated input processing under provider-specific rules; it does not generally eliminate generation cost. Claude documents cache lifetimes and pricing behavior in its prompt-caching guide. Gemini documents its own behavior in context caching. Check current eligibility and prices rather than assuming the same discount across providers.
Application response caching reuses a previous answer. Only do that when the answer remains valid for the requesting user, permissions, input, and freshness requirements. Include the relevant tenant and authorization context in the cache design. A cheaper response that exposes another customer's data is not an acceptable optimization.
Can distillation lower inference costs? #
It can be worth evaluating for a repeated, well-defined task when you have permission to use the teacher outputs and the training data. Include data generation, training, evaluation, hosting, and maintenance in the comparison. A smaller student must meet your task's quality threshold; it is not automatically a cheaper replacement for every capability. Read how distillation works in practice.
What if the bill comes from stolen or shared access? #
Check whether the owner can reconcile the usage to approved jobs. For confirmed exposure, follow your containment process promptly. Otherwise investigate unfamiliar populations without equating a new network or larger token count with theft. Use the API-key investigation guide.
How do I know the optimization worked? #
Compare equivalent tasks before and after the change. Record total cost, completion rate, quality, latency, and fallback/retry rate. Change one major factor at a time where practical. Keep a rollback path for quality regressions. Use workload baselines for ongoing measurement and budgets and limits to bound exposure; a cap alone does not make a task more efficient.