Spend tokens where they actually matter.
LLM costs scale linearly with usage. A system processing 10,000 requests a day at $0.01 per request costs $100 daily — $365 a year. At enterprise scale, that’s over $10,000.
Cost optimization isn’t about cutting corners. It’s about spending tokens where they matter.
Every token you waste is a token you could have spent on a better answer.
Token budgeting #
The simplest way to control costs is to set limits. Per session, per task, or per day.
Strategy 1: Per-Session Budgets
Per-session budgets are straightforward:
class SessionBudget:
def __init__(self, budget_tokens: int = 10000):
self.budget = budget_tokens
self.used = 0
def allocate(self, tokens: int) -> bool:
if self.used + tokens <= self.budget:
self.used += tokens
return True
return False
def remaining(self) -> int:
return self.budget - self.used
Strategy 2: Per-Task Budgets
Per-task budgets are more useful. Different tasks need different amounts of context:
task_budgets:
classify:
max_tokens: 100
model: qwen2.5-1.5b
summarize:
max_tokens: 500
model: qwen2.5-7b
code_review:
max_tokens: 2000
model: qwen2.5-coder-7b
reason:
max_tokens: 4000
model: qwen2.5-32b
Strategy 3: Adaptive Budgets
Adaptive budgets adjust based on what actually happens. If classification tasks consistently use 80 tokens, stop allocating 100:
class AdaptiveBudget:
def __init__(self):
self.task_history = {}
def allocate(self, task_type: str) -> int:
if task_type in self.task_history:
return int(self.task_history[task_type] * 1.5)
return 1000
def record(self, task_type: str, tokens_used: int):
if task_type not in self.task_history:
self.task_history[task_type] = tokens_used
else:
self.task_history[task_type] = (
0.9 * self.task_history[task_type] + 0.1 * tokens_used
)
The exponential moving average (0.9 weight) means recent usage matters more than history. Adjust the weight based on how volatile your workloads are.
API vs local inference #
Local inference is cheaper at scale. The break-even depends on your hardware and API rates.
| Model | API ($/M tokens) | Local cost/hour | Break-even |
|---|---|---|---|
| GPT-4o | $2.50 / $10.00 | — | N/A |
| Claude Sonnet 4 | $3.00 / $15.00 | — | N/A |
| Qwen2.5-72B | $0.50 / $2.00 | ~$0.50 | ~4 hours/day |
| Qwen2.5-32B | $0.30 / $1.20 | ~$0.20 | ~2 hours/day |
| Qwen2.5-7B | $0.10 / $0.40 | ~$0.05 | ~1 hour/day |
The hardware math:
| Hardware | Upfront | Monthly electricity | Break-even vs API |
|---|---|---|---|
| RTX 3090 (used) | $600 | $15 | ~4 months |
| RTX 4090 | $1,500 | $20 | ~6 months |
| RTX 5080 | $1,000 | $18 | ~5 months |
| DGX Spark | $2,000 | $30 | ~8 months |
At moderate usage — an hour or more per day — local inference pays for itself. At high usage, the savings are dramatic. The catch is upfront capital. A RTX 5080 is $1,000. An API bill you can . Hardware you can’t.
Fallback strategies #
When your preferred model is too expensive or too slow, fall back to something cheaper. The key is knowing when quality is “good enough.”
Strategy 1: Quality-Based Fallback
Quality-based fallback tries models until the output meets a threshold:
class QualityFallback:
def __init__(self, quality_threshold: float = 0.8):
self.threshold = quality_threshold
self.models = [
{"model": "claude-sonnet-4", "cost": 0.015},
{"model": "qwen2.5-72b", "cost": 0.002},
{"model": "qwen2.5-32b", "cost": 0.001},
{"model": "qwen2.5-7b", "cost": 0.0004},
]
def route(self, prompt: str) -> str:
for model_config in self.models:
result = self.call_model(model_config["model"], prompt)
if self.evaluate_quality(result) >= self.threshold:
return result
return self.call_model(self.models[0]["model"], prompt)
The problem is evaluation itself. How do you measure quality without calling another model? Some systems use a small classifier. Others use heuristic checks — length, structure, keyword presence. None of these are perfect.
Strategy 2: Latency-Based Fallback
Latency-based fallback is simpler. Route to the fastest model that meets your time budget:
class LatencyFallback:
def __init__(self, max_latency: float = 5.0):
self.max_latency = max_latency
self.models = [
{"model": "qwen2.5-1.5b", "latency": 0.5},
{"model": "qwen2.5-7b", "latency": 2.0},
{"model": "qwen2.5-32b", "latency": 10.0},
{"model": "claude-sonnet-4", "latency": 5.0},
]
def route(self, prompt: str) -> str:
for model_config in sorted(self.models, key=lambda x: x["latency"]):
if model_config["latency"] <= self.max_latency:
return self.call_model(model_config["model"], prompt)
return self.call_model(self.models[0]["model"], prompt)
Caching #
Caching is the most underrated cost optimization. Identical prompts happen more often than you think — classification requests, FAQ-style queries, repeated tool calls.
Strategy 1: Prompt Caching
Exact prompt caching is simple:
import hashlib
class PromptCache:
def __init__(self, max_size: int = 1000):
self.cache = {}
self.max_size = max_size
def get(self, prompt: str) -> str | None:
key = hashlib.sha256(prompt.encode()).hexdigest()
return self.cache.get(key)
def set(self, prompt: str, response: str):
key = hashlib.sha256(prompt.encode()).hexdigest()
if len(self.cache) >= self.max_size:
self.cache.pop(next(iter(self.cache)))
self.cache[key] = response
Strategy 2: Semantic Caching
Semantic caching is more useful. It catches prompts that are different but mean the same thing:
from sentence_transformers import SentenceTransformer
class SemanticCache:
def __init__(self, similarity_threshold: float = 0.95):
self.model = SentenceTransformer('all-MiniLM-L6-v2')
self.cache = {}
self.threshold = similarity_threshold
def get(self, prompt: str) -> str | None:
prompt_embedding = self.model.encode([prompt])[0]
for cached_prompt, cached_response in self.cache.items():
cached_embedding = self.model.encode([cached_prompt])[0]
similarity = self.cosine_similarity(
prompt_embedding, cached_embedding
)
if similarity >= self.threshold:
return cached_response
return None
def set(self, prompt: str, response: str):
self.cache[prompt] = response
The threshold matters. 0.95 is aggressive — only very similar prompts match. 0.85 is more forgiving but risks returning wrong answers. Measure your miss rate and adjust.
Response caching for common queries is worth it too. If users ask “what’s the weather” or “what time is it” repeatedly, cache the pattern, not just the exact prompt:
class ResponseCache:
def __init__(self):
self.common_queries = {
"what is the weather": "Check weather API",
"what is the time": "Check system time",
"who is the president": "Check current president",
}
def get(self, query: str) -> str | None:
query_lower = query.lower()
for common_query, response in self.common_queries.items():
if common_query in query_lower:
return response
return None
This isn’t sophisticated, but it works. Common queries are common for a reason.
When optimization helps #
Optimization matters when you’re processing high volumes, running mixed workloads, or paying API costs that add up.
It doesn’t matter when you’re prototyping, using a single model, or processing low volumes. The complexity of budgeting, fallback, and caching isn’t worth it for a system that makes 100 requests a day.
Get the basic flow working first. Add optimization when the bill comes in.
Tradeoffs #
| Strategy | Cost | Quality | Complexity |
|---|---|---|---|
| No optimization | Highest | Consistent | Lowest |
| Token budgeting | Moderate | Variable | Medium |
| Fallback models | Low-Medium | Variable | Medium |
| Caching | Lowest | High (for cache hits) | Medium |
| Hybrid | Optimized | Optimized | Highest |
Production systems usually run hybrid. Budget per session, fall back on quality or latency, cache what you can. The complexity is real, but so are the savings.
Related #
Model Routing Strategies— capability-based, cost-aware, latency-aware routingLLM Guardrails in Practice— input validation, output filtering, safetyMulti-Model System Design— architecture for multiple modelsLLM Architecture— system design pillar: routing, cost, guardrails, and orchestration