# Cost Optimization for LLM Systems: Where the Money Actually Goes

> Source: <https://www.glukhov.org/llm-architecture/cost-optimization/cost-optimization-for-llm-systems/>
> Published: 2026-06-15 00:00:00+00:00

# Cost Optimization for LLM Systems: Where the Money Actually Goes

Spend tokens where they actually matter.

LLM costs scale linearly with usage. A system processing 10,000 requests a day at $0.01 per request costs $100 daily — $365 a year. At enterprise scale, that’s over $10,000.

Cost optimization isn’t about cutting corners. It’s about spending tokens where they matter.

Every token you waste is a token you could have spent on a better answer.

## Token budgeting

The simplest way to control costs is to set limits. Per session, per task, or per day.

### Strategy 1: Per-Session Budgets

Per-session budgets are straightforward:

``` python
class SessionBudget:
    def __init__(self, budget_tokens: int = 10000):
        self.budget = budget_tokens
        self.used = 0

    def allocate(self, tokens: int) -> bool:
        if self.used + tokens <= self.budget:
            self.used += tokens
            return True
        return False

    def remaining(self) -> int:
        return self.budget - self.used
```

### Strategy 2: Per-Task Budgets

Per-task budgets are more useful. Different tasks need different amounts of context:

```
task_budgets:
  classify:
    max_tokens: 100
    model: qwen2.5-1.5b
  summarize:
    max_tokens: 500
    model: qwen2.5-7b
  code_review:
    max_tokens: 2000
    model: qwen2.5-coder-7b
  reason:
    max_tokens: 4000
    model: qwen2.5-32b
```

### Strategy 3: Adaptive Budgets

Adaptive budgets adjust based on what actually happens. If classification tasks consistently use 80 tokens, stop allocating 100:

``` python
class AdaptiveBudget:
    def __init__(self):
        self.task_history = {}

    def allocate(self, task_type: str) -> int:
        if task_type in self.task_history:
            return int(self.task_history[task_type] * 1.5)
        return 1000

    def record(self, task_type: str, tokens_used: int):
        if task_type not in self.task_history:
            self.task_history[task_type] = tokens_used
        else:
            self.task_history[task_type] = (
                0.9 * self.task_history[task_type] + 0.1 * tokens_used
            )
```

The exponential moving average (0.9 weight) means recent usage matters more than history. Adjust the weight based on how volatile your workloads are.

## API vs local inference

Local inference is cheaper at scale. The break-even depends on your hardware and API rates.

| Model | API ($/M tokens) | Local cost/hour | Break-even |
|---|---|---|---|
| GPT-4o | $2.50 / $10.00 | — | N/A |
| Claude Sonnet 4 | $3.00 / $15.00 | — | N/A |
| Qwen2.5-72B | $0.50 / $2.00 | ~$0.50 | ~4 hours/day |
| Qwen2.5-32B | $0.30 / $1.20 | ~$0.20 | ~2 hours/day |
| Qwen2.5-7B | $0.10 / $0.40 | ~$0.05 | ~1 hour/day |

The hardware math:

| Hardware | Upfront | Monthly electricity | Break-even vs API |
|---|---|---|---|
| RTX 3090 (used) | $600 | $15 | ~4 months |
| RTX 4090 | $1,500 | $20 | ~6 months |
| RTX 5080 | $1,000 | $18 | ~5 months |
| DGX Spark | $2,000 | $30 | ~8 months |

At moderate usage — an hour or more per day — local inference pays for itself. At high usage, the savings are dramatic. The catch is upfront capital. A RTX 5080 is $1,000. An API bill you can pause. Hardware you can’t.

## Fallback strategies

When your preferred model is too expensive or too slow, fall back to something cheaper. The key is knowing when quality is “good enough.”

### Strategy 1: Quality-Based Fallback

Quality-based fallback tries models until the output meets a threshold:

``` python
class QualityFallback:
    def __init__(self, quality_threshold: float = 0.8):
        self.threshold = quality_threshold
        self.models = [
            {"model": "claude-sonnet-4", "cost": 0.015},
            {"model": "qwen2.5-72b", "cost": 0.002},
            {"model": "qwen2.5-32b", "cost": 0.001},
            {"model": "qwen2.5-7b", "cost": 0.0004},
        ]

    def route(self, prompt: str) -> str:
        for model_config in self.models:
            result = self.call_model(model_config["model"], prompt)
            if self.evaluate_quality(result) >= self.threshold:
                return result
        return self.call_model(self.models[0]["model"], prompt)
```

The problem is evaluation itself. How do you measure quality without calling another model? Some systems use a small classifier. Others use heuristic checks — length, structure, keyword presence. None of these are perfect.

### Strategy 2: Latency-Based Fallback

Latency-based fallback is simpler. Route to the fastest model that meets your time budget:

``` python
class LatencyFallback:
    def __init__(self, max_latency: float = 5.0):
        self.max_latency = max_latency
        self.models = [
            {"model": "qwen2.5-1.5b", "latency": 0.5},
            {"model": "qwen2.5-7b", "latency": 2.0},
            {"model": "qwen2.5-32b", "latency": 10.0},
            {"model": "claude-sonnet-4", "latency": 5.0},
        ]

    def route(self, prompt: str) -> str:
        for model_config in sorted(self.models, key=lambda x: x["latency"]):
            if model_config["latency"] <= self.max_latency:
                return self.call_model(model_config["model"], prompt)
        return self.call_model(self.models[0]["model"], prompt)
```

## Caching

Caching is the most underrated cost optimization. Identical prompts happen more often than you think — classification requests, FAQ-style queries, repeated tool calls.

### Strategy 1: Prompt Caching

Exact prompt caching is simple:

``` python
import hashlib

class PromptCache:
    def __init__(self, max_size: int = 1000):
        self.cache = {}
        self.max_size = max_size

    def get(self, prompt: str) -> str | None:
        key = hashlib.sha256(prompt.encode()).hexdigest()
        return self.cache.get(key)

    def set(self, prompt: str, response: str):
        key = hashlib.sha256(prompt.encode()).hexdigest()
        if len(self.cache) >= self.max_size:
            self.cache.pop(next(iter(self.cache)))
        self.cache[key] = response
```

### Strategy 2: Semantic Caching

Semantic caching is more useful. It catches prompts that are different but mean the same thing:

``` python
from sentence_transformers import SentenceTransformer

class SemanticCache:
    def __init__(self, similarity_threshold: float = 0.95):
        self.model = SentenceTransformer('all-MiniLM-L6-v2')
        self.cache = {}
        self.threshold = similarity_threshold

    def get(self, prompt: str) -> str | None:
        prompt_embedding = self.model.encode([prompt])[0]
        for cached_prompt, cached_response in self.cache.items():
            cached_embedding = self.model.encode([cached_prompt])[0]
            similarity = self.cosine_similarity(
                prompt_embedding, cached_embedding
            )
            if similarity >= self.threshold:
                return cached_response
        return None

    def set(self, prompt: str, response: str):
        self.cache[prompt] = response
```

The threshold matters. 0.95 is aggressive — only very similar prompts match. 0.85 is more forgiving but risks returning wrong answers. Measure your miss rate and adjust.

Response caching for common queries is worth it too. If users ask “what’s the weather” or “what time is it” repeatedly, cache the pattern, not just the exact prompt:

``` python
class ResponseCache:
    def __init__(self):
        self.common_queries = {
            "what is the weather": "Check weather API",
            "what is the time": "Check system time",
            "who is the president": "Check current president",
        }

    def get(self, query: str) -> str | None:
        query_lower = query.lower()
        for common_query, response in self.common_queries.items():
            if common_query in query_lower:
                return response
        return None
```

This isn’t sophisticated, but it works. Common queries are common for a reason.

## When optimization helps

Optimization matters when you’re processing high volumes, running mixed workloads, or paying API costs that add up.

It doesn’t matter when you’re prototyping, using a single model, or processing low volumes. The complexity of budgeting, fallback, and caching isn’t worth it for a system that makes 100 requests a day.

Get the basic flow working first. Add optimization when the bill comes in.

## Tradeoffs

| Strategy | Cost | Quality | Complexity |
|---|---|---|---|
| No optimization | Highest | Consistent | Lowest |
| Token budgeting | Moderate | Variable | Medium |
| Fallback models | Low-Medium | Variable | Medium |
| Caching | Lowest | High (for cache hits) | Medium |
| Hybrid | Optimized | Optimized | Highest |

Production systems usually run hybrid. Budget per session, fall back on quality or latency, cache what you can. The complexity is real, but so are the savings.

## Related

[Model Routing Strategies](https://www.glukhov.org/llm-architecture/model-routing/model-routing-strategies/)— capability-based, cost-aware, latency-aware routing[LLM Guardrails in Practice](https://www.glukhov.org/llm-architecture/guardrails/llm-guardrails-in-practice/)— input validation, output filtering, safety[Multi-Model System Design](https://www.glukhov.org/llm-architecture/model-routing/multi-model-system-design/)— architecture for multiple models[LLM Architecture](https://www.glukhov.org/llm-architecture/)— system design pillar: routing, cost, guardrails, and orchestration