# When the Token Well Runs Dry: A Degradation State Machine for LLM Services

> Source: <https://dev.to/apprs_6334/when-the-token-well-runs-dry-a-degradation-state-machine-for-llm-services-1fhm>
> Published: 2026-09-04 11:06:12+00:00

The alert fires at 3:14 AM. Your LLM service returns 500s. The free quota is gone. You check the dashboard. 10,000,000 tokens. Zero remaining.

Most guides teach prevention. Budgets. Ledgers. Pre-checks. This one teaches survival. What happens after the quota dies? The answer determines if your users stay.

I built a degradation state machine. It turns quota exhaustion into a designed state. Not a crash. This article shows the full implementation.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Free tiers end. That is a fact. Design for it.

| State | Trigger | Behavior |
|---|---|---|
| NORMAL | Usage < 70% | Full service |
| WARNING | Usage >= 70% | Cache responses |
| DEGRADED | Usage >= 90% | Critical requests only |
| EXHAUSTED | Usage >= 100% | Cache or 503 |

The thresholds are config. Tune them for your traffic. The state machine is the core.

MonkeyCode's free tier includes 10M tokens and a free server option. Quotas change. Verify the current numbers first.

The tracker polls the usage endpoint. It stores the remaining count in memory.

``` python
class QuotaTracker:
    def __init__(self, cap: int):
        self.cap = cap
        self.used = 0

    def record(self, tokens: int):
        self.used += tokens

    def ratio(self) -> float:
        return self.used / self.cap

    def state(self) -> str:
        r = self.ratio()
        if r >= 1.0:
            return "EXHAUSTED"
        if r >= 0.9:
            return "DEGRADED"
        if r >= 0.7:
            return "WARNING"
        return "NORMAL"
```

Simple arithmetic. Four states. No magic.

The WARNING state needs a cache. Every response gets stored. Keyed by the prompt hash.

``` python
import hashlib

class ResponseCache:
    def __init__(self):
        self.store = {}

    def key(self, prompt: str) -> str:
        return hashlib.sha256(prompt.encode()).hexdigest()

    def get(self, prompt: str):
        return self.store.get(self.key(prompt))

    def put(self, prompt: str, response: str):
        self.store[self.key(prompt)] = response
```

The cache is a dictionary. For production, use Redis. For a free server, a dict is fine.

Not all requests are equal. Some deserve the last tokens. Others can wait.

``` php
def classify_request(payload: dict) -> str:
    if payload.get("priority") == "critical":
        return "critical"
    if payload.get("task") == "classify":
        return "cheap"
    return "normal"
```

Critical requests keep the service alive. Cheap requests can use cached answers. Normal requests wait.

This is the heart. The router checks the state before every call.

``` python
def route(payload: dict, tracker: QuotaTracker, cache: ResponseCache):
    state = tracker.state()
    task = classify_request(payload)

    if state == "EXHAUSTED":
        cached = cache.get(payload["text"])
        if cached:
            return {"source": "cache", "label": cached}
        return {"source": "none", "error": "quota exhausted"}, 503

    if state == "DEGRADED" and task != "critical":
        cached = cache.get(payload["text"])
        if cached:
            return {"source": "cache", "label": cached}
        return {"source": "none", "error": "degraded"}, 503

    if state == "WARNING":
        cached = cache.get(payload["text"])
        if cached:
            return {"source": "cache", "label": cached}

    # live call to the model
    result = call_model(payload["text"])
    cache.put(payload["text"], result)
    tracker.record(estimate_tokens(payload["text"], result))
    return {"source": "live", "label": result}
```

Read the logic top to bottom. Each state adds a restriction. The service never crashes. It degrades.

A state machine needs a test. Simulate quota exhaustion. Verify each state.

``` python
def test_degradation():
    tracker = QuotaTracker(cap=1000)
    cache = ResponseCache()

    # fill the cache
    cache.put("hello", "greeting")

    # NORMAL
    assert tracker.state() == "NORMAL"

    # WARNING
    tracker.used = 700
    assert tracker.state() == "WARNING"

    # DEGRADED
    tracker.used = 900
    assert tracker.state() == "DEGRADED"

    # EXHAUSTED
    tracker.used = 1000
    assert tracker.state() == "EXHAUSTED"

    # EXHAUSTED serves from cache
    result = route({"text": "hello"}, tracker, cache)
    assert result["source"] == "cache"

    # EXHAUSTED rejects uncached
    result = route({"text": "unknown"}, tracker, cache)
    assert result[1] == 503

    print("All degradation tests passed")
```

Run it.

```
python test_degradation.py
```

The free server option hosts this. No credit card. No billing alarm.

```
git clone <your-repo>
cd <your-repo>
pip install -r requirements.txt
uvicorn app:app --host 0.0.0.0 --port 8000
```

Add a health endpoint that reports the current state.

```
curl http://localhost:8000/state
# {"state": "NORMAL", "ratio": 0.42}
```

A simple curl tells you the truth. No dashboard needed.

Caching has limits. Unique prompts miss. Long conversations miss. Time-sensitive answers miss.

The cache hit rate determines your survival time. Measure it.

```
hits = sum(1 for r in results if r["source"] == "cache")
print(f"Hit rate: {hits / len(results):.0%}")
```

A 40% hit rate extends the service by days. A 5% hit rate barely helps.

Teams with SLAs need paid capacity. Teams with real-time requirements need dedicated endpoints. Teams with high concurrency need more than a free server.

This pattern is for prototypes. For internal tools. For services where a 503 is acceptable. For developers who want a service that fails gracefully.

Quota exhaustion is a state. Design for it. The state machine turns a crash into a graceful degradation. Cache what you can. Protect what matters. Fail loudly when you must.

If you want to experiment with this pattern, MonkeyCode's free tier is a place to start. Check the current quota first. Then build something that survives the month.

MonkeyCode provides free models that can run this workflow.