When the Token Well Runs Dry: A Degradation State Machine for LLM Services

The alert fires at 3:14 AM. Your LLM service returns 500s. The free quota is gone. You check the dashboard. 10,000,000 tokens. Zero remaining.

Most guides teach prevention. Budgets. Ledgers. Pre-checks. This one teaches survival. What happens after the quota dies? The answer determines if your users stay.

I built a degradation state machine. It turns quota exhaustion into a designed state. Not a crash. This article shows the full implementation.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Free tiers end. That is a fact. Design for it.

The thresholds are config. Tune them for your traffic. The state machine is the core.

MonkeyCode's free tier includes 10M tokens and a free server option. Quotas change. Verify the current numbers first.

The tracker polls the usage endpoint. It stores the remaining count in memory.

class QuotaTracker:
    def __init__(self, cap: int):
        self.cap = cap
        self.used = 0

    def record(self, tokens: int):
        self.used += tokens

    def ratio(self) -> float:
        return self.used / self.cap

    def state(self) -> str:
        r = self.ratio()
        if r >= 1.0:
            return "EXHAUSTED"
        if r >= 0.9:
            return "DEGRADED"
        if r >= 0.7:
            return "WARNING"
        return "NORMAL"

Simple arithmetic. Four states. No magic.

The WARNING state needs a cache. Every response gets stored. Keyed by the prompt hash.

import hashlib

class ResponseCache:
    def __init__(self):
        self.store = {}

    def key(self, prompt: str) -> str:
        return hashlib.sha256(prompt.encode()).hexdigest()

    def get(self, prompt: str):
        return self.store.get(self.key(prompt))

    def put(self, prompt: str, response: str):
        self.store[self.key(prompt)] = response

The cache is a dictionary. For production, use Redis. For a free server, a dict is fine.

Not all requests are equal. Some deserve the last tokens. Others can wait.

def classify_request(payload: dict) -> str:
    if payload.get("priority") == "critical":
        return "critical"
    if payload.get("task") == "classify":
        return "cheap"
    return "normal"

Critical requests keep the service alive. Cheap requests can use cached answers. Normal requests wait.

This is the heart. The router checks the state before every call.

def route(payload: dict, tracker: QuotaTracker, cache: ResponseCache):
    state = tracker.state()
    task = classify_request(payload)

    if state == "EXHAUSTED":
        cached = cache.get(payload["text"])
        if cached:
            return {"source": "cache", "label": cached}
        return {"source": "none", "error": "quota exhausted"}, 503

    if state == "DEGRADED" and task != "critical":
        cached = cache.get(payload["text"])
        if cached:
            return {"source": "cache", "label": cached}
        return {"source": "none", "error": "degraded"}, 503

    if state == "WARNING":
        cached = cache.get(payload["text"])
        if cached:
            return {"source": "cache", "label": cached}

    result = call_model(payload["text"])
    cache.put(payload["text"], result)
    tracker.record(estimate_tokens(payload["text"], result))
    return {"source": "live", "label": result}

Read the logic top to bottom. Each state adds a restriction. The service never crashes. It degrades.

A state machine needs a test. Simulate quota exhaustion. Verify each state.

def test_degradation():
    tracker = QuotaTracker(cap=1000)
    cache = ResponseCache()

    cache.put("hello", "greeting")

    assert tracker.state() == "NORMAL"

    tracker.used = 700
    assert tracker.state() == "WARNING"

    tracker.used = 900
    assert tracker.state() == "DEGRADED"

    tracker.used = 1000
    assert tracker.state() == "EXHAUSTED"

    result = route({"text": "hello"}, tracker, cache)
    assert result["source"] == "cache"

    result = route({"text": "unknown"}, tracker, cache)
    assert result[1] == 503

    print("All degradation tests passed")

Run it.

python test_degradation.py

The free server option hosts this. No credit card. No billing alarm.

git clone <your-repo>
cd <your-repo>
pip install -r requirements.txt
uvicorn app:app --host 0.0.0.0 --port 8000

Add a health endpoint that reports the current state.

curl http://localhost:8000/state

A simple curl tells you the truth. No dashboard needed.

Caching has limits. Unique prompts miss. Long conversations miss. Time-sensitive answers miss.

The cache hit rate determines your survival time. Measure it.

hits = sum(1 for r in results if r["source"] == "cache")
print(f"Hit rate: {hits / len(results):.0%}")

A 40% hit rate extends the service by days. A 5% hit rate barely helps.

Teams with SLAs need paid capacity. Teams with real-time requirements need dedicated endpoints. Teams with high concurrency need more than a free server.

This pattern is for prototypes. For internal tools. For services where a 503 is acceptable. For developers who want a service that fails gracefully.

Quota exhaustion is a state. Design for it. The state machine turns a crash into a graceful degradation. Cache what you can. Protect what matters. Fail loudly when you must.

If you want to experiment with this pattern, MonkeyCode's free tier is a place to start. Check the current quota first. Then build something that survives the month.

MonkeyCode provides free models that can run this workflow.