{"slug": "when-the-token-well-runs-dry-a-degradation-state-machine-for-llm-services", "title": "When the Token Well Runs Dry: A Degradation State Machine for LLM Services", "summary": "A developer has detailed a degradation state machine for LLM services that handles quota exhaustion gracefully, transitioning through NORMAL, WARNING, DEGRADED, and EXHAUSTED states to cache responses or return 503s instead of crashing. The implementation, shared as part of MonkeyCode's product outreach, includes a quota tracker, response cache, and request router to prioritize critical requests and serve cached answers when tokens run low.", "body_md": "The alert fires at 3:14 AM. Your LLM service returns 500s. The free quota is gone. You check the dashboard. 10,000,000 tokens. Zero remaining.\n\nMost guides teach prevention. Budgets. Ledgers. Pre-checks. This one teaches survival. What happens after the quota dies? The answer determines if your users stay.\n\nI built a degradation state machine. It turns quota exhaustion into a designed state. Not a crash. This article shows the full implementation.\n\nDisclosure: This article was prepared as part of MonkeyCode's product outreach.\n\nFree tiers end. That is a fact. Design for it.\n\n| State | Trigger | Behavior |\n|---|---|---|\n| NORMAL | Usage < 70% | Full service |\n| WARNING | Usage >= 70% | Cache responses |\n| DEGRADED | Usage >= 90% | Critical requests only |\n| EXHAUSTED | Usage >= 100% | Cache or 503 |\n\nThe thresholds are config. Tune them for your traffic. The state machine is the core.\n\nMonkeyCode's free tier includes 10M tokens and a free server option. Quotas change. Verify the current numbers first.\n\nThe tracker polls the usage endpoint. It stores the remaining count in memory.\n\n``` python\nclass QuotaTracker:\n    def __init__(self, cap: int):\n        self.cap = cap\n        self.used = 0\n\n    def record(self, tokens: int):\n        self.used += tokens\n\n    def ratio(self) -> float:\n        return self.used / self.cap\n\n    def state(self) -> str:\n        r = self.ratio()\n        if r >= 1.0:\n            return \"EXHAUSTED\"\n        if r >= 0.9:\n            return \"DEGRADED\"\n        if r >= 0.7:\n            return \"WARNING\"\n        return \"NORMAL\"\n```\n\nSimple arithmetic. Four states. No magic.\n\nThe WARNING state needs a cache. Every response gets stored. Keyed by the prompt hash.\n\n``` python\nimport hashlib\n\nclass ResponseCache:\n    def __init__(self):\n        self.store = {}\n\n    def key(self, prompt: str) -> str:\n        return hashlib.sha256(prompt.encode()).hexdigest()\n\n    def get(self, prompt: str):\n        return self.store.get(self.key(prompt))\n\n    def put(self, prompt: str, response: str):\n        self.store[self.key(prompt)] = response\n```\n\nThe cache is a dictionary. For production, use Redis. For a free server, a dict is fine.\n\nNot all requests are equal. Some deserve the last tokens. Others can wait.\n\n``` php\ndef classify_request(payload: dict) -> str:\n    if payload.get(\"priority\") == \"critical\":\n        return \"critical\"\n    if payload.get(\"task\") == \"classify\":\n        return \"cheap\"\n    return \"normal\"\n```\n\nCritical requests keep the service alive. Cheap requests can use cached answers. Normal requests wait.\n\nThis is the heart. The router checks the state before every call.\n\n``` python\ndef route(payload: dict, tracker: QuotaTracker, cache: ResponseCache):\n    state = tracker.state()\n    task = classify_request(payload)\n\n    if state == \"EXHAUSTED\":\n        cached = cache.get(payload[\"text\"])\n        if cached:\n            return {\"source\": \"cache\", \"label\": cached}\n        return {\"source\": \"none\", \"error\": \"quota exhausted\"}, 503\n\n    if state == \"DEGRADED\" and task != \"critical\":\n        cached = cache.get(payload[\"text\"])\n        if cached:\n            return {\"source\": \"cache\", \"label\": cached}\n        return {\"source\": \"none\", \"error\": \"degraded\"}, 503\n\n    if state == \"WARNING\":\n        cached = cache.get(payload[\"text\"])\n        if cached:\n            return {\"source\": \"cache\", \"label\": cached}\n\n    # live call to the model\n    result = call_model(payload[\"text\"])\n    cache.put(payload[\"text\"], result)\n    tracker.record(estimate_tokens(payload[\"text\"], result))\n    return {\"source\": \"live\", \"label\": result}\n```\n\nRead the logic top to bottom. Each state adds a restriction. The service never crashes. It degrades.\n\nA state machine needs a test. Simulate quota exhaustion. Verify each state.\n\n``` python\ndef test_degradation():\n    tracker = QuotaTracker(cap=1000)\n    cache = ResponseCache()\n\n    # fill the cache\n    cache.put(\"hello\", \"greeting\")\n\n    # NORMAL\n    assert tracker.state() == \"NORMAL\"\n\n    # WARNING\n    tracker.used = 700\n    assert tracker.state() == \"WARNING\"\n\n    # DEGRADED\n    tracker.used = 900\n    assert tracker.state() == \"DEGRADED\"\n\n    # EXHAUSTED\n    tracker.used = 1000\n    assert tracker.state() == \"EXHAUSTED\"\n\n    # EXHAUSTED serves from cache\n    result = route({\"text\": \"hello\"}, tracker, cache)\n    assert result[\"source\"] == \"cache\"\n\n    # EXHAUSTED rejects uncached\n    result = route({\"text\": \"unknown\"}, tracker, cache)\n    assert result[1] == 503\n\n    print(\"All degradation tests passed\")\n```\n\nRun it.\n\n```\npython test_degradation.py\n```\n\nThe free server option hosts this. No credit card. No billing alarm.\n\n```\ngit clone <your-repo>\ncd <your-repo>\npip install -r requirements.txt\nuvicorn app:app --host 0.0.0.0 --port 8000\n```\n\nAdd a health endpoint that reports the current state.\n\n```\ncurl http://localhost:8000/state\n# {\"state\": \"NORMAL\", \"ratio\": 0.42}\n```\n\nA simple curl tells you the truth. No dashboard needed.\n\nCaching has limits. Unique prompts miss. Long conversations miss. Time-sensitive answers miss.\n\nThe cache hit rate determines your survival time. Measure it.\n\n```\nhits = sum(1 for r in results if r[\"source\"] == \"cache\")\nprint(f\"Hit rate: {hits / len(results):.0%}\")\n```\n\nA 40% hit rate extends the service by days. A 5% hit rate barely helps.\n\nTeams with SLAs need paid capacity. Teams with real-time requirements need dedicated endpoints. Teams with high concurrency need more than a free server.\n\nThis pattern is for prototypes. For internal tools. For services where a 503 is acceptable. For developers who want a service that fails gracefully.\n\nQuota exhaustion is a state. Design for it. The state machine turns a crash into a graceful degradation. Cache what you can. Protect what matters. Fail loudly when you must.\n\nIf you want to experiment with this pattern, MonkeyCode's free tier is a place to start. Check the current quota first. Then build something that survives the month.\n\nMonkeyCode provides free models that can run this workflow.", "url": "https://wpnews.pro/news/when-the-token-well-runs-dry-a-degradation-state-machine-for-llm-services", "canonical_source": "https://dev.to/apprs_6334/when-the-token-well-runs-dry-a-degradation-state-machine-for-llm-services-1fhm", "published_at": "2026-09-04 11:06:12+00:00", "updated_at": "2026-09-04 11:25:16.584056+00:00", "lang": "en", "topics": ["large-language-models", "developer-tools", "ai-infrastructure"], "entities": ["MonkeyCode"], "alternates": {"html": "https://wpnews.pro/news/when-the-token-well-runs-dry-a-degradation-state-machine-for-llm-services", "markdown": "https://wpnews.pro/news/when-the-token-well-runs-dry-a-degradation-state-machine-for-llm-services.md", "text": "https://wpnews.pro/news/when-the-token-well-runs-dry-a-degradation-state-machine-for-llm-services.txt", "jsonld": "https://wpnews.pro/news/when-the-token-well-runs-dry-a-degradation-state-machine-for-llm-services.jsonld"}}