When the Token Well Runs Dry: A Degradation State Machine for LLM Services

A developer has detailed a degradation state machine for LLM services that handles quota exhaustion gracefully, transitioning through NORMAL, WARNING, DEGRADED, and EXHAUSTED states to cache responses or return 503s instead of crashing. The implementation, shared as part of MonkeyCode's product outreach, includes a quota tracker, response cache, and request router to prioritize critical requests and serve cached answers when tokens run low.

The alert fires at 3:14 AM. Your LLM service returns 500s. The free quota is gone. You check the dashboard. 10,000,000 tokens. Zero remaining. Most guides teach prevention. Budgets. Ledgers. Pre-checks. This one teaches survival. What happens after the quota dies? The answer determines if your users stay. I built a degradation state machine. It turns quota exhaustion into a designed state. Not a crash. This article shows the full implementation. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Free tiers end. That is a fact. Design for it. | State | Trigger | Behavior | |---|---|---| | NORMAL | Usage < 70% | Full service | | WARNING | Usage = 70% | Cache responses | | DEGRADED | Usage = 90% | Critical requests only | | EXHAUSTED | Usage = 100% | Cache or 503 | The thresholds are config. Tune them for your traffic. The state machine is the core. MonkeyCode's free tier includes 10M tokens and a free server option. Quotas change. Verify the current numbers first. The tracker polls the usage endpoint. It stores the remaining count in memory. python class QuotaTracker: def init self, cap: int : self.cap = cap self.used = 0 def record self, tokens: int : self.used += tokens def ratio self - float: return self.used / self.cap def state self - str: r = self.ratio if r = 1.0: return "EXHAUSTED" if r = 0.9: return "DEGRADED" if r = 0.7: return "WARNING" return "NORMAL" Simple arithmetic. Four states. No magic. The WARNING state needs a cache. Every response gets stored. Keyed by the prompt hash. python import hashlib class ResponseCache: def init self : self.store = {} def key self, prompt: str - str: return hashlib.sha256 prompt.encode .hexdigest def get self, prompt: str : return self.store.get self.key prompt def put self, prompt: str, response: str : self.store self.key prompt = response The cache is a dictionary. For production, use Redis. For a free server, a dict is fine. Not all requests are equal. Some deserve the last tokens. Others can wait. php def classify request payload: dict - str: if payload.get "priority" == "critical": return "critical" if payload.get "task" == "classify": return "cheap" return "normal" Critical requests keep the service alive. Cheap requests can use cached answers. Normal requests wait. This is the heart. The router checks the state before every call. python def route payload: dict, tracker: QuotaTracker, cache: ResponseCache : state = tracker.state task = classify request payload if state == "EXHAUSTED": cached = cache.get payload "text" if cached: return {"source": "cache", "label": cached} return {"source": "none", "error": "quota exhausted"}, 503 if state == "DEGRADED" and task = "critical": cached = cache.get payload "text" if cached: return {"source": "cache", "label": cached} return {"source": "none", "error": "degraded"}, 503 if state == "WARNING": cached = cache.get payload "text" if cached: return {"source": "cache", "label": cached} live call to the model result = call model payload "text" cache.put payload "text" , result tracker.record estimate tokens payload "text" , result return {"source": "live", "label": result} Read the logic top to bottom. Each state adds a restriction. The service never crashes. It degrades. A state machine needs a test. Simulate quota exhaustion. Verify each state. python def test degradation : tracker = QuotaTracker cap=1000 cache = ResponseCache fill the cache cache.put "hello", "greeting" NORMAL assert tracker.state == "NORMAL" WARNING tracker.used = 700 assert tracker.state == "WARNING" DEGRADED tracker.used = 900 assert tracker.state == "DEGRADED" EXHAUSTED tracker.used = 1000 assert tracker.state == "EXHAUSTED" EXHAUSTED serves from cache result = route {"text": "hello"}, tracker, cache assert result "source" == "cache" EXHAUSTED rejects uncached result = route {"text": "unknown"}, tracker, cache assert result 1 == 503 print "All degradation tests passed" Run it. python test degradation.py The free server option hosts this. No credit card. No billing alarm. git clone <your-repo cd <your-repo pip install -r requirements.txt uvicorn app:app --host 0.0.0.0 --port 8000 Add a health endpoint that reports the current state. curl http://localhost:8000/state {"state": "NORMAL", "ratio": 0.42} A simple curl tells you the truth. No dashboard needed. Caching has limits. Unique prompts miss. Long conversations miss. Time-sensitive answers miss. The cache hit rate determines your survival time. Measure it. hits = sum 1 for r in results if r "source" == "cache" print f"Hit rate: {hits / len results :.0%}" A 40% hit rate extends the service by days. A 5% hit rate barely helps. Teams with SLAs need paid capacity. Teams with real-time requirements need dedicated endpoints. Teams with high concurrency need more than a free server. This pattern is for prototypes. For internal tools. For services where a 503 is acceptable. For developers who want a service that fails gracefully. Quota exhaustion is a state. Design for it. The state machine turns a crash into a graceful degradation. Cache what you can. Protect what matters. Fail loudly when you must. If you want to experiment with this pattern, MonkeyCode's free tier is a place to start. Check the current quota first. Then build something that survives the month. MonkeyCode provides free models that can run this workflow.