Reject Coding-Agent Queue Depth Before Your Latency SLO Breaks An engineer at MonkeyCode built an admission-control proxy to prevent queue-depth-induced latency SLO breaches in their AI-assisted code review service. The proxy rejects new tasks when queue age exceeds 25 seconds or queue depth exceeds twice the in-flight limit, and was tested against MonkeyCode's free server endpoint. The runbook includes a capacity envelope, fault-injection harness, alert thresholds, and rollback path. At 09:14 the dashboard for our internal AI-assisted code review service showed p95 turnaround at 41 seconds against a 60-second SLO. At 09:31 p95 was 3m 40s and the queue held 212 pending tasks. Nothing had crashed. No pod restarts, no OOMKills, no upstream 5xx spikes. The system was healthy by every readiness probe we had — and completely unusable. The failure mode was admission, not capacity: we were accepting every coding task into a single FIFO queue regardless of queue depth, so a slow upstream model turn translated directly into head-of-line blocking for everyone behind it. Which operational action follows from that evidence? Not "scale up" — the workers were idle waiting on the model — but reject or shed work before the queue becomes the outage . This post is the runbook we built from that morning: a capacity envelope, a local fault-injection harness, alert thresholds, and a rollback path. I ran the harness against MonkeyCode's free model access with their free server option as the backing endpoint, because it let me rehearse the whole loop without touching production quota or billing. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The harness below is endpoint-agnostic — point it at any OpenAI-compatible chat endpoint and the measurements still mean the same thing. php CLI / IDE clients | v admission proxy --reject-- 429 + Retry-After | checks queue age + in-flight v task queue ---- worker pool, N=4 | v model endpoint MonkeyCode free server, or any compatible API The admission proxy is the only new component. Workers and queue are unchanged. That matters for rollback: removing the proxy restores the old behavior exactly. Everything below was measured on one machine, and you should re-measure on yours before trusting any number: Observed vs. inferred: the latency numbers below are observed on my box during one run. The reason the queue grows under burst is an inference from queue-age telemetry correlated with arrival rate; I label it as such. Reject new work when either of these is true: oldest queued task age 25s queue is already older than ~40% of SLO — new work cannot make its deadline , or in flight == max in flight AND queue depth 2 max in flight even with instant model responses, the new task waits at least two full worker cycles .I chose queue age over utilization as the primary signal because utilization hides deadline slack: 100% worker utilization with a 2-second-old queue is fine; 25% utilization with a 45-second-old queue is already an SLO breach in progress. Deadline slack SLO minus current queue age is the cleaner metric if your tasks carry per-task deadlines; ours don't, so a global age threshold is the pragmatic approximation. Save as admit.py . Requires only the Python 3.10+ standard library. bash /usr/bin/env python3 """Admission-control harness for a coding-task queue. Labels: this is a reproducible local harness. The model call is a real HTTP POST; results depend on the endpoint you point it at. Run against a free/dev endpoint first, not production. """ import json, os, queue, random, threading, time, urllib.request ENDPOINT = os.environ.get "MC ENDPOINT", "http://localhost:8080/v1/chat/completions" API KEY = os.environ.get "MC API KEY", "local-dev" MODEL = os.environ.get "MC MODEL", "default" SLO SEC = 60.0 AGE REJECT = 25.0 threshold 1: oldest queued task age DEPTH MULT = 2 threshold 2: queue depth multiple WORKERS = 4 ARRIVAL S = 300 run length BASE RATE = 0.8 tasks/sec baseline BURST FROM, BURST TO = 120, 240 seconds: burst window q = queue.Queue in flight = threading.Semaphore WORKERS stats = {"admitted": 0, "rejected": 0, "done": 0, "slo breach": 0} lock = threading.Lock def model call prompt : body = json.dumps { "model": MODEL, "messages": {"role": "user", "content": prompt} , "max tokens": 256, } .encode req = urllib.request.Request ENDPOINT, data=body, headers={"Content-Type": "application/json", "Authorization": f"Bearer {API KEY}"} with urllib.request.urlopen req, timeout=120 as r: return r.status def worker : while True: item = q.get if item is None: return enqueued, prompt = item try: model call prompt turnaround = time.monotonic - enqueued with lock: stats "done" += 1 if turnaround SLO SEC: stats "slo breach" += 1 finally: in flight.release q.task done def admit prompt : try: oldest age = time.monotonic - q.queue 0 0 except IndexError: oldest age = 0.0 if oldest age AGE REJECT or in flight.locked and q.qsize DEPTH MULT WORKERS : with lock: stats "rejected" += 1 return 429 in flight.acquire q.put time.monotonic , prompt with lock: stats "admitted" += 1 return 202 def arrivals : start = time.monotonic small = "Review this diff for correctness: def f x : return x+1" large = "Rewrite this module for clarity: " + "x = 1; " 400 while time.monotonic - start < ARRIVAL S: t = time.monotonic - start rate = BASE RATE 2 if BURST FROM <= t < BURST TO else BASE RATE time.sleep random.expovariate rate admit small if random.random < 0.7 else large threads = threading.Thread target=worker for in range WORKERS for t in threads: t.start arrivals q.join for in threads: q.put None for t in threads: t.join print json.dumps stats, indent=2 Run it against a local stub first python -m http.server -style fake that sleeps 5–30s randomly is enough to validate the admission logic , then against the real free endpoint: export MC ENDPOINT="