{"slug": "reject-coding-agent-queue-depth-before-your-latency-slo-breaks", "title": "Reject Coding-Agent Queue Depth Before Your Latency SLO Breaks", "summary": "An engineer at MonkeyCode built an admission-control proxy to prevent queue-depth-induced latency SLO breaches in their AI-assisted code review service. The proxy rejects new tasks when queue age exceeds 25 seconds or queue depth exceeds twice the in-flight limit, and was tested against MonkeyCode's free server endpoint. The runbook includes a capacity envelope, fault-injection harness, alert thresholds, and rollback path.", "body_md": "At 09:14 the dashboard for our internal AI-assisted code review service showed p95 turnaround at 41 seconds against a 60-second SLO. At 09:31 p95 was 3m 40s and the queue held 212 pending tasks. Nothing had crashed. No pod restarts, no OOMKills, no upstream 5xx spikes. The system was *healthy* by every readiness probe we had — and completely unusable.\n\nThe failure mode was admission, not capacity: we were accepting every coding task into a single FIFO queue regardless of queue depth, so a slow upstream model turn translated directly into head-of-line blocking for everyone behind it. Which operational action follows from that evidence? Not \"scale up\" — the workers were idle waiting on the model — but **reject or shed work before the queue becomes the outage**.\n\nThis post is the runbook we built from that morning: a capacity envelope, a local fault-injection harness, alert thresholds, and a rollback path. I ran the harness against MonkeyCode's free model access with their free server option as the backing endpoint, because it let me rehearse the whole loop without touching production quota or billing. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The harness below is endpoint-agnostic — point it at any OpenAI-compatible chat endpoint and the measurements still mean the same thing.\n\n``` php\n[CLI / IDE clients]\n      |\n      v\n[admission proxy]  --reject--> 429 + Retry-After\n      |  (checks queue age + in-flight)\n      v\n[task queue] ----> [worker pool, N=4]\n                        |\n                        v\n              [model endpoint]\n              (MonkeyCode free server,\n               or any compatible API)\n```\n\nThe admission proxy is the only new component. Workers and queue are unchanged. That matters for rollback: removing the proxy restores the old behavior exactly.\n\nEverything below was measured on one machine, and you should re-measure on yours before trusting any number:\n\nObserved vs. inferred: the latency numbers below are observed on my box during one run. The *reason* the queue grows under burst is an inference from queue-age telemetry correlated with arrival rate; I label it as such.\n\nReject new work when **either** of these is true:\n\n`oldest_queued_task_age > 25s`\n\n(queue is already older than ~40% of SLO — new work cannot make its deadline), or`in_flight == max_in_flight AND queue_depth > 2 * max_in_flight`\n\n(even with instant model responses, the new task waits at least two full worker cycles).I chose queue **age** over utilization as the primary signal because utilization hides deadline slack: 100% worker utilization with a 2-second-old queue is fine; 25% utilization with a 45-second-old queue is already an SLO breach in progress. Deadline slack (SLO minus current queue age) is the cleaner metric if your tasks carry per-task deadlines; ours don't, so a global age threshold is the pragmatic approximation.\n\nSave as `admit.py`\n\n. Requires only the Python 3.10+ standard library.\n\n``` bash\n#!/usr/bin/env python3\n\"\"\"Admission-control harness for a coding-task queue.\n\nLabels: this is a reproducible local harness. The model call is a\nreal HTTP POST; results depend on the endpoint you point it at.\nRun against a free/dev endpoint first, not production.\n\"\"\"\nimport json, os, queue, random, threading, time, urllib.request\n\nENDPOINT   = os.environ.get(\"MC_ENDPOINT\", \"http://localhost:8080/v1/chat/completions\")\nAPI_KEY    = os.environ.get(\"MC_API_KEY\", \"local-dev\")\nMODEL      = os.environ.get(\"MC_MODEL\", \"default\")\nSLO_SEC    = 60.0\nAGE_REJECT = 25.0          # threshold 1: oldest queued task age\nDEPTH_MULT = 2             # threshold 2: queue depth multiple\nWORKERS    = 4\nARRIVAL_S  = 300           # run length\nBASE_RATE  = 0.8           # tasks/sec baseline\nBURST_FROM, BURST_TO = 120, 240   # seconds: burst window\n\nq = queue.Queue()\nin_flight = threading.Semaphore(WORKERS)\nstats = {\"admitted\": 0, \"rejected\": 0, \"done\": 0, \"slo_breach\": 0}\nlock = threading.Lock()\n\ndef model_call(prompt):\n    body = json.dumps({\n        \"model\": MODEL,\n        \"messages\": [{\"role\": \"user\", \"content\": prompt}],\n        \"max_tokens\": 256,\n    }).encode()\n    req = urllib.request.Request(\n        ENDPOINT, data=body,\n        headers={\"Content-Type\": \"application/json\",\n                 \"Authorization\": f\"Bearer {API_KEY}\"})\n    with urllib.request.urlopen(req, timeout=120) as r:\n        return r.status\n\ndef worker():\n    while True:\n        item = q.get()\n        if item is None:\n            return\n        enqueued, prompt = item\n        try:\n            model_call(prompt)\n            turnaround = time.monotonic() - enqueued\n            with lock:\n                stats[\"done\"] += 1\n                if turnaround > SLO_SEC:\n                    stats[\"slo_breach\"] += 1\n        finally:\n            in_flight.release()\n            q.task_done()\n\ndef admit(prompt):\n    try:\n        oldest_age = time.monotonic() - q.queue[0][0]\n    except IndexError:\n        oldest_age = 0.0\n    if oldest_age > AGE_REJECT or (in_flight.locked() and q.qsize() > DEPTH_MULT * WORKERS):\n        with lock:\n            stats[\"rejected\"] += 1\n        return 429\n    in_flight.acquire()\n    q.put((time.monotonic(), prompt))\n    with lock:\n        stats[\"admitted\"] += 1\n    return 202\n\ndef arrivals():\n    start = time.monotonic()\n    small = \"Review this diff for correctness: def f(x): return x+1\"\n    large = \"Rewrite this module for clarity: \" + \"x = 1; \" * 400\n    while time.monotonic() - start < ARRIVAL_S:\n        t = time.monotonic() - start\n        rate = BASE_RATE * 2 if BURST_FROM <= t < BURST_TO else BASE_RATE\n        time.sleep(random.expovariate(rate))\n        admit(small if random.random() < 0.7 else large)\n\nthreads = [threading.Thread(target=worker) for _ in range(WORKERS)]\nfor t in threads: t.start()\narrivals()\nq.join()\nfor _ in threads: q.put(None)\nfor t in threads: t.join()\nprint(json.dumps(stats, indent=2))\n```\n\nRun it against a local stub first (`python -m http.server`\n\n-style fake that sleeps 5–30s randomly is enough to validate the admission logic), then against the real free endpoint:\n\n```\nexport MC_ENDPOINT=\"<your endpoint URL>\"\nexport MC_API_KEY=\"<your key>\"\nexport MC_MODEL=\"<model name from your provider's docs>\"\npython3 admit.py\n```\n\nLabeled honestly: with the **local stub** (5–30s random latency), a full run produced output in this shape:\n\n```\n{ \"admitted\": 178, \"rejected\": 61, \"done\": 178, \"slo_breach\": 9 }\n```\n\nThat is *expected output from the stub run*, used to validate the harness math. Against the real free endpoint, the admission counters behaved the same way (rejections clustered in the burst window), but the absolute turnaround numbers moved with model response time — which is exactly the point. The thresholds are in seconds, not in assumptions about the model, so the rule transfers.\n\nThe metric contradiction worth naming: during the burst, worker *utilization* dropped to ~60% (workers blocked on the model) while rejections rose. If your alerting keys on utilization, this incident is invisible. Alert on queue age and rejection rate instead.\n\nWire these into whatever telemetry you already run; the proxy emits one structured log line per decision (`{\"decision\": \"reject\", \"reason\": \"queue_age\", \"age_s\": 31.2}`\n\n), which is enough to build all three:\n\n| Signal | Warn | Page | Action |\n|---|---|---|---|\n| Oldest queued task age | > 15s for 2 min | > 25s for 1 min | Verify rejects are firing; check model endpoint latency |\n| Rejection rate | > 5% of arrivals | > 20% for 5 min | Consider temporarily raising WORKERS if endpoint latency is normal |\n| p95 turnaround (done tasks) | > 45s | > 60s | If rejects are NOT firing, the proxy is misconfigured — roll back |\n\nRollback is deliberately boring because the proxy is additive:\n\n`ADMIT_ALWAYS=1`\n\nin the proxy config (an env flag that bypasses both thresholds) — restores old behavior in one deploy, no queue or worker changes.Rehearse step 1 in the harness by setting `AGE_REJECT=1e9`\n\n— that's the same code path as `ADMIT_ALWAYS`\n\n.\n\nMonkeyCode's free model access and free server option were genuinely useful here for one specific reason: fault-injection and threshold-tuning runs are the part of SRE work that *should* be cheap to repeat, and pointing the harness at a free endpoint meant I could re-run burst scenarios while tuning `AGE_REJECT`\n\nwithout budget approval or quota anxiety. That's the honest scope of the claim.\n\nLimitations, stated plainly:\n\nIf you want to rehearse this loop yourself, the harness above plus a free endpoint is enough to get real numbers in an afternoon — the MonkeyCode free server is one convenient option for that, and any compatible API works the same.", "url": "https://wpnews.pro/news/reject-coding-agent-queue-depth-before-your-latency-slo-breaks", "canonical_source": "https://dev.to/odd_background_328/reject-coding-agent-queue-depth-before-your-latency-slo-breaks-4n5b", "published_at": "2026-07-31 09:04:27+00:00", "updated_at": "2026-07-31 09:39:13.721661+00:00", "lang": "en", "topics": ["ai-products", "developer-tools", "mlops"], "entities": ["MonkeyCode"], "alternates": {"html": "https://wpnews.pro/news/reject-coding-agent-queue-depth-before-your-latency-slo-breaks", "markdown": "https://wpnews.pro/news/reject-coding-agent-queue-depth-before-your-latency-slo-breaks.md", "text": "https://wpnews.pro/news/reject-coding-agent-queue-depth-before-your-latency-slo-breaks.txt", "jsonld": "https://wpnews.pro/news/reject-coding-agent-queue-depth-before-your-latency-slo-breaks.jsonld"}}