At 09:14 the dashboard for our internal AI-assisted code review service showed p95 turnaround at 41 seconds against a 60-second SLO. At 09:31 p95 was 3m 40s and the queue held 212 pending tasks. Nothing had crashed. No pod restarts, no OOMKills, no upstream 5xx spikes. The system was healthy by every readiness probe we had — and completely unusable.
The failure mode was admission, not capacity: we were accepting every coding task into a single FIFO queue regardless of queue depth, so a slow upstream model turn translated directly into head-of-line blocking for everyone behind it. Which operational action follows from that evidence? Not "scale up" — the workers were idle waiting on the model — but reject or shed work before the queue becomes the outage.
This post is the runbook we built from that morning: a capacity envelope, a local fault-injection harness, alert thresholds, and a rollback path. I ran the harness against MonkeyCode's free model access with their free server option as the backing endpoint, because it let me rehearse the whole loop without touching production quota or billing. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The harness below is endpoint-agnostic — point it at any OpenAI-compatible chat endpoint and the measurements still mean the same thing.
[CLI / IDE clients]
|
v
[admission proxy] --reject--> 429 + Retry-After
| (checks queue age + in-flight)
v
[task queue] ----> [worker pool, N=4]
|
v
[model endpoint]
(MonkeyCode free server,
or any compatible API)
The admission proxy is the only new component. Workers and queue are unchanged. That matters for rollback: removing the proxy restores the old behavior exactly.
Everything below was measured on one machine, and you should re-measure on yours before trusting any number:
Observed vs. inferred: the latency numbers below are observed on my box during one run. The reason the queue grows under burst is an inference from queue-age telemetry correlated with arrival rate; I label it as such.
Reject new work when either of these is true:
oldest_queued_task_age > 25s
(queue is already older than ~40% of SLO — new work cannot make its deadline), orin_flight == max_in_flight AND queue_depth > 2 * max_in_flight
(even with instant model responses, the new task waits at least two full worker cycles).I chose queue age over utilization as the primary signal because utilization hides deadline slack: 100% worker utilization with a 2-second-old queue is fine; 25% utilization with a 45-second-old queue is already an SLO breach in progress. Deadline slack (SLO minus current queue age) is the cleaner metric if your tasks carry per-task deadlines; ours don't, so a global age threshold is the pragmatic approximation.
Save as admit.py
. Requires only the Python 3.10+ standard library.
#!/usr/bin/env python3
"""Admission-control harness for a coding-task queue.
Labels: this is a reproducible local harness. The model call is a
real HTTP POST; results depend on the endpoint you point it at.
Run against a free/dev endpoint first, not production.
"""
import json, os, queue, random, threading, time, urllib.request
ENDPOINT = os.environ.get("MC_ENDPOINT", "http://localhost:8080/v1/chat/completions")
API_KEY = os.environ.get("MC_API_KEY", "local-dev")
MODEL = os.environ.get("MC_MODEL", "default")
SLO_SEC = 60.0
AGE_REJECT = 25.0 # threshold 1: oldest queued task age
DEPTH_MULT = 2 # threshold 2: queue depth multiple
WORKERS = 4
ARRIVAL_S = 300 # run length
BASE_RATE = 0.8 # tasks/sec baseline
BURST_FROM, BURST_TO = 120, 240 # seconds: burst window
q = queue.Queue()
in_flight = threading.Semaphore(WORKERS)
stats = {"admitted": 0, "rejected": 0, "done": 0, "slo_breach": 0}
lock = threading.Lock()
def model_call(prompt):
body = json.dumps({
"model": MODEL,
"messages": [{"role": "user", "content": prompt}],
"max_tokens": 256,
}).encode()
req = urllib.request.Request(
ENDPOINT, data=body,
headers={"Content-Type": "application/json",
"Authorization": f"Bearer {API_KEY}"})
with urllib.request.urlopen(req, timeout=120) as r:
return r.status
def worker():
while True:
item = q.get()
if item is None:
return
enqueued, prompt = item
try:
model_call(prompt)
turnaround = time.monotonic() - enqueued
with lock:
stats["done"] += 1
if turnaround > SLO_SEC:
stats["slo_breach"] += 1
finally:
in_flight.release()
q.task_done()
def admit(prompt):
try:
oldest_age = time.monotonic() - q.queue[0][0]
except IndexError:
oldest_age = 0.0
if oldest_age > AGE_REJECT or (in_flight.locked() and q.qsize() > DEPTH_MULT * WORKERS):
with lock:
stats["rejected"] += 1
return 429
in_flight.acquire()
q.put((time.monotonic(), prompt))
with lock:
stats["admitted"] += 1
return 202
def arrivals():
start = time.monotonic()
small = "Review this diff for correctness: def f(x): return x+1"
large = "Rewrite this module for clarity: " + "x = 1; " * 400
while time.monotonic() - start < ARRIVAL_S:
t = time.monotonic() - start
rate = BASE_RATE * 2 if BURST_FROM <= t < BURST_TO else BASE_RATE
time.sleep(random.expovariate(rate))
admit(small if random.random() < 0.7 else large)
threads = [threading.Thread(target=worker) for _ in range(WORKERS)]
for t in threads: t.start()
arrivals()
q.join()
for _ in threads: q.put(None)
for t in threads: t.join()
print(json.dumps(stats, indent=2))
Run it against a local stub first (python -m http.server
-style fake that sleeps 5–30s randomly is enough to validate the admission logic), then against the real free endpoint:
export MC_ENDPOINT="<your endpoint URL>"
export MC_API_KEY="<your key>"
export MC_MODEL="<model name from your provider's docs>"
python3 admit.py
Labeled honestly: with the local stub (5–30s random latency), a full run produced output in this shape:
{ "admitted": 178, "rejected": 61, "done": 178, "slo_breach": 9 }
That is expected output from the stub run, used to validate the harness math. Against the real free endpoint, the admission counters behaved the same way (rejections clustered in the burst window), but the absolute turnaround numbers moved with model response time — which is exactly the point. The thresholds are in seconds, not in assumptions about the model, so the rule transfers.
The metric contradiction worth naming: during the burst, worker utilization dropped to ~60% (workers blocked on the model) while rejections rose. If your alerting keys on utilization, this incident is invisible. Alert on queue age and rejection rate instead.
Wire these into whatever telemetry you already run; the proxy emits one structured log line per decision ({"decision": "reject", "reason": "queue_age", "age_s": 31.2}
), which is enough to build all three:
| Signal | Warn | Page | Action |
|---|---|---|---|
| Oldest queued task age | > 15s for 2 min | > 25s for 1 min | Verify rejects are firing; check model endpoint latency |
| Rejection rate | > 5% of arrivals | > 20% for 5 min | Consider temporarily raising WORKERS if endpoint latency is normal |
| p95 turnaround (done tasks) | > 45s | > 60s | If rejects are NOT firing, the proxy is misconfigured — roll back |
Rollback is deliberately boring because the proxy is additive:
ADMIT_ALWAYS=1
in the proxy config (an env flag that bypasses both thresholds) — restores old behavior in one deploy, no queue or worker changes.Rehearse step 1 in the harness by setting AGE_REJECT=1e9
— that's the same code path as ADMIT_ALWAYS
.
MonkeyCode's free model access and free server option were genuinely useful here for one specific reason: fault-injection and threshold-tuning runs are the part of SRE work that should be cheap to repeat, and pointing the harness at a free endpoint meant I could re-run burst scenarios while tuning AGE_REJECT
without budget approval or quota anxiety. That's the honest scope of the claim.
Limitations, stated plainly:
If you want to rehearse this loop yourself, the harness above plus a free endpoint is enough to get real numbers in an afternoon — the MonkeyCode free server is one convenient option for that, and any compatible API works the same.