Your Free AI Lane Is Slow in the Queue, Not in the Model

A developer published a two-clock instrumentation approach for diagnosing why batches run on free AI capacity miss their deadlines, arguing that queue wait time, not model speed, is usually the culprit. The method records five timestamps per job (enqueue, start, accept, deadline, attempt) to derive wait, generation time and wait share, then runs a small Python script that prints a hold-or-spill verdict based on a p95 wait-share threshold of 0.60 and a 15% deadline safety margin. The author recommends deliberately congesting a lane to tune thresholds rather than calibrating on quiet traffic.

You moved a batch to free capacity. The token bill dropped to zero. The deadline still slipped. Nothing in the model got slower. The jobs sat in a queue you never charted. Most AI cost dashboards start counting at the first token. That is the wrong start line for a free lane. On free capacity, the wall clock usually belongs to scheduling, not to generation. This post is a drill, not a vendor pitch. You record two clocks per job. Then you run one script and get a verdict: hold the batch, or spill it to paid capacity before the cutoff. Your job record needs five timestamps. Anything less and you cannot separate waiting from working. enqueue ts — the job was accepted by your scheduler. start ts — the first request hit the serving path. accept ts — the artifact passed your acceptance check. deadline ts — the wall-clock deadline for that batch. attempt — the retry index, so rework stays visible. From those you get three derived numbers: wait = start ts - enqueue ts gen = accept ts - start ts wait share = wait / accept ts - enqueue ts Gate on wait share . A lane with p95 wait share of 0.7 is a scheduling problem wearing a model costume. Buying faster tokens there changes almost nothing. The script below reads JSON Lines and prints a verdict. It is deliberately dumb. No dashboards, no agent, no database. bash /usr/bin/env python3 two clock.py — lane verdict from job events JSON Lines import json, sys def load path : with open path as fh: for line in fh: line = line.strip if line: yield json.loads line def pct values, q : if not values: return 0.0 s = sorted values i = min len s - 1, int round q len s - 1 return s i def main path, wait share max=0.60, margin frac=0.15 : rows = list load path waits, gen, share = , , for r in rows: total = r "accept ts" - r "enqueue ts" if total <= 0: continue wait = r "start ts" - r "enqueue ts" waits.append wait gen.append total - wait share.append wait / total drain = max r "accept ts" for r in rows first = min r "enqueue ts" for r in rows deadline = max r "deadline ts" for r in rows slack = deadline - drain margin = deadline - first margin frac print f"jobs={len rows } wait p50={pct waits,0.5 :.0f}s wait p95={pct waits,0.95 :.0f}s" print f"gen p50={pct gen,0.5 :.0f}s wait share p95={pct share,0.95 :.2f}" print f"slack={slack:.0f}s margin budget={margin:.0f}s" if pct share, 0.95 wait share max: print "VERDICT: spill — wait dominates, throughput is not your lever" elif slack < margin: print "VERDICT: spill — projected drain lands inside the safety margin" else: print "VERDICT: hold — the lane is inside budget" if name == " main ": main sys.argv 1 Run it against a finished batch: python3 two clock.py lane-free.jsonl Expected output shape illustrative, not a measurement from your system : jobs=412 wait p50=38s wait p95=611s gen p50=22s wait share p95=0.81 slack=1740s margin budget=1080s VERDICT: spill — wait dominates, throughput is not your lever Note what the verdict ignores: tokens. They live in the same record if you need them, but they do not move this decision. Free capacity is a scheduling bet first and a billing bet second. wait share max = 0.60 says: if most of a job's life is spent queued, the model is not the constraint. Tighten it if your acceptance check is slow. Loosen it only with data. margin frac = 0.15 is a reserve against your own estimate. Drain forecasts are optimistic at the tail. Keep a slice of the window unspent so a late retry still fits. Do not tune thresholds on a quiet lane. Congest it yourself, on purpose, in your own shell. bash /usr/bin/env bash fake lane.sh — synthetic congestion, zero production impact set -euo pipefail HOLD=${HOLD SECONDS:-90} DEADLINE=$ $ date +%s + ${WINDOW SECONDS:-3600} while read -r id; do ENQ=$ date +%s sleep "$HOLD" echo "{\"job id\":\"$id\",\"enqueue ts\":$ENQ,\"start ts\":$ date +%s ,\"accept ts\":$ date +%s ,\"deadline ts\":$DEADLINE}" done Generate job ids, then run the same 40 jobs twice: seq 1 40 ids.txt HOLD SECONDS=5 ./fake lane.sh < ids.txt lane-fast.jsonl HOLD SECONDS=90 ./fake lane.sh < ids.txt lane-slow.jsonl python3 two clock.py lane-fast.jsonl python3 two clock.py lane-slow.jsonl The fast file should print hold . The slow file should print spill . If both print the same verdict, your gate is dead code and you should fix it before touching production routing. | Signal | Threshold | Action | |---|---|---| | p95 wait share | 0.60 | Spill the deadline-critical slice to paid capacity | | Slack vs margin | slack < margin | Spill now, not at the next cron tick | | Retry rate | rising while wait share rises | Fix admission, not the model | | Rework per accepted artifact | above your baseline | Free tokens are buying wasted artifacts | | Interactive p99 latency | any dependency | Do not route interactive traffic here at all | Read the table top-down. The first row that trips decides. You do not average these signals into one score. You can rehearse this whole loop without a paid commitment. MonkeyCode is an open-source project, and its operator states that it offers free model access, a free server option, and a free allowance of 10M tokens. Verify the current limits on the project page yourself before you plan any capacity around them; treat them as a starting condition, not a contract. Disclosure: This article was prepared as part of MonkeyCode's product outreach. What the free lane genuinely buys you here is rehearsal time. You get real request paths, real timestamps, and a real queue to measure. What it does not buy you is a guarantee about wait distributions under your deadline. That number only exists after you run the ledger on your own workload. One practical pairing: keep the free server as the lane under test, and keep the spill path wired to whatever paid capacity you already trust. The drill is only useful if the spill branch is reachable in one config change. Keep the escape hatch boring and tested. FREE LANE ENABLED=0 . two clock.py on the spill window and compare wait shares. If the spill window shows the same p95 wait share, the queue was never the free lane's fault. Look upstream at your own admission logic. If you already run a batch queue, this is a one-afternoon exercise. Write four timestamps, run one script, and let the verdict argue with your assumptions.