Your Free AI Lane Is Slow in the Queue, Not in the Model

You moved a batch to free capacity. The token bill dropped to zero. The deadline still slipped.

Nothing in the model got slower. The jobs sat in a queue you never charted.

Most AI cost dashboards start counting at the first token. That is the wrong start line for a free lane. On free capacity, the wall clock usually belongs to scheduling, not to generation.

This post is a drill, not a vendor pitch. You record two clocks per job. Then you run one script and get a verdict: hold the batch, or spill it to paid capacity before the cutoff.

Your job record needs five timestamps. Anything less and you cannot separate waiting from working.

enqueue_ts — the job was accepted by your scheduler.start_ts — the first request hit the serving path.accept_ts — the artifact passed your acceptance check.deadline_ts — the wall-clock deadline for that batch.attempt — the retry index, so rework stays visible. From those you get three derived numbers:

wait = start_ts - enqueue_ts`` gen = accept_ts - start_ts``wait_share = wait / (accept_ts - enqueue_ts) Gate on wait_share. A lane with p95 wait share of 0.7 is a scheduling problem wearing a model costume. Buying faster tokens there changes almost nothing.

The script below reads JSON Lines and prints a verdict. It is deliberately dumb. No dashboards, no agent, no database.

#!/usr/bin/env python3
import json, sys

def load(path):
    with open(path) as fh:
        for line in fh:
            line = line.strip()
            if line:
                yield json.loads(line)

def pct(values, q):
    if not values:
        return 0.0
    s = sorted(values)
    i = min(len(s) - 1, int(round(q * (len(s) - 1))))
    return s[i]

def main(path, wait_share_max=0.60, margin_frac=0.15):
    rows = list(load(path))
    waits, gen, share = [], [], []
    for r in rows:
        total = r["accept_ts"] - r["enqueue_ts"]
        if total <= 0:
            continue
        wait = r["start_ts"] - r["enqueue_ts"]
        waits.append(wait)
        gen.append(total - wait)
        share.append(wait / total)

    drain = max(r["accept_ts"] for r in rows)
    first = min(r["enqueue_ts"] for r in rows)
    deadline = max(r["deadline_ts"] for r in rows)
    slack = deadline - drain
    margin = (deadline - first) * margin_frac

    print(f"jobs={len(rows)} wait_p50={pct(waits,0.5):.0f}s wait_p95={pct(waits,0.95):.0f}s")
    print(f"gen_p50={pct(gen,0.5):.0f}s wait_share_p95={pct(share,0.95):.2f}")
    print(f"slack={slack:.0f}s margin_budget={margin:.0f}s")

    if pct(share, 0.95) > wait_share_max:
        print("VERDICT: spill — wait dominates, throughput is not your lever")
    elif slack < margin:
        print("VERDICT: spill — projected drain lands inside the safety margin")
    else:
        print("VERDICT: hold — the lane is inside budget")

if __name__ == "__main__":
    main(sys.argv[1])

Run it against a finished batch:

python3 two_clock.py lane-free.jsonl

Expected output shape (illustrative, not a measurement from your system):

jobs=412 wait_p50=38s wait_p95=611s gen_p50=22s wait_share_p95=0.81
slack=1740s margin_budget=1080s
VERDICT: spill — wait dominates, throughput is not your lever

Note what the verdict ignores: tokens. They live in the same record if you need them, but they do not move this decision. Free capacity is a scheduling bet first and a billing bet second.

wait_share_max = 0.60 says: if most of a job's life is spent queued, the model is not the constraint. Tighten it if your acceptance check is slow. Loosen it only with data.

margin_frac = 0.15 is a reserve against your own estimate. Drain forecasts are optimistic at the tail. Keep a slice of the window unspent so a late retry still fits.

Do not tune thresholds on a quiet lane. Congest it yourself, on purpose, in your own shell.

#!/usr/bin/env bash
set -euo pipefail
HOLD=${HOLD_SECONDS:-90}
DEADLINE=$(( $(date +%s) + ${WINDOW_SECONDS:-3600} ))
while read -r id; do
  ENQ=$(date +%s)
  sleep "$HOLD"
  echo "{\"job_id\":\"$id\",\"enqueue_ts\":$ENQ,\"start_ts\":$(date +%s),\"accept_ts\":$(date +%s),\"deadline_ts\":$DEADLINE}"
done

Generate job ids, then run the same 40 jobs twice:

seq 1 40 > ids.txt
HOLD_SECONDS=5  ./fake_lane.sh < ids.txt > lane-fast.jsonl
HOLD_SECONDS=90 ./fake_lane.sh < ids.txt > lane-slow.jsonl
python3 two_clock.py lane-fast.jsonl
python3 two_clock.py lane-slow.jsonl

The fast file should print hold. The slow file should print spill. If both print the same verdict, your gate is dead code and you should fix it before touching production routing.

Read the table top-down. The first row that trips decides. You do not average these signals into one score.

You can rehearse this whole loop without a paid commitment. MonkeyCode is an open-source project, and its operator states that it offers free model access, a free server option, and a free allowance of 10M tokens. Verify the current limits on the project page yourself before you plan any capacity around them; treat them as a starting condition, not a contract.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What the free lane genuinely buys you here is rehearsal time. You get real request paths, real timestamps, and a real queue to measure. What it does not buy you is a guarantee about wait distributions under your deadline. That number only exists after you run the ledger on your own workload.

One practical pairing: keep the free server as the lane under test, and keep the spill path wired to whatever paid capacity you already trust. The drill is only useful if the spill branch is reachable in one config change.

Keep the escape hatch boring and tested.

FREE_LANE_ENABLED=0. two_clock.py on the spill window and compare wait shares. If the spill window shows the same p95 wait share, the queue was never the free lane's fault. Look upstream at your own admission logic.

If you already run a batch queue, this is a one-afternoon exercise. Write four timestamps, run one script, and let the verdict argue with your assumptions.