{"slug": "time-in-queue-is-not-free", "title": "Time in Queue Is Not Free", "summary": "A proposed instrumentation pattern for shared inference queues argues that free model access and free servers absorb bursts by making jobs wait rather than adding workers, so teams should track queue wait, generation time, and retry time separately instead of only counting tokens. The pattern wraps an existing client and writes one JSON line per attempt, including failures, using an InferenceSpan dataclass with fields for queued_at, started_at, first_token_at, finished_at, tokens_in, tokens_out, retries, ok, and error, plus wait_s(), generate_s(), and wait_ratio() methods. The author recommends folding a day of spans by job name and lane and examining p50 and p95 wait, generate time, wait_ratio, and retry count rather than averages, since averages hide the single job that blocked a person.", "body_md": "# Time in Queue Is Not Free\n\nFree capacity is cheap only when nobody is waiting on it. The moment a job sits in a shared queue, you start paying in wall-clock time: CI runners stay reserved, reviewers refresh a spinner, and retries take another ticket in the same line. The token meter can read zero while the pipeline gets more\n\nFree capacity is cheap only when nobody is waiting on it. The moment a job sits in a shared queue, you start paying in wall-clock time: CI runners stay reserved, reviewers refresh a spinner, and retries take another ticket in the same line. The token meter can read zero while the pipeline gets more expensive. A free lane is a courtesy checkout. It is not a reserved counter. If twelve other carts are ahead of you, the unpaid register still costs you the afternoon. Shared inference works the same way. Free model access and a free server absorb burst by making you wait, not by printing extra workers on demand. You already track tokens. Track the wait. The useful split is not prompt versus completion. It is queued versus generating versus retrying. Queue wait is the gap between \"I submitted this\" and \"a worker actually started.\" Generation time is the model doing work. Retry time is the same job buying another ticket. If wait dominates, you do not have a model-cost problem. You have a scheduling problem, and free capacity is often the wrong bet. The block below is a proposed instrumentation pattern, not a production war story. It does not assume a vendor SDK. You wrap the client you already use and write one JSON line per attempt, including failures. # Proposed span schema. Unexecuted example — wire it to your client. from __future__ import annotations from dataclasses import asdict, dataclass from time import time import json import os from typing import Callable, TypeVar T = TypeVar(\"T\") @dataclass class InferenceSpan: job: str lane: str # \"free\" | \"paid\" | \"local\" queued_at: float started_at: float | None = None first_token_at: float | None = None finished_at: float | None = None tokens_in: int = 0 tokens_out: int = 0 retries: int = 0 ok: bool = False error: str | None = None def wait_s(self) -> float: if self.started_at is None: return max(0.0, (self.finished_at or time()) - self.queued_at) return max(0.0, self.started_at - self.queued_at) def generate_s(self) -> float: if self.started_at is None or self.finished_at is None: return 0.0 return max(0.0, self.finished_at - self.started_at) def wait_ratio(self) -> float: total = self.wait_s() + self.generate_s() return 0.0 if total <= 0 else self.wait_s() / total def timed_call(job: str, lane: str, send: Callable[[], T], span_path: str) -> T: span = InferenceSpan(job=job, lane=lane, queued_at=time()) try: span.started_at = time() # move this to first byte if your client can result = send() span.first_token_at = span.first_token_at or span.started_at span.finished_at = time() span.ok = True return result except Exception as exc: span.finished_at = time() span.error = type(exc).__name__ raise finally: with open(span_path, \"a\", encoding=\"utf-8\") as fh: fh.write(json.dumps(asdict(span)) + \"\\n\") os.environ[\"LAST_WAIT_RATIO\"] = f\"{span.wait_ratio():.3f}\" Emit the span when the call finishes, even on failure. Especially on failure. A timeout that never started is still a wait. A throttle that bounced you to the tail of the queue is a wait plus a retry. If you only log successful completions, you hide the expensive part of the free lane. Once you have a day of spans, fold them. Do not start with averages. Averages hide the one job that blocked a person. Group by job name and lane, then look at p50 and p95 wait, generate time, wait_ratio, and retry count. The script below is a proposal you can run against newline-delimited logs. # Proposed log fold. Unexecuted example. import json, statistics, sys from collections import defaultdict rows = defaultdict(list) for line in sys.stdin: s = json.loads(line) wait = 0.0 if s[\"started_at\"] is None else max(0.0, s[\"started_at\"] - s[\"queued_at\"]) gen = 0.0 if not s[\"started_at\"] or not s[\"finished_at\"] else max(0.0, s[\"finished_at\"] - s[\"started_at\"]) total = wait + gen ratio = 0.0 if total <= 0 else wait / total rows[(s[\"job\"], s[\"lane\"])].append((wait, gen, ratio, int(s.get(\"retries\") or 0), bool(s[\"ok\"]))) def pct(xs, p): if not xs: return 0.0 xs = sorted(xs) i = min(len(xs) - 1, max(0, round((p / 100) * (len(xs) - 1)))) return xs[i] print(\"job,lane,n,p50_wait_s,p95_wait_s,p50_ratio,retries,ok_rate\") for (job, lane), items in sorted(rows.items()): waits, gens, ratios, retries, oks = zip(*items) ok_rate = sum(oks) / len(oks) print(f\"{job},{lane},{len(items)},{pct(waits,50):.2f},{pct(waits,95):.2f},{pct(ratios,50):.2f},{sum(retries)},{ok_rate:.2f}\") Read wait_ratio as a tax rate. If a nightly batch sits for eight minutes and generates for forty seconds, the tax is ugly on paper and still fine in practice, because nothing else was blocked. If a pull-request summary sits for eight minutes while the author is in the review thread, the tax is the author's afternoon. Same queue. Different bill. Translate that into money without inventing a vendor price. You already know your CI rate and a rough loaded cost for an engineer minute. Wait cost is wait_seconds times those rates. Token savings is the tokens you did not send to a paid endpoint. When the job is on a blocked path and wait_cost exceeds token_savings, leave the free lane. When the job can drift to a later hour, stay. That comparison is the whole policy. You do not need a platform committee to run it. # Proposed decision, not a benchmark. Fill in your own rates. wait_cost = wait_seconds * (ci_usd_per_minute / 60 + engineer_usd_per_hour / 3600) token_savings = tokens * paid_usd_per_token blocked = job_holds_human_or_release_gate if blocked and wait_cost > token_savings: move job off the free lane elif not blocked: keep the free lane and give the job a deadline you can miss Retries make the comparison worse, because each retry is a fresh ticket. A cheap job that times out in a long queue does not fail once. It fails, waits, fails, waits. You have seen this in build systems: a flaky test is not expensive because the assertion is hard. It is expensive because the queue keeps taking it back. Treat model retries the same way. Cap them on the free lane. If you need reliability, you need a lane with admission control, not more optimism. Here is a small gate you can drop into CI. It reads the span file for jobs that block merge, and it fails the step when wait_ratio on the free lane crosses a threshold you chose. The numbers are placeholders. Replace them after you look at a week of your own spans. #!/usr/bin/env bash # Proposed CI gate. Unexecuted example. set -euo pipefail SPAN_FILE=\"${SPAN_FILE:-.inference-spans.jsonl}\" MAX_RATIO=\"${MAX_RATIO:-0.70}\" BLOCKING_JOBS=\"${BLOCKING_JOBS:-pr-summary ci-review-bot}\" if [[ ! -f \"$SPAN_FILE\" ]]; then echo \"no span file; refusing to claim the free lane was cheap\" >&2 exit 2 fi python - \"$SPAN_FILE\" \"$MAX_RATIO\" $BLOCKING_JOBS <<'PY' import json, sys path, max_ratio, *jobs = sys.argv[1:] blocking = set(jobs) max_ratio = float(max_ratio) bad = [] with open(path, encoding=\"utf-8\") as fh: for line in fh: s = json.loads(line) if s.get(\"lane\") != \"free\" or s.get(\"job\") not in blocking: continue started, queued, finished = s.get(\"started_at\"), s[\"queued_at\"], s.get(\"finished_at\") wait = (finished or started or queued) - queued if started is None else started - queued gen = 0.0 if not started or not finished else finished - started total = max(0.0, wait) + max(0.0, gen) ratio = 0.0 if total <= 0 else wait / total if ratio > max_ratio: bad.append((s[\"job\"], ratio, wait)) if bad: for job, ratio, wait in bad: print(f\"free lane too slow for {job}: wait_ratio={ratio:.2f} wait_s={wait:.1f}\") sys.exit(1) print(\"free lane wait_ratio within budget for blocking jobs\") PY That gate is not a performance test of a model. It is a policy test of a lane. You are asking a simple question: did this work wait so long that \"free\" became a fiction? If yes, reroute that job. Keep the free lane for work that can be late. If you need a place to park the late work, MonkeyCode currently offers free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Use that overflow the same way you would use any other unpaid shared lane: batch it, label the span, and keep it off the critical path. This article does not attach model names, quotas, hardware, or uptime claims, because those change and your span file is the only meter that matters. A practical split looks like this. Overnight evals, prompt diffs you can rerun, log summarization, and backfills can live on free model access or a free server. Anything a person stares at, anything a pager can fire on, and anything that holds a deploy should not. You can move a job between those two columns without rewriting the product. You change the lane label in the span and the endpoint the wrapper calls. Limitations follow from the shared nature of the lane. Free capacity has no promise of fairness. Your wait_ratio today is not a contract for next Tuesday. Logs will lie if your client reports \"start\" at enqueue instead of at first byte. If you cannot tell those timestamps apart, you will blame the model for queueing. Do not put secrets, customer transcripts, or regulated records on a free shared server just because the token meter is quiet. Cost is not the same as permission. Skip this approach if you already run dedicated capacity and your queues are empty. You are solving the wrong problem. Skip it if your product is interactive chat with a latency SLO; do not A/B the SLO against a free lane \"to see.\" You already know the answer. Skip it if you cannot add timestamps to the client. A policy without spans is a slogan. The habit is small. Write queued_at. Write started_at. Write the ratio. Then decide whether the job may wait. Free inference is a tool for work that can be late. It is a bad hiding place for work that cannot. If you try an overflow lane, export wait_ratio before you move a merge job onto it.\n\n## Key Takeaways\n\n- •Free capacity is cheap only when nobody is waiting on it\n- •This story was reported by **Dev.to** , covering developments in the**dev** space.\n- •AI advancements continue to reshape industries — read the full article on Dev.to for complete coverage.\n\n📖 Continue reading the full article:\n\n[Read Full Article on Dev.to →](https://dev.to/hackrs_3352/time-in-queue-is-not-free-3fo0)", "url": "https://wpnews.pro/news/time-in-queue-is-not-free", "canonical_source": "https://ainexusdaily.vercel.app/article/2026-09-19-time-in-queue-is-not-free", "published_at": "2026-09-19 09:44:30+00:00", "updated_at": "2026-09-19 10:54:06.351715+00:00", "lang": "en", "topics": ["ai-infrastructure", "mlops", "ai-tools"], "entities": ["InferenceSpan", "timed_call", "LAST_WAIT_RATIO"], "alternates": {"html": "https://wpnews.pro/news/time-in-queue-is-not-free", "markdown": "https://wpnews.pro/news/time-in-queue-is-not-free.md", "text": "https://wpnews.pro/news/time-in-queue-is-not-free.txt", "jsonld": "https://wpnews.pro/news/time-in-queue-is-not-free.jsonld"}}