Time in Queue Is Not Free

A proposed instrumentation pattern for shared inference queues argues that free model access and free servers absorb bursts by making jobs wait rather than adding workers, so teams should track queue wait, generation time, and retry time separately instead of only counting tokens. The pattern wraps an existing client and writes one JSON line per attempt, including failures, using an InferenceSpan dataclass with fields for queued_at, started_at, first_token_at, finished_at, tokens_in, tokens_out, retries, ok, and error, plus wait_s(), generate_s(), and wait_ratio() methods. The author recommends folding a day of spans by job name and lane and examining p50 and p95 wait, generate time, wait_ratio, and retry count rather than averages, since averages hide the single job that blocked a person.

Time in Queue Is Not Free Free capacity is cheap only when nobody is waiting on it. The moment a job sits in a shared queue, you start paying in wall-clock time: CI runners stay reserved, reviewers refresh a spinner, and retries take another ticket in the same line. The token meter can read zero while the pipeline gets more Free capacity is cheap only when nobody is waiting on it. The moment a job sits in a shared queue, you start paying in wall-clock time: CI runners stay reserved, reviewers refresh a spinner, and retries take another ticket in the same line. The token meter can read zero while the pipeline gets more expensive. A free lane is a courtesy checkout. It is not a reserved counter. If twelve other carts are ahead of you, the unpaid register still costs you the afternoon. Shared inference works the same way. Free model access and a free server absorb burst by making you wait, not by printing extra workers on demand. You already track tokens. Track the wait. The useful split is not prompt versus completion. It is queued versus generating versus retrying. Queue wait is the gap between "I submitted this" and "a worker actually started." Generation time is the model doing work. Retry time is the same job buying another ticket. If wait dominates, you do not have a model-cost problem. You have a scheduling problem, and free capacity is often the wrong bet. The block below is a proposed instrumentation pattern, not a production war story. It does not assume a vendor SDK. You wrap the client you already use and write one JSON line per attempt, including failures. Proposed span schema. Unexecuted example — wire it to your client. from future import annotations from dataclasses import asdict, dataclass from time import time import json import os from typing import Callable, TypeVar T = TypeVar "T" @dataclass class InferenceSpan: job: str lane: str "free" | "paid" | "local" queued at: float started at: float | None = None first token at: float | None = None finished at: float | None = None tokens in: int = 0 tokens out: int = 0 retries: int = 0 ok: bool = False error: str | None = None def wait s self - float: if self.started at is None: return max 0.0, self.finished at or time - self.queued at return max 0.0, self.started at - self.queued at def generate s self - float: if self.started at is None or self.finished at is None: return 0.0 return max 0.0, self.finished at - self.started at def wait ratio self - float: total = self.wait s + self.generate s return 0.0 if total <= 0 else self.wait s / total def timed call job: str, lane: str, send: Callable , T , span path: str - T: span = InferenceSpan job=job, lane=lane, queued at=time try: span.started at = time move this to first byte if your client can result = send span.first token at = span.first token at or span.started at span.finished at = time span.ok = True return result except Exception as exc: span.finished at = time span.error = type exc . name raise finally: with open span path, "a", encoding="utf-8" as fh: fh.write json.dumps asdict span + "\n" os.environ "LAST WAIT RATIO" = f"{span.wait ratio :.3f}" Emit the span when the call finishes, even on failure. Especially on failure. A timeout that never started is still a wait. A throttle that bounced you to the tail of the queue is a wait plus a retry. If you only log successful completions, you hide the expensive part of the free lane. Once you have a day of spans, fold them. Do not start with averages. Averages hide the one job that blocked a person. Group by job name and lane, then look at p50 and p95 wait, generate time, wait ratio, and retry count. The script below is a proposal you can run against newline-delimited logs. Proposed log fold. Unexecuted example. import json, statistics, sys from collections import defaultdict rows = defaultdict list for line in sys.stdin: s = json.loads line wait = 0.0 if s "started at" is None else max 0.0, s "started at" - s "queued at" gen = 0.0 if not s "started at" or not s "finished at" else max 0.0, s "finished at" - s "started at" total = wait + gen ratio = 0.0 if total <= 0 else wait / total rows s "job" , s "lane" .append wait, gen, ratio, int s.get "retries" or 0 , bool s "ok" def pct xs, p : if not xs: return 0.0 xs = sorted xs i = min len xs - 1, max 0, round p / 100 len xs - 1 return xs i print "job,lane,n,p50 wait s,p95 wait s,p50 ratio,retries,ok rate" for job, lane , items in sorted rows.items : waits, gens, ratios, retries, oks = zip items ok rate = sum oks / len oks print f"{job},{lane},{len items },{pct waits,50 :.2f},{pct waits,95 :.2f},{pct ratios,50 :.2f},{sum retries },{ok rate:.2f}" Read wait ratio as a tax rate. If a nightly batch sits for eight minutes and generates for forty seconds, the tax is ugly on paper and still fine in practice, because nothing else was blocked. If a pull-request summary sits for eight minutes while the author is in the review thread, the tax is the author's afternoon. Same queue. Different bill. Translate that into money without inventing a vendor price. You already know your CI rate and a rough loaded cost for an engineer minute. Wait cost is wait seconds times those rates. Token savings is the tokens you did not send to a paid endpoint. When the job is on a blocked path and wait cost exceeds token savings, leave the free lane. When the job can drift to a later hour, stay. That comparison is the whole policy. You do not need a platform committee to run it. Proposed decision, not a benchmark. Fill in your own rates. wait cost = wait seconds ci usd per minute / 60 + engineer usd per hour / 3600 token savings = tokens paid usd per token blocked = job holds human or release gate if blocked and wait cost token savings: move job off the free lane elif not blocked: keep the free lane and give the job a deadline you can miss Retries make the comparison worse, because each retry is a fresh ticket. A cheap job that times out in a long queue does not fail once. It fails, waits, fails, waits. You have seen this in build systems: a flaky test is not expensive because the assertion is hard. It is expensive because the queue keeps taking it back. Treat model retries the same way. Cap them on the free lane. If you need reliability, you need a lane with admission control, not more optimism. Here is a small gate you can drop into CI. It reads the span file for jobs that block merge, and it fails the step when wait ratio on the free lane crosses a threshold you chose. The numbers are placeholders. Replace them after you look at a week of your own spans. /usr/bin/env bash Proposed CI gate. Unexecuted example. set -euo pipefail SPAN FILE="${SPAN FILE:-.inference-spans.jsonl}" MAX RATIO="${MAX RATIO:-0.70}" BLOCKING JOBS="${BLOCKING JOBS:-pr-summary ci-review-bot}" if -f "$SPAN FILE" ; then echo "no span file; refusing to claim the free lane was cheap" &2 exit 2 fi python - "$SPAN FILE" "$MAX RATIO" $BLOCKING JOBS <<'PY' import json, sys path, max ratio, jobs = sys.argv 1: blocking = set jobs max ratio = float max ratio bad = with open path, encoding="utf-8" as fh: for line in fh: s = json.loads line if s.get "lane" = "free" or s.get "job" not in blocking: continue started, queued, finished = s.get "started at" , s "queued at" , s.get "finished at" wait = finished or started or queued - queued if started is None else started - queued gen = 0.0 if not started or not finished else finished - started total = max 0.0, wait + max 0.0, gen ratio = 0.0 if total <= 0 else wait / total if ratio max ratio: bad.append s "job" , ratio, wait if bad: for job, ratio, wait in bad: print f"free lane too slow for {job}: wait ratio={ratio:.2f} wait s={wait:.1f}" sys.exit 1 print "free lane wait ratio within budget for blocking jobs" PY That gate is not a performance test of a model. It is a policy test of a lane. You are asking a simple question: did this work wait so long that "free" became a fiction? If yes, reroute that job. Keep the free lane for work that can be late. If you need a place to park the late work, MonkeyCode currently offers free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Use that overflow the same way you would use any other unpaid shared lane: batch it, label the span, and keep it off the critical path. This article does not attach model names, quotas, hardware, or uptime claims, because those change and your span file is the only meter that matters. A practical split looks like this. Overnight evals, prompt diffs you can rerun, log summarization, and backfills can live on free model access or a free server. Anything a person stares at, anything a pager can fire on, and anything that holds a deploy should not. You can move a job between those two columns without rewriting the product. You change the lane label in the span and the endpoint the wrapper calls. Limitations follow from the shared nature of the lane. Free capacity has no promise of fairness. Your wait ratio today is not a contract for next Tuesday. Logs will lie if your client reports "start" at enqueue instead of at first byte. If you cannot tell those timestamps apart, you will blame the model for queueing. Do not put secrets, customer transcripts, or regulated records on a free shared server just because the token meter is quiet. Cost is not the same as permission. Skip this approach if you already run dedicated capacity and your queues are empty. You are solving the wrong problem. Skip it if your product is interactive chat with a latency SLO; do not A/B the SLO against a free lane "to see." You already know the answer. Skip it if you cannot add timestamps to the client. A policy without spans is a slogan. The habit is small. Write queued at. Write started at. Write the ratio. Then decide whether the job may wait. Free inference is a tool for work that can be late. It is a bad hiding place for work that cannot. If you try an overflow lane, export wait ratio before you move a merge job onto it. Key Takeaways - •Free capacity is cheap only when nobody is waiting on it - •This story was reported by Dev.to , covering developments in the dev space. - •AI advancements continue to reshape industries — read the full article on Dev.to for complete coverage. 📖 Continue reading the full article: Read Full Article on Dev.to → https://dev.to/hackrs 3352/time-in-queue-is-not-free-3fo0