# Deliver Completions With a Job Row, Not a Held Connection > Source: > Published: 2026-09-21 12:18:12+00:00 Last Tuesday I sat with a teammate who clicked Generate on a supposedly simple report screen. The spinner behaved like a prayer, nginx hit sixty seconds, and the gateway returned 504 while a model process kept talking to nobody. Did the user get a paragraph? No. Did a refresh spend another free inference call on the same prompt? Yes, immediately, because the POST had been the product. I am done treating that architecture as a prototype shortcut. If a browser is holding a TCP connection while a model thinks, you do not have a feature. You have a costume. Free tokens do not change the physics of a reverse proxy, and a free server does not make a blocked request thread honest. Why do we keep doing this? Because the UI looks finished when a button calls `/generate` and awaits JSON. AI-assisted scaffolding makes that lie cheaper to type. The first layer that fails is not the prompt. It is the handoff between the user's POST, the process that actually calls the model, and the row that should still exist after a timeout. A user-facing completion should be accepted as work, not performed inside the request that asked for it. Return `202 Accepted` with a job identifier, persist the prompt under the caller's identity, and let a worker touch the provider. Poll or subscribe for the artifact. If that sounds heavy for a demo, the demo is already lying about production. I do not mean you need a workflow engine on day one. I mean a job row with a status, an owner, a budget, and an idempotency key. Think of it as a coat-check ticket. The coat can still be lost, but at least you stop pretending the doorway is a closet. People argue that free inference makes the queue optional. That is backwards. A constrained or cold server is exactly when the request thread will 504, and a free path is exactly when the client will retry without shame. If you cannot survive that, you cannot survive a billed provider either. Label this as a proposed working slice, not a trophy benchmark. I want one authenticated action: `POST /reports` with a prompt, then `GET /reports/{id}` until the body exists. Permissions and persistence show up before the model does, or the rest is theater. ``` CREATE TABLE report_jobs ( id TEXT PRIMARY KEY, user_id TEXT NOT NULL, idempotency_key TEXT NOT NULL, prompt TEXT NOT NULL, status TEXT NOT NULL CHECK (status IN ('queued','running','succeeded','failed')), http_status INTEGER, result_text TEXT, error_code TEXT, token_estimate INTEGER NOT NULL, created_at TEXT NOT NULL, updated_at TEXT NOT NULL, UNIQUE (user_id, idempotency_key) ); CREATE TABLE inference_budgets ( user_id TEXT PRIMARY KEY, daily_tokens INTEGER NOT NULL, spent_tokens INTEGER NOT NULL, day_utc TEXT NOT NULL ); ``` The unique pair is the whole point. A double click, a React remount, or a browser retry must collide on the same row instead of inventing a second conversation with the model. If your table cannot say no, your provider will say yes twice. ``` python # report_api.py — proposed slice, not a framework manifesto import os, uuid, datetime as dt from fastapi import FastAPI, Header, HTTPException from pydantic import BaseModel, Field app = FastAPI() DAILY_TOKEN_CAP = int(os.environ["DAILY_TOKEN_CAP"]) class CreateReport(BaseModel): prompt: str = Field(min_length=1, max_length=8000) def estimate_tokens(prompt: str) -> int: return max(1, len(prompt) // 4) @app.post("/reports", status_code=202) def enqueue_report( body: CreateReport, authorization: str = Header(...), idempotency_key: str = Header(..., alias="Idempotency-Key"), ): user_id = require_user(authorization) # 401 if the bearer is junk tokens = estimate_tokens(body.prompt) existing = db.fetch_job(user_id, idempotency_key) if existing: return {"job_id": existing.id, "status": existing.status} if not db.reserve_budget(user_id, tokens, DAILY_TOKEN_CAP): raise HTTPException(status_code=429, detail={ "error": "budget_exhausted", "retry": "tomorrow", }) job_id = str(uuid.uuid4()) now = dt.datetime.utcnow().isoformat() + "Z" db.insert_job( id=job_id, user_id=user_id, idempotency_key=idempotency_key, prompt=body.prompt, status="queued", token_estimate=tokens, created_at=now, updated_at=now, ) broker.publish("report.jobs", job_id) return {"job_id": job_id, "status": "queued"} @app.get("/reports/{job_id}") def read_report(job_id: str, authorization: str = Header(...)): user_id = require_user(authorization) job = db.get_job(job_id) if job is None or job.user_id != user_id: raise HTTPException(status_code=404) payload = {"job_id": job.id, "status": job.status, "http_status": job.http_status} if job.status == "succeeded": payload["text"] = job.result_text if job.status == "failed": payload["error_code"] = job.error_code return payload ``` Notice what the handler refuses to do. It does not import an SDK. It does not stream tokens through the user's socket. It reserves a cheap local estimate against a daily envelope, then it leaves. If reservation feels pessimistic, good. A free path still creates paid side effects in logs, disks, and support, and I want those side effects named. The worker is where the provider seam lives. Keep the URL and the key in the environment. Do not sprinkle them through route files like seasoning. ``` python # report_worker.py — proposed slice import os, time, httpx PROVIDER_URL = os.environ["INFERENCE_URL"] PROVIDER_KEY = os.environ["INFERENCE_KEY"] TIMEOUT_S = float(os.environ.get("INFERENCE_TIMEOUT_S", "45")) def handle(job_id: str) -> None: job = db.claim_job(job_id) # queued -> running, skip if already terminal if job is None: return try: response = httpx.post( PROVIDER_URL, headers={"Authorization": f"Bearer {PROVIDER_KEY}"}, json={"prompt": job.prompt}, timeout=TIMEOUT_S, ) if response.status_code >= 500: db.fail_job(job_id, error_code="provider_5xx", http_status=response.status_code) db.release_budget(job.user_id, job.token_estimate) # only on hard failure return if response.status_code != 200: db.fail_job(job_id, error_code="provider_rejected", http_status=response.status_code) db.release_budget(job.user_id, job.token_estimate) return text = response.json()["text"] db.complete_job(job_id, text=text, http_status=200) except httpx.TimeoutException: db.fail_job(job_id, error_code="provider_timeout", http_status=504) db.release_budget(job.user_id, job.token_estimate) ``` That `claim_job` update is the difference between a queue and a wish. If two workers see the same message after a crash, only one should move `queued` to `running`. Completions are not naturally idempotent. Your table has to be. I still want a constrained inference box in staging, because timeouts and cold processes show up there first. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project that currently offers free model access and a free server option, which is enough to point `INFERENCE_URL` at a real process without wiring a billed provider into the first vertical slice. I will not pretend that free access is a capacity plan, a latency SLO, or a reason to skip auth. It is a seam. If the worker can fail closed against that seam, swapping the URL later is boring, and boring is the goal. If you cannot name the status code you expect when the free server is down, you are not ready to point a UI at it. Use it to rehearse the ugly path, not to skip the job row. A free server that you await inside `POST /reports` is still a held connection. You just stopped paying the model vendor for the privilege of hanging nginx. I do not start with a golden prompt. I start with the user action and the first lie. Replay the same `Idempotency-Key` twice and assert one row, one `202`, and one provider call. Kill the worker after `running` and assert the client still reads a job, not a blank 500 from a vanished POST. Return a provider 503 and assert the job is `failed` with `provider_5xx`, budget released, and the UI not spinning forever. Wait past your gateway timeout on purpose and assert the user never held that wait on their socket. ``` # test_report_contract.py — proposed tests def test_duplicate_key_does_not_double_spend(client, db, provider): headers = { "Authorization": "Bearer user-1", "Idempotency-Key": "click-77", } a = client.post("/reports", json={"prompt": "summarize the invoice"}, headers=headers) b = client.post("/reports", json={"prompt": "summarize the invoice"}, headers=headers) assert a.status_code == 202 assert b.status_code == 202 assert a.json()["job_id"] == b.json()["job_id"] assert provider.call_count == 0 # enqueue only worker.drain_one() assert provider.call_count == 1 worker.drain_one() assert provider.call_count == 1 def test_reader_does_not_see_someone_elses_job(client, db): job_id = db.insert_owned(user_id="user-1", prompt="secret") r = client.get(f"/reports/{job_id}", headers={"Authorization": "Bearer user-2"}) assert r.status_code == 404 ``` If those tests feel like ceremony, ask what your spinner does on a 504 today. Ceremony is cheaper than two silent completions and a support thread that cannot find a job id. If you are writing a local CLI that you watch in a terminal, await the model. If you are running an overnight batch with no interactive client, a queue is still nice, but a blocked process is not a UX lie. This argument is for anything a human clicks, especially anything you will one day put behind a shared domain. Also skip this if you cannot store the prompt. A job row that holds user text is a data store, with retention and access control attached. Moving secrets from a request body into a table is not an upgrade unless you meant to keep them. A `202` does not validate output. A budget envelope is not a legal policy. A free server can disappear without a deprecation essay, so keep the provider behind that environment variable and keep the client on job state. I am not offering model names, quotas, or duration claims, because those go stale while the contract stays useful. The maintainability tradeoff is real. You now operate a worker, a broker, and a table. In return you get a place to put timeouts, retries, and permissions that is not the browser. I will take that trade every time the alternative is a spinner negotiating with nginx. Reuse this when you cut the slice: authenticate first, persist the prompt, reserve a budget, return `202`, claim the row, call one provider, and test the duplicate key until the call count stays at one. Then, and only then, let the UI poll. If a layer cannot explain its failure with a status code, it is not a layer yet. Which handoff is least stable in your stack right now, the POST to the queue, the worker to the provider, or the poll back to the UI? Send me a concrete failure state or a response code, not a vibe. If you want a constrained box for the worker URL while you rehearse that path, MonkeyCode's free model access and free server option are sufficient to fail in staging instead of in front of a customer.