Deliver Completions With a Job Row, Not a Held Connection

A developer argues that LLM completions should be handled as asynchronous jobs rather than held HTTP connections, proposing a 202 Accepted pattern with a persisted job row, idempotency key, and token budget. The proposed slice uses a report_jobs table with a unique (user_id, idempotency_key) constraint and a FastAPI endpoint that enqueues work for a worker to process. The developer frames this as a working slice, not a benchmark, to prevent duplicate inference calls on retries and timeouts.

Last Tuesday I sat with a teammate who clicked Generate on a supposedly simple report screen. The spinner behaved like a prayer, nginx hit sixty seconds, and the gateway returned 504 while a model process kept talking to nobody. Did the user get a paragraph? No. Did a refresh spend another free inference call on the same prompt? Yes, immediately, because the POST had been the product. I am done treating that architecture as a prototype shortcut. If a browser is holding a TCP connection while a model thinks, you do not have a feature. You have a costume. Free tokens do not change the physics of a reverse proxy, and a free server does not make a blocked request thread honest. Why do we keep doing this? Because the UI looks finished when a button calls /generate and awaits JSON. AI-assisted scaffolding makes that lie cheaper to type. The first layer that fails is not the prompt. It is the handoff between the user's POST, the process that actually calls the model, and the row that should still exist after a timeout. A user-facing completion should be accepted as work, not performed inside the request that asked for it. Return 202 Accepted with a job identifier, persist the prompt under the caller's identity, and let a worker touch the provider. Poll or subscribe for the artifact. If that sounds heavy for a demo, the demo is already lying about production. I do not mean you need a workflow engine on day one. I mean a job row with a status, an owner, a budget, and an idempotency key. Think of it as a coat-check ticket. The coat can still be lost, but at least you stop pretending the doorway is a closet. People argue that free inference makes the queue optional. That is backwards. A constrained or cold server is exactly when the request thread will 504, and a free path is exactly when the client will retry without shame. If you cannot survive that, you cannot survive a billed provider either. Label this as a proposed working slice, not a trophy benchmark. I want one authenticated action: POST /reports with a prompt, then GET /reports/{id} until the body exists. Permissions and persistence show up before the model does, or the rest is theater. CREATE TABLE report jobs id TEXT PRIMARY KEY, user id TEXT NOT NULL, idempotency key TEXT NOT NULL, prompt TEXT NOT NULL, status TEXT NOT NULL CHECK status IN 'queued','running','succeeded','failed' , http status INTEGER, result text TEXT, error code TEXT, token estimate INTEGER NOT NULL, created at TEXT NOT NULL, updated at TEXT NOT NULL, UNIQUE user id, idempotency key ; CREATE TABLE inference budgets user id TEXT PRIMARY KEY, daily tokens INTEGER NOT NULL, spent tokens INTEGER NOT NULL, day utc TEXT NOT NULL ; The unique pair is the whole point. A double click, a React remount, or a browser retry must collide on the same row instead of inventing a second conversation with the model. If your table cannot say no, your provider will say yes twice. python report api.py — proposed slice, not a framework manifesto import os, uuid, datetime as dt from fastapi import FastAPI, Header, HTTPException from pydantic import BaseModel, Field app = FastAPI DAILY TOKEN CAP = int os.environ "DAILY TOKEN CAP" class CreateReport BaseModel : prompt: str = Field min length=1, max length=8000 def estimate tokens prompt: str - int: return max 1, len prompt // 4 @app.post "/reports", status code=202 def enqueue report body: CreateReport, authorization: str = Header ... , idempotency key: str = Header ..., alias="Idempotency-Key" , : user id = require user authorization 401 if the bearer is junk tokens = estimate tokens body.prompt existing = db.fetch job user id, idempotency key if existing: return {"job id": existing.id, "status": existing.status} if not db.reserve budget user id, tokens, DAILY TOKEN CAP : raise HTTPException status code=429, detail={ "error": "budget exhausted", "retry": "tomorrow", } job id = str uuid.uuid4 now = dt.datetime.utcnow .isoformat + "Z" db.insert job id=job id, user id=user id, idempotency key=idempotency key, prompt=body.prompt, status="queued", token estimate=tokens, created at=now, updated at=now, broker.publish "report.jobs", job id return {"job id": job id, "status": "queued"} @app.get "/reports/{job id}" def read report job id: str, authorization: str = Header ... : user id = require user authorization job = db.get job job id if job is None or job.user id = user id: raise HTTPException status code=404 payload = {"job id": job.id, "status": job.status, "http status": job.http status} if job.status == "succeeded": payload "text" = job.result text if job.status == "failed": payload "error code" = job.error code return payload Notice what the handler refuses to do. It does not import an SDK. It does not stream tokens through the user's socket. It reserves a cheap local estimate against a daily envelope, then it leaves. If reservation feels pessimistic, good. A free path still creates paid side effects in logs, disks, and support, and I want those side effects named. The worker is where the provider seam lives. Keep the URL and the key in the environment. Do not sprinkle them through route files like seasoning. python report worker.py — proposed slice import os, time, httpx PROVIDER URL = os.environ "INFERENCE URL" PROVIDER KEY = os.environ "INFERENCE KEY" TIMEOUT S = float os.environ.get "INFERENCE TIMEOUT S", "45" def handle job id: str - None: job = db.claim job job id queued - running, skip if already terminal if job is None: return try: response = httpx.post PROVIDER URL, headers={"Authorization": f"Bearer {PROVIDER KEY}"}, json={"prompt": job.prompt}, timeout=TIMEOUT S, if response.status code = 500: db.fail job job id, error code="provider 5xx", http status=response.status code db.release budget job.user id, job.token estimate only on hard failure return if response.status code = 200: db.fail job job id, error code="provider rejected", http status=response.status code db.release budget job.user id, job.token estimate return text = response.json "text" db.complete job job id, text=text, http status=200 except httpx.TimeoutException: db.fail job job id, error code="provider timeout", http status=504 db.release budget job.user id, job.token estimate That claim job update is the difference between a queue and a wish. If two workers see the same message after a crash, only one should move queued to running . Completions are not naturally idempotent. Your table has to be. I still want a constrained inference box in staging, because timeouts and cold processes show up there first. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project that currently offers free model access and a free server option, which is enough to point INFERENCE URL at a real process without wiring a billed provider into the first vertical slice. I will not pretend that free access is a capacity plan, a latency SLO, or a reason to skip auth. It is a seam. If the worker can fail closed against that seam, swapping the URL later is boring, and boring is the goal. If you cannot name the status code you expect when the free server is down, you are not ready to point a UI at it. Use it to rehearse the ugly path, not to skip the job row. A free server that you await inside POST /reports is still a held connection. You just stopped paying the model vendor for the privilege of hanging nginx. I do not start with a golden prompt. I start with the user action and the first lie. Replay the same Idempotency-Key twice and assert one row, one 202 , and one provider call. Kill the worker after running and assert the client still reads a job, not a blank 500 from a vanished POST. Return a provider 503 and assert the job is failed with provider 5xx , budget released, and the UI not spinning forever. Wait past your gateway timeout on purpose and assert the user never held that wait on their socket. test report contract.py — proposed tests def test duplicate key does not double spend client, db, provider : headers = { "Authorization": "Bearer user-1", "Idempotency-Key": "click-77", } a = client.post "/reports", json={"prompt": "summarize the invoice"}, headers=headers b = client.post "/reports", json={"prompt": "summarize the invoice"}, headers=headers assert a.status code == 202 assert b.status code == 202 assert a.json "job id" == b.json "job id" assert provider.call count == 0 enqueue only worker.drain one assert provider.call count == 1 worker.drain one assert provider.call count == 1 def test reader does not see someone elses job client, db : job id = db.insert owned user id="user-1", prompt="secret" r = client.get f"/reports/{job id}", headers={"Authorization": "Bearer user-2"} assert r.status code == 404 If those tests feel like ceremony, ask what your spinner does on a 504 today. Ceremony is cheaper than two silent completions and a support thread that cannot find a job id. If you are writing a local CLI that you watch in a terminal, await the model. If you are running an overnight batch with no interactive client, a queue is still nice, but a blocked process is not a UX lie. This argument is for anything a human clicks, especially anything you will one day put behind a shared domain. Also skip this if you cannot store the prompt. A job row that holds user text is a data store, with retention and access control attached. Moving secrets from a request body into a table is not an upgrade unless you meant to keep them. A 202 does not validate output. A budget envelope is not a legal policy. A free server can disappear without a deprecation essay, so keep the provider behind that environment variable and keep the client on job state. I am not offering model names, quotas, or duration claims, because those go stale while the contract stays useful. The maintainability tradeoff is real. You now operate a worker, a broker, and a table. In return you get a place to put timeouts, retries, and permissions that is not the browser. I will take that trade every time the alternative is a spinner negotiating with nginx. Reuse this when you cut the slice: authenticate first, persist the prompt, reserve a budget, return 202 , claim the row, call one provider, and test the duplicate key until the call count stays at one. Then, and only then, let the UI poll. If a layer cannot explain its failure with a status code, it is not a layer yet. Which handoff is least stable in your stack right now, the POST to the queue, the worker to the provider, or the poll back to the UI? Send me a concrete failure state or a response code, not a vibe. If you want a constrained box for the worker URL while you rehearse that path, MonkeyCode's free model access and free server option are sufficient to fail in staging instead of in front of a customer.