{"slug": "batch-llm-jobs-without-breaking-the-bank-a-queue-first-architecture-for-free", "title": "Batch LLM Jobs Without Breaking the Bank: A Queue-First Architecture for Free Tiers", "summary": "An engineer detailed a queue-first architecture that turns free-tier LLM quotas into batch engines, using a SQLite-backed task queue and Python workers to achieve ten times the throughput of synchronous calls. The approach inverts control flow by writing tasks to a queue instead of waiting per request, avoiding timeout waste and retry feedback loops.", "body_md": "Free model quotas have a hidden enemy: synchronous calls. Every request blocks on the network. Timeouts get wasted. Retries pile up.\n\nCore conclusion: an asynchronous queue turns a free model quota from a demo tool into a batch engine. A SQLite-backed queue, a few Python workers, and a dead-letter table can extract ten times the throughput of synchronous code.\n\nSynchronous calls look simple. Send a request, wait for a response, process the result. The problem is the waiting.\n\nA 30-second timeout means 30 seconds of idle. Ten concurrent requests mean ten threads waiting. Free endpoints get slow, so your threads wait longer. Soon your application becomes a queueing system without the reliability of a queue.\n\nRetries make it worse. A timeout triggers a retry. The retry adds load. The load causes more timeouts. It is a feedback loop.\n\nThe fix is to invert the control flow. Instead of waiting for a response per task, write tasks to a queue and let workers process them.\n\nThe architecture is deliberately simple:\n\nSQLite is a deliberate choice. It is everywhere, zero-config, and handles thousands of tasks without complaint. You do not need Redis for a batch pipeline.\n\n``` python\n# queue.py — SQLite-backed task queue for LLM batch jobs\nimport sqlite3, time, uuid\nfrom pathlib import Path\n\nDB_PATH = Path(\"tasks.db\")\n\ndef init_db():\n    conn = sqlite3.connect(DB_PATH)\n    conn.execute(\"\"\"\n        CREATE TABLE IF NOT EXISTS tasks (\n            id TEXT PRIMARY KEY,\n            prompt TEXT NOT NULL,\n            status TEXT DEFAULT 'pending',\n            attempts INTEGER DEFAULT 0,\n            result TEXT,\n            error TEXT,\n            created_at REAL,\n            updated_at REAL\n        )\n    \"\"\")\n    conn.commit()\n    conn.close()\n\ndef enqueue(prompt):\n    conn = sqlite3.connect(DB_PATH)\n    task_id = str(uuid.uuid4())\n    conn.execute(\n        \"INSERT INTO tasks (id, prompt, status, created_at, updated_at) VALUES (?, ?, 'pending', ?, ?)\",\n        (task_id, prompt, time.time(), time.time())\n    )\n    conn.commit()\n    conn.close()\n    return task_id\n\ndef claim_next():\n    conn = sqlite3.connect(DB_PATH)\n    row = conn.execute(\n        \"SELECT id, prompt FROM tasks WHERE status = 'pending' ORDER BY created_at LIMIT 1\"\n    ).fetchone()\n    if row:\n        conn.execute(\n            \"UPDATE tasks SET status = 'running', updated_at = ? WHERE id = ?\",\n            (time.time(), row[0])\n        )\n        conn.commit()\n    conn.close()\n    return row\n```\n\nThe worker loop:\n\n``` python\n# worker.py — process tasks from the queue\nimport os, time, sqlite3, httpx\nfrom queue import init_db, claim_next\n\ndef complete(task_id, result):\n    conn = sqlite3.connect(\"tasks.db\")\n    conn.execute(\n        \"UPDATE tasks SET status = 'done', result = ?, updated_at = ? WHERE id = ?\",\n        (result, time.time(), task_id)\n    )\n    conn.commit()\n    conn.close()\n\ndef fail(task_id, error):\n    conn = sqlite3.connect(\"tasks.db\")\n    conn.execute(\n        \"UPDATE tasks SET status = 'failed', error = ?, updated_at = ? WHERE id = ?\",\n        (error, time.time(), task_id)\n    )\n    conn.commit()\n    conn.close()\n\ndef process(prompt):\n    url = os.environ[\"LLM_URL\"]\n    key = os.environ[\"LLM_KEY\"]\n    model = os.environ[\"LLM_MODEL\"]\n    with httpx.Client(timeout=60) as client:\n        r = client.post(\n            url,\n            headers={\"Authorization\": f\"Bearer {key}\"},\n            json={\"model\": model,\n                  \"messages\": [{\"role\": \"user\", \"content\": prompt}],\n                  \"max_tokens\": 200},\n        )\n        r.raise_for_status()\n        return r.json()[\"choices\"][0][\"message\"][\"content\"]\n\ndef main():\n    init_db()\n    while True:\n        row = claim_next()\n        if not row:\n            time.sleep(1)\n            continue\n        task_id, prompt = row\n        try:\n            result = process(prompt)\n            complete(task_id, result)\n        except Exception as exc:\n            fail(task_id, type(exc).__name__)\n\nif __name__ == \"__main__\":\n    main()\n```\n\nRun three workers against one database:\n\n```\npython worker.py &\npython worker.py &\npython worker.py &\n```\n\nThree workers, one database. Each worker handles tasks independently. If one crashes, the tasks stay in the database.\n\nFailed tasks need a second chance. Unlimited retries become an infinite loop.\n\nAdd a retry counter. After three attempts, move the task to a dead-letter state.\n\n``` python\ndef main():\n    init_db()\n    while True:\n        row = claim_next()\n        if not row:\n            time.sleep(1)\n            continue\n        task_id, prompt = row\n        try:\n            result = process(prompt)\n            complete(task_id, result)\n        except Exception as exc:\n            conn = sqlite3.connect(\"tasks.db\")\n            conn.execute(\n                \"UPDATE tasks SET attempts = attempts + 1, status = 'pending' WHERE id = ?\",\n                (task_id,)\n            )\n            conn.commit()\n            conn.close()\n```\n\nSimple but effective. A task fails, the counter increments, the task returns to the queue. After three failures, inspect it manually.\n\nThe queue's value is throughput. Synchronous code wastes idle time waiting for responses. Queue code is always working.\n\nLoad the queue with a batch:\n\n``` python\n# enqueue_batch.py — load the queue with test tasks\nfrom queue import init_db, enqueue\n\ninit_db()\nfor i in range(100):\n    enqueue(f\"Write a one-line Python function that returns {i}\")\nprint(\"queued 100 tasks\")\ntime python enqueue_batch.py\ntime python worker.py  # run until the queue is empty\n```\n\nCompare that wall-clock time against a synchronous loop that sends 100 requests and waits for each. The gap is your free quota's real capacity.\n\nMonkeyCode's free server option is a natural home for these workers. It stays online, it is free, and it can reach the same network. Deploy the worker as a background process and let it drain the queue.\n\nThe free model access supplies the tokens for processing. The advertised 10 million token figure is a starting point, not a contract.\n\nDisclosure: This article was prepared as part of MonkeyCode's product outreach.\n\nSQLite is not a high-throughput queue. It performs well at thousands of tasks and degrades at millions. For production-scale workloads, use a real message broker.\n\nThe free server is shared infrastructure. Quotas shift. Latency fluctuates. Your results will differ from mine.\n\nWho should not use this pattern? Teams that need real-time responses. Teams that need transactional guarantees. Teams that need throughput beyond what a free tier can deliver.\n\nWho should use it? Teams processing batch jobs. People building prototypes. Anyone who wants actual work out of a free quota.\n\nSynchronous calls are the enemy of free. Queues are the friend of free. Move work out of waiting, into a database, and let workers chew through it.\n\nThe pattern is not new. It is just rarely applied to LLM calls. Now you have the code. Run it, measure your throughput, and share the numbers.", "url": "https://wpnews.pro/news/batch-llm-jobs-without-breaking-the-bank-a-queue-first-architecture-for-free", "canonical_source": "https://dev.to/gitjs_8094/batch-llm-jobs-without-breaking-the-bank-a-queue-first-architecture-for-free-tiers-j3g", "published_at": "2026-08-22 15:51:09+00:00", "updated_at": "2026-08-22 16:14:04.334906+00:00", "lang": "en", "topics": ["large-language-models", "developer-tools", "mlops"], "entities": ["SQLite", "Python", "httpx"], "alternates": {"html": "https://wpnews.pro/news/batch-llm-jobs-without-breaking-the-bank-a-queue-first-architecture-for-free", "markdown": "https://wpnews.pro/news/batch-llm-jobs-without-breaking-the-bank-a-queue-first-architecture-for-free.md", "text": "https://wpnews.pro/news/batch-llm-jobs-without-breaking-the-bank-a-queue-first-architecture-for-free.txt", "jsonld": "https://wpnews.pro/news/batch-llm-jobs-without-breaking-the-bank-a-queue-first-architecture-for-free.jsonld"}}