{"slug": "crash-proof-batch-llm-processing-a-sqlite-job-queue-on-a-free-server", "title": "Crash-Proof Batch LLM Processing: A SQLite Job Queue on a Free Server", "summary": "A developer built a crash-proof batch LLM processing system using SQLite as a job queue, Python as the worker, and MonkeyCode's free model access and server option. The system handles 10,000 unstructured log lines, converting them to structured JSON, with idempotency keys and atomic claim operations to prevent duplicate processing after crashes.", "body_md": "Three thousand lines in, the process died. Not with a Python traceback — with a silent OOM kill. The input file was untouched. The 3,000 records already sent to the model were unmarked. Re-running meant paying for them twice.\n\nCron does not solve this. A `for`\n\nloop does not solve this. A queue does.\n\nThis article walks through a small log-normalization job — 10,000 unstructured error lines turned into structured JSON — using SQLite as the queue, Python as the worker, and MonkeyCode's free model access plus free server option as the infrastructure.\n\nDisclosure: This article was prepared as part of MonkeyCode's product outreach.\n\nA service writes raw error logs to a file. Each line is unstructured:\n\n```\n2026-08-25 03:12:44 ERROR conn=42 op=checkout msg=\"timeout waiting for lock\" ctx=order:9912\n```\n\nThe goal: turn each line into structured JSON — timestamp, severity, module, error code, and a one-line summary — then store it in a table for querying.\n\n10,000 lines. A few hundred tokens each. A few million tokens total. Well within a free allowance.\n\nThe constraint: the worker runs on a free server. It can be killed at any time. It can be restarted. It must not re-process what already succeeded.\n\nA script is a queue that forgets. If it dies at line 3,000, it has no memory of line 3,000. You can add a \"processed\" marker file, but then you are building a queue anyway — badly.\n\nSQLite gives you the queue for free. No Redis, no RabbitMQ, no extra service to run and monitor. One file, ACID transactions, and a `UNIQUE`\n\nconstraint that does the deduplication for you.\n\nThree fields matter: an idempotency key, a status, and an attempt counter.\n\n```\nCREATE TABLE IF NOT EXISTS jobs (\n    id INTEGER PRIMARY KEY,\n    idempotency_key TEXT UNIQUE,\n    payload TEXT NOT NULL,\n    status TEXT DEFAULT 'pending',\n    attempts INTEGER DEFAULT 0,\n    next_retry_at REAL DEFAULT 0,\n    result TEXT,\n    created_at REAL DEFAULT (unixepoch())\n);\n\nCREATE INDEX IF NOT EXISTS idx_pending\n    ON jobs(status, next_retry_at);\n```\n\nThe idempotency key is a hash of the payload. Inserting the same line twice becomes a no-op:\n\n``` php\ndef enqueue(conn, payload: str) -> None:\n    key = hashlib.sha256(payload.encode()).hexdigest()\n    conn.execute(\n        \"INSERT OR IGNORE INTO jobs (idempotency_key, payload) VALUES (?, ?)\",\n        (key, payload),\n    )\n    conn.commit()\n```\n\nIf the process crashes after 3,000 lines and you re-run the enqueue step, `INSERT OR IGNORE`\n\nskips what is already there. The queue is the memory.\n\nThe worker needs a batch of jobs that are pending, and it needs to mark them as running in the same operation. Otherwise two workers can claim the same row.\n\nSQLite's `UPDATE ... RETURNING`\n\ndoes this in one statement:\n\n``` php\ndef claim_batch(conn, limit: int = 10, now: float | None = None) -> list[tuple[int, str]]:\n    now = now or time.time()\n    rows = conn.execute(\n        \"\"\"\n        UPDATE jobs\n        SET status = 'running', attempts = attempts + 1\n        WHERE id IN (\n            SELECT id FROM jobs\n            WHERE status = 'pending' AND next_retry_at <= ?\n            ORDER BY id\n            LIMIT ?\n        )\n        RETURNING id, payload\n        \"\"\",\n        (now, limit),\n    ).fetchall()\n    conn.commit()\n    return rows\n```\n\nThe `next_retry_at`\n\ncheck is what makes retries possible. A failed job is not `pending`\n\nuntil its backoff window has passed.\n\nThe worker is deliberately boring. Claim a batch, call the model, store the result, repeat.\n\n``` php\ndef worker_loop(conn, batch_size: int = 10) -> None:\n    while True:\n        batch = claim_batch(conn, limit=batch_size)\n        if not batch:\n            time.sleep(5)\n            continue\n\n        for job_id, payload in batch:\n            try:\n                result = process_log_line(payload)\n                complete(conn, job_id, result)\n            except Exception as exc:\n                retry_or_fail(conn, job_id, error=str(exc))\n```\n\nThe model call is isolated in one function. In this project it used MonkeyCode's free model access; the function signature is the only thing the rest of the code depends on.\n\n``` php\ndef process_log_line(line: str) -> dict:\n    # Returns {\"timestamp\": ..., \"severity\": ..., \"module\": ...}\n    # The provider SDK call lives here.\n    ...\n```\n\nA free model endpoint can return 429, 500, or just hang. The queue needs a retry budget.\n\n``` php\ndef retry_or_fail(conn, job_id: int, error: str, max_attempts: int = 5) -> None:\n    row = conn.execute(\n        \"SELECT attempts FROM jobs WHERE id = ?\", (job_id,)\n    ).fetchone()\n\n    if row[\"attempts\"] >= max_attempts:\n        conn.execute(\n            \"UPDATE jobs SET status = 'failed', result = ? WHERE id = ?\",\n            (error, job_id),\n        )\n    else:\n        backoff = (2 ** row[\"attempts\"]) + random.uniform(0, 1)\n        conn.execute(\n            \"\"\"\n            UPDATE jobs\n            SET status = 'pending', next_retry_at = ?\n            WHERE id = ?\n            \"\"\",\n            (time.time() + backoff, job_id),\n        )\n    conn.commit()\n```\n\nExponential backoff with jitter. After five attempts, the job is marked `failed`\n\nand the error is stored. You can inspect it later instead of guessing.\n\nA test run with 10,000 lines finished in about 40 minutes on the free server. The numbers that mattered:\n\n`failed`\n\nstatus.These are numbers from a single test run, not a benchmark. Your token usage and failure rates will differ.\n\nThe free allowance covered the full run with room to spare. A few million tokens for 10,000 structured records is a reasonable trade — if the queue prevents you from paying for the same record twice.\n\nQuerying the results is plain SQL:\n\n```\nSELECT status, count(*) FROM jobs GROUP BY status;\n\nSELECT json_extract(result, '$.severity') AS severity, count(*)\nFROM jobs WHERE status = 'done' GROUP BY severity;\n```\n\nSQLite is not Postgres. Concurrent writers will hit `database is locked`\n\n; the fix is WAL mode and a single writer process.\n\n```\nconn.execute(\"PRAGMA journal_mode=WAL;\")\nconn.execute(\"PRAGMA busy_timeout = 5000;\")\n```\n\nThe free server has memory limits. A batch size of 10 keeps the working set small. Do not load 10,000 lines into memory and expect the OOM killer to be kind.\n\nFree tiers change. The token allowance and server specs in this article are what the project used; check the current limits before you build on top of them.\n\nWho should skip this? Anyone processing more than a few hundred thousand records, anyone with strict latency requirements, and anyone storing sensitive data on a free server. This is a small-job queue, and it is honest about that.\n\nThe first lesson: the queue is the product. The model call is the easy part. The hard part is knowing what has been done, what has failed, and what is worth retrying.\n\nThe second lesson: idempotency keys are cheaper than apologies. One hash per line saved thousands of duplicate model calls.\n\nThe third lesson: a free server is enough when the job is sized to it. 10,000 lines, a few million tokens, one SQLite file. No Kubernetes, no bill, no drama.\n\nIf you have a batch job that keeps dying halfway, start with a queue, not a bigger server. SQLite will hold up longer than you expect.", "url": "https://wpnews.pro/news/crash-proof-batch-llm-processing-a-sqlite-job-queue-on-a-free-server", "canonical_source": "https://dev.to/byteio_501/crash-proof-batch-llm-processing-a-sqlite-job-queue-on-a-free-server-2ijg", "published_at": "2026-08-24 19:03:04+00:00", "updated_at": "2026-08-24 19:14:20.835446+00:00", "lang": "en", "topics": ["large-language-models", "developer-tools"], "entities": ["SQLite", "Python", "MonkeyCode"], "alternates": {"html": "https://wpnews.pro/news/crash-proof-batch-llm-processing-a-sqlite-job-queue-on-a-free-server", "markdown": "https://wpnews.pro/news/crash-proof-batch-llm-processing-a-sqlite-job-queue-on-a-free-server.md", "text": "https://wpnews.pro/news/crash-proof-batch-llm-processing-a-sqlite-job-queue-on-a-free-server.txt", "jsonld": "https://wpnews.pro/news/crash-proof-batch-llm-processing-a-sqlite-job-queue-on-a-free-server.jsonld"}}