{"slug": "the-lease-loop-is-not-a-chat-completion", "title": "The Lease Loop Is Not a Chat Completion", "summary": "A developer argues that distributed lease and leader-election loops should rely on deterministic fencing tokens rather than LLM chat completions, citing latency, non-determinism, and vendor-controlled safety layers as failure modes. The post provides a worked PostgreSQL example with a worker_leases table and a Python renewer that increments an epoch on a deadline and refuses writes when the lease is lost.", "body_md": "A fencing token is not a paragraph. It is a counter that must rise on a clock, die on a deadline, and never ask a model whether the holder still deserves the lock. Free inference has no place in that loop.\n\nTeams keep stuffing chat completions into leader election anyway. A node goes quiet. Logs look messy. Someone wires a cheap model to read the noise and vote on who should keep the lease. The analogy is a lighthouse that phones a critic before each flash. The critic is witty. The reef does not care.\n\nLatency is the first crack. Lease renewal is a budget measured in hundreds of milliseconds, not in token streams. A free path can stall, queue, or return a truncated object when the provider is busy. The lock expires while the prose is still arriving. Split brain is not a literary device. It is two writers sharing one offset.\n\nNon-determinism is the second crack. The same health snapshot can yield keep, drop, or a sentence that fails JSON. Control planes need the same inputs to produce the same fence. A model that hedges is not a quorum. It is a coin with extra faces.\n\nThe third crack is authority. A completion can be filtered, rate limited, or rewritten by a safety layer the caller does not own. Leader election that depends on a vendor's refusal class has already given that vendor a veto over the write path. The veto stays invisible in dashboards that only chart HTTP 200.\n\nA lease is a timed exclusive. The holder proves liveness by raising an epoch before expiry. Downstream storage must reject any write whose epoch is stale. The model of the world is small: holder id, epoch, expiry. Nothing in that tuple is improved by a temperature setting.\n\nPostgreSQL can carry a fencing token without ceremony. The schema and renewer below are a worked example, not a multi-region consensus protocol. Treat them as labeled sample code.\n\n```\nCREATE TABLE worker_leases (\n  name        text PRIMARY KEY,\n  holder      text NOT NULL,\n  epoch       bigint NOT NULL,\n  expires_at  timestamptz NOT NULL\n);\n\nCREATE INDEX ON worker_leases (expires_at);\npython\n# lease_loop.py — keep this module free of HTTP clients and model SDKs\nimport os\nimport socket\nimport time\nimport psycopg\n\nLEASE_NAME = \"order-writer\"\nTTL_SECONDS = 15\nRENEW_EVERY = 5\nHOLDER = f\"{socket.gethostname()}:{os.getpid()}\"\nDSN = os.environ[\"LEASE_DATABASE_URL\"]\n\nACQUIRE = \"\"\"\nINSERT INTO worker_leases (name, holder, epoch, expires_at)\nVALUES (%s, %s, 1, now() + make_interval(secs => %s))\nON CONFLICT (name) DO UPDATE\nSET holder = EXCLUDED.holder,\n    epoch = worker_leases.epoch + 1,\n    expires_at = EXCLUDED.expires_at\nWHERE worker_leases.expires_at < now()\nRETURNING epoch;\n\"\"\"\n\nRENEW = \"\"\"\nUPDATE worker_leases\nSET epoch = epoch + 1,\n    expires_at = now() + make_interval(secs => %s)\nWHERE name = %s\n  AND holder = %s\n  AND expires_at > now()\nRETURNING epoch;\n\"\"\"\n\ndef acquire(conn):\n    with conn.cursor() as cur:\n        cur.execute(ACQUIRE, (LEASE_NAME, HOLDER, TTL_SECONDS))\n        row = cur.fetchone()\n        conn.commit()\n        return None if row is None else int(row[0])\n\ndef renew(conn):\n    with conn.cursor() as cur:\n        cur.execute(RENEW, (TTL_SECONDS, LEASE_NAME, HOLDER))\n        row = cur.fetchone()\n        conn.commit()\n        return None if row is None else int(row[0])\n\ndef hold():\n    with psycopg.connect(DSN) as conn:\n        epoch = acquire(conn)\n        if epoch is None:\n            raise SystemExit(\"lease held elsewhere\")\n        while True:\n            time.sleep(RENEW_EVERY)\n            epoch = renew(conn)\n            if epoch is None:\n                raise SystemExit(\"lost lease; refuse all writes\")\n            print(f\"fence={epoch}\", flush=True)\n\nif __name__ == \"__main__\":\n    hold()\n```\n\nThe control flow is the design. Acquire or renew returns an integer fence, or it returns nothing. There is no branch that asks a model to interpret the row. Writers that use the lease must send that epoch with every mutating call. A storage layer that cannot see the fence is not protected by the loop.\n\n``` python\n# fenced_write.py\ndef append_order(conn, epoch, payload):\n    with conn.cursor() as cur:\n        cur.execute(\n            \"\"\"\n            INSERT INTO orders (payload, fence)\n            SELECT %s, %s\n            WHERE EXISTS (\n              SELECT 1 FROM worker_leases\n              WHERE name = %s\n                AND holder = %s\n                AND epoch = %s\n                AND expires_at > now()\n            )\n            \"\"\",\n            (payload, epoch, LEASE_NAME, HOLDER, epoch),\n        )\n        if cur.rowcount != 1:\n            conn.rollback()\n            raise RuntimeError(\"stale fence; drop the write\")\n        conn.commit()\n```\n\nA free completion that explains why the insert failed is optional theatre after the fact. It must not retry the insert, bump the epoch, or pick a new holder. Commentary can wait. Membership cannot.\n\nOpinion is cheap. Import graphs are not. The next artifact fails CI when the lease package grows an inference client. It is a proposed gate, not a claim that a production fleet already runs it.\n\n``` python\n# test_lease_has_no_inference.py\nfrom pathlib import Path\nimport ast\n\nLEASE_DIR = Path(\"lease\")\nFORBIDDEN_IMPORTS = {\n    \"openai\",\n    \"anthropic\",\n    \"httpx\",\n    \"aiohttp\",\n    \"requests\",\n    \"google\",\n}\nFORBIDDEN_NAME_FRAGMENTS = (\n    \"chat.completions\",\n    \"messages.create\",\n    \"generate_content\",\n)\n\ndef _imported_roots(tree: ast.AST) -> set[str]:\n    roots = set()\n    for node in ast.walk(tree):\n        if isinstance(node, ast.Import):\n            for alias in node.names:\n                roots.add(alias.name.split(\".\")[0])\n        elif isinstance(node, ast.ImportFrom) and node.module:\n            roots.add(node.module.split(\".\")[0])\n    return roots\n\ndef test_lease_modules_stay_offline():\n    files = list(LEASE_DIR.glob(\"*.py\"))\n    assert files, \"lease package missing\"\n    offenders = []\n    for path in files:\n        text = path.read_text(encoding=\"utf-8\")\n        tree = ast.parse(text)\n        bad = _imported_roots(tree) & FORBIDDEN_IMPORTS\n        hits = [frag for frag in FORBIDDEN_NAME_FRAGMENTS if frag in text]\n        if bad or hits:\n            offenders.append(f\"{path}: imports={sorted(bad)} calls={hits}\")\n    assert not offenders, \"inference reached the lease path:\\n\" + \"\\n\".join(offenders)\n```\n\nRun it the boring way.\n\n```\nmkdir -p lease\ncp lease_loop.py fenced_write.py lease/\npython -m pytest test_lease_has_no_inference.py -q\n```\n\nThe test is heuristic. A determined engineer can hide a completion behind a local wrapper named `health.py`. That is why the gate is an exit criterion, not a personality test. If the wrapper appears, the lease package is already on the wrong side of the line. Metrics exporters belong in another process, so the import ban does not become an excuse to smuggle a model through a telemetry helper.\n\nA free model is a useful intern for reading a postmortem. It is a reckless intern for holding the flashlight. The red flag is not \"an LLM exists in the company.\" The red flag is a prompt that can change membership. If the text of a completion can extend a lease, drop a peer, or choose which replica drains a queue, the design has confused explanation with authority.\n\nTimeout arithmetic is the next flag. If the model's p95 exceeds half the TTL, the loop is already insolvent. Padding the TTL to wait for tokens just lengthens the split-brain window. Schema drift in the reply is a third. Control messages that need a JSON repair step do not belong next to a fencing token. A fourth flag is softer and easier to miss: a dashboard that treats \"the model said the node looks sick\" as equivalent to a missed heartbeat.\n\nBetter alternatives are dull on purpose. A leases table plus `make_interval` covers a single primary. PostgreSQL advisory locks cover even smaller footprints. etcd elections and Consul sessions cover a cluster that already speaks consensus. ZooKeeper remains a valid, if heavy, fencing service. None of these tools ask for a system prompt, and none of them improve when the invoice for tokens is zero.\n\nUse the table as a stop/go card during review, not as a personality quiz.\n\n| Observation | Stay on the local lease | Exit the design | \n|---|---|---|\n| Renewer import graph includes an LLM or generic HTTP client | Keep shipping the SQL loop | Fail closed; strip the client | \n| p95 of any completion exceeds half the TTL | Do not add a model to buy time | Remove the model; shorten the path | \n| Failure drill succeeds with inference unreachable | Model is commentary only | Keep commentary off the renewer | \n| Failure drill fails without inference | — | The model was holding the fence | \n| A human cannot state the fence rule in one sentence without mentioning a model | Rewrite the sentence until the model disappears | Stop the rollout | \n\nExit when the renewer imports a network client that is not the database. Exit when a human cannot state the fence rule without mentioning a model. Exit when a failure drill cannot be run with the inference provider unreachable. If the drill cannot pass in that condition, inference was never optional.\n\nScratch generation is a different job. A reviewer may want extra negative fixtures: a truncated JSON body, a 429, a refusal, a slow stream. Those fixtures can be drafted on a disposable box.\n\nDisclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option are a reasonable place to generate those fixtures and to run the import checker against a throwaway clone. They are not a reasonable place to host `lease_loop.py`. The lease either talks to Postgres, or it does not talk.\n\nThe example ignores clock jump, long garbage-collection pauses, and network partitions that leave the SQL session half-open. `now()` inside PostgreSQL is better than the client's wall clock, and it is still not Raft. Teams with a multi-region write path need a real consensus group, not a Python `while True`.\n\nThe CI checker does not prove liveness. It only forbids a class of imports. It will miss a sidecar that renews the lease over a Unix socket to an agent. Treat that sidecar as part of the lease package and apply the same rule. The checker will also nag a legitimate `requests` call if someone parks it in `lease/`; move the call, do not weaken the gate.\n\nFree inference is the wrong owner of adjacent decisions that look like commentary and act like membership. Draining a Kafka partition, choosing which replica is mostly caught up, or rewriting a readiness probe because logs seem unhealthy are fencing problems in costume. The same stop/go card applies.\n\nOperators who already run etcd or ZooKeeper and never call a model from the election path can skim the checker and move on. Researchers studying lock contention on synthetic traces may use a model offline, against recorded data, with no ability to write `epoch`. That study is not a control loop.\n\nThe approach is a poor fit for product demos that exist to show an agent running the cluster. A demo can narrate a lease. It should not hold one. It is also a poor fit for anyone hoping a free completion will replace a quorum because the invoice is zero. The invoice is not the failure domain. Dual writers are.\n\nA lease loop that stays boring will look under-engineered in a screenshot. That is the point. The lighthouse does not workshop the next flash. It flashes, or it goes dark, and the chart records the gap.", "url": "https://wpnews.pro/news/the-lease-loop-is-not-a-chat-completion", "canonical_source": "https://dev.to/aiio_6471/the-lease-loop-is-not-a-chat-completion-2ggd", "published_at": "2026-09-19 09:56:33+00:00", "updated_at": "2026-09-19 10:24:41.100057+00:00", "lang": "en", "topics": ["ai-agents", "ai-infrastructure", "developer-tools", "mlops"], "entities": ["PostgreSQL", "Python", "psycopg"], "alternates": {"html": "https://wpnews.pro/news/the-lease-loop-is-not-a-chat-completion", "markdown": "https://wpnews.pro/news/the-lease-loop-is-not-a-chat-completion.md", "text": "https://wpnews.pro/news/the-lease-loop-is-not-a-chat-completion.txt", "jsonld": "https://wpnews.pro/news/the-lease-loop-is-not-a-chat-completion.jsonld"}}