{"slug": "the-lock-that-never-asks-anyone-dodging-consensus-with-a-model-checker", "title": "The Lock That Never Asks Anyone: Dodging Consensus with a Model Checker", "summary": "A developer built a shared NFS directory lock that avoids consensus entirely by never asking whether a remote process is dead, using a model checker to verify the design instead of implementing Raft, Paxos, or etcd. The lock relies on three assumptions specific to the author's pipeline — no timeouts, heartbeats, clocks, tokens, leaders, or quorums — with the only shared state being the lock record on NFS, and the spec runs in the browser at the Caelum docs. The work matters because the FLP impossibility result means a remote observer cannot distinguish a crashed process from a slow one, so any timeout-based stale-lock scheme eventually admits two writers and silent data corruption.", "body_md": "(AI Generated)\n\nThis is the story of a lock. A boring lock. A lock so boring it doesn’t talk to anybody, doesn’t elect a leader, doesn’t heartbeat, doesn’t read the clock, and still can’t corrupt your data or get stuck. Getting to “boring” took a model checker, three assumptions, and a firm refusal to implement Raft on a Tuesday.\n\nThe spec is live, runs in your browser, and you can break it yourself:\n[NFS Shared Lock in the Caelum docs](https://dhilst.github.io/caelum/real-world/nfs-shared-lock.html).\n\nI have a big pipeline running across several machines. It produces a *lot* of\ndata. Terabytes-don’t-fit-in-your-mental-model a lot. Moving that data between\nmachines is the one thing I really want to avoid.\n\nThe twist: the machine that **produces** a dataset is not the one that\n**validates** it, and validation may write files next to the data it checks.\n\nSo the plan: every machine mounts the same NFS export and uses it as a shared scratch space. Data stays put, and everyone brings their compute to it.\n\nNow two jobs could write into the same directory at the same time. That’s the\nkind of bug that doesn’t crash. It just quietly produces garbage, which you then\nvalidate, publish, and discover three weeks later during a meeting. I want the\nopposite: if a bug makes two jobs collide, one of them should **fail loudly**\n(wait, or time out) instead of silently overwriting the other.\n\nSo: a lock. Lock a directory, write into it, unlock. Anyone may read; writing happens only under the lock. It’s a convention, but I control the pipeline code, so I can enforce and check it.\n\nEasy, right? Let me show you how fast “easy” turns into a PhD thesis.\n\n**Step 1.** “I’ll just put a lock file there.” Great. Job takes lock, job writes,\njob releases lock.\n\n**Step 2.** “What if the job crashes while holding the lock?” The lock file stays\nthere forever. The next run of that job waits forever. The pipeline is now a\nvery expensive way to heat the server room.\n\n**Step 3.** “OK, if the lock is old, someone else can break it.” Old according to\nwhom? After how long? What if the holder is just *slow*: swapping, stuck on I/O,\npaused by a VM migration, or working on an honestly large file? Now two jobs\nbelieve they hold the lock. That’s exactly the silent corruption we built the\nlock to prevent. Congratulations, the cure is the disease.\n\n**Step 4.** “Fine, the holder heartbeats, and we add fencing tokens, and a\nmonotonic counter, and the storage checks the token, and someone has to hand\nout the tokens, and that someone can crash, so we need several of them, and\nthey need to agree…”\n\nAnd there it is. You are now implementing consensus. Paxos is on the whiteboard. Someone says “we could just use etcd” and nobody laughs.\n\nThis isn’t bad luck; it’s a theorem-shaped wall. In an asynchronous system,\n**a remote observer cannot tell a crashed process from a slow one.** That’s the\nheart of the FLP impossibility result, and it’s why every “just break stale\nlocks after a timeout” scheme eventually lets two writers in. Any time you need\na *remote* node to decide “that other node is dead”, you’re buying a failure\ndetector, and good failure detectors are built out of consensus.\n\nThe lesson: **the moment workers need to coordinate, you’re one bad week away\nfrom a consensus problem.** So the trick is not to coordinate better. It’s to\nnot coordinate at all.\n\nHere’s what’s true about *my* pipeline, which is not true about distributed\nsystems in general:\n\nPut those together, and look at who ever needs to ask “is the holder dead?”:\n\nThe impossible question, “is that remote process dead?”, is never asked. Not\nanswered cleverly: *never asked*. No timeouts, no heartbeats, no clocks, no\ntokens, no leader, no quorum. Hosts don’t talk to each other. The only shared\nstate is the lock record on NFS.\n\nThat’s the whole design. It fits on a napkin. But napkins are famously bad at finding race conditions, so let’s not trust the napkin.\n\n[Caelum](https://github.com/dhilst/caelum) is an LTL model checker I built. You\nwrite down states, transitions, and properties; it explores every reachable\nstate and every interleaving, and either proves the properties or hands you a\ncounterexample trace.\n\nThe model is tiny on purpose:\n\n`free`, `waiting`, or `holding`.` none`, `live`, or `stale` (the holder crashed), plus the\n`owner` host written in the record.\nThe heart of the protocol is a single guard:\n\n```\n// acquire → ok (atomic CAS, A1): take a free lock (any host), or recover a\n// stale one — only on the owner's host (A0).\ntransition acquire(p ∈ Proc) {\n  st[p] = waiting ∧\n  (rec = none ∨ (rec = stale ∧ owner = p / 2)) ∧\n  st[p]' = holding ∧\n  rec' = live ∧\n  owner' = p / 2 ∧\n  unchanged(st except p)\n}\n```\n\nRead the guard out loud: *you may take the lock if nobody has it, or if it’s\nstale and it’s yours to judge.* `owner = p / 2` is the whole “never ask a\nremote host” idea, in thirteen characters.\n\nA crash is one transition:\n\n```\n// The holder crashes: its record is left behind, stale.\ntransition crash(p ∈ Proc) {\n  st[p] = holding ∧\n  st[p]' = free ∧\n  rec' = stale ∧\n  unchanged(st except p, owner)\n}\n```\n\nAnd here is what we demand:\n\n```\n// S1. At most one process holds the lock.\nproperty mutual_exclusion {\n  □ (∀ p ∈ Proc: ∀ q ∈ Proc: (p ≠ q ∧ st[p] = holding) → st[q] ≠ holding)\n}\n\n// L1. A crash while holding the lock doesn't deadlock: the crash host\n// eventually takes the lock back.\nproperty crash_recoverable {\n  □ (∀ h ∈ Host: (rec = stale ∧ owner = h) → ◇ (rec = live ∧ owner = h))\n}\n\n// L2. Every waiting process eventually gets the lock.\nproperty work_progresses {\n  □ (∀ p ∈ Proc: st[p] = waiting → ◇ (st[p] = holding))\n}\n```\n\nSafety (“nothing bad happens”): no two holders, ever, across all hosts. Liveness (“something good eventually happens”): a crash never wedges the pipeline, and nobody waits forever.\n\nAll of it passes. **44 states, checked in about 150 ms, in your browser.** Go\npress the button: [the live spec](https://dhilst.github.io/caelum/real-world/nfs-shared-lock.html).\n\nThe interesting bit isn’t the green checkmarks. It’s everything the model forced me to say out loud:\n\n`fairness` block. Weaken A3 and\n`work_progresses` fails, with a trace showing a process losing every race\nforever.`rm -rf`, and I’m fine with that.\nFor fun, I also modelled the “tempting” version, where a remote host may\nrecover a lock it *thinks* is dead (say, “held for too long”). Since it can’t\ntell live from stale, the guess is sometimes wrong. Caelum found the bug in\nfour steps:\n\n```\npid 0 (host 0) acquires the lock\npid 2 (host 1) decides pid 0 \"looks dead\" and takes over   ← pid 0 is alive\n→ both hold the lock: silent data corruption\n```\n\nThat’s Step 3 of the consensus spiral, caught by a machine in milliseconds, not by me at 2 a.m.\n\nOne technical wrinkle. My scratch space is a **re-exported** NFS mount, and the\nkernel refuses file locks on re-exports. From the\n[kernel docs](https://docs.kernel.org/filesystems/nfs/reexport.html):\n*“Clients are not allowed to get file locks or delegations from a reexport\nserver, any attempts will fail with operation not supported.”* So\n`flock`/` fcntl` return `EOPNOTSUPP`.\n\nNo problem. The model never said “use flock”; it said “A1: taking the lock is\natomic”. NFSv4 gives us server-side atomic operations: `mkdir` fails if the name\nexists, and `rename` is atomic. The lock becomes a directory, built from those.\nThe spec didn’t change at all. I just needed a different way to make A1 true.\n\nThat’s the point of separating assumptions from mechanisms: when the platform takes a tool away, you know exactly which property you need to rebuild, and nothing else moves.\n\nHere’s what I did **not** have to write:\n\nWhat I wrote instead is an acquire loop: try to take the lock atomically; if\nit’s stale *and mine to judge*, recover it; otherwise wait. A release. A pid +\nstart time + boot id check. That’s the whole thing.\n\nThis is what lightweight formal verification is actually good for. Not\n(only) for proving that complicated code is correct, but for discovering that\nyou **don’t need the complicated code**. Once you write down which assumptions\nreally hold (same host, exact local death check, atomic create), the\ndesign collapses from “distributed systems problem” to “careful file handling”.\nAnd the model checker tells you that the collapse is sound, instead of your gut.\n\nFormal methods have a reputation for making things heavier. Here it made the code lighter. Forty-four states, one guard, zero consensus.\n\nThe full spec, with the explanation and a Check button, is here:\n**[NFS Shared Lock — Caelum real-world examples](https://dhilst.github.io/caelum/real-world/nfs-shared-lock.html)**.", "url": "https://wpnews.pro/news/the-lock-that-never-asks-anyone-dodging-consensus-with-a-model-checker", "canonical_source": "https://dhilst.github.com/formal-verification/model-checking/caelum/distributed-systems/nfs/2026/09/26/The-Lock-That-Never-Asks-Anyone/", "published_at": "2026-09-26 00:00:00+00:00", "updated_at": "2026-09-26 15:00:55.964341+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-research", "developer-tools"], "entities": ["Caelum", "NFS", "Raft", "Paxos", "etcd", "FLP impossibility result"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/the-lock-that-never-asks-anyone-dodging-consensus-with-a-model-checker", "markdown": "https://wpnews.pro/news/the-lock-that-never-asks-anyone-dodging-consensus-with-a-model-checker.md", "text": "https://wpnews.pro/news/the-lock-that-never-asks-anyone-dodging-consensus-with-a-model-checker.txt", "jsonld": "https://wpnews.pro/news/the-lock-that-never-asks-anyone-dodging-consensus-with-a-model-checker.jsonld"}}