{"slug": "polygraph-now-your-tests-can-t-lie", "title": "Polygraph – Now your tests can't lie", "summary": "Polygraph, a new Claude Code plugin and standalone CLI, uses LLMs to automatically generate formal models of stateful JavaScript code and exhaustively check for bugs, finding violations like double charges that unit tests miss. In a production subscription-billing system, it independently corroborated a genuine double-charge bug and found all 5 seeded bugs in a controlled eval where replay alone found none. The tool aims to make formal verification practical by having an LLM produce and update the model on every code change, keeping developers in JavaScript.", "body_md": "**Your tests check the paths you thought of. Polygraph checks the ones you didn't.**\n\nPolygraph is a Claude Code plugin (and standalone CLI) that finds bugs in\n**stateful code** — workflows, reducers, protocol handlers, checkout flows,\nsession managers — by exhaustively exploring every state the code can reach\nand flagging the ones that break rules you care about, like *\"a customer is\nnever charged twice.\"*\n\nYou don't need to know anything about formal verification to use it. You write the rules as plain JavaScript predicates. The heavy lifting — deriving a formal model of your code, exploring the state space, producing a shortest path to each violation — is done for you.\n\nDisclosure — read this first.Polygraph isexperimental, not peer-reviewed, unproven technology.It is aconsistency check, not a proof: a clean run means the code's observable behavior matches an independent reading of its own source, and nothing more. Every finding is alead to investigate by hand, not an established result. Do not rely on it as your only safeguard for correctness- or safety-critical code.\n\nUnit tests execute the scenarios you wrote. A state machine of even modest size has orders of magnitude more reachable states than any test suite visits — and the bugs that hurt in production live in the combinations nobody wrote a test for: a retry landing after a cancel, a timeout racing a confirmation, an event arriving in a state where nobody expected it.\n\nPolygraph attacks that gap in two ways:\n\n-\n**Audit existing code**(`/polygraph:verify`\n\n). An LLM reads your source and writes an independent, executable specification of it — a second opinion on what the code does. Polygraph then (1) replays real execution traces against that spec to confirm it's faithful, and (2)**model-checks** it: starting from the initial state, it tries every action with every relevant payload, visits every reachable state, and reports any state that violates one of your rules — with the shortest sequence of actions that gets there. That counterexample path is a ready-made repro for the bug. -\n**Author new code**(`polygen`\n\n). Give it a one-sentence feature description and it writes the state machine*and*its rules, model-checks its own output, repairs violations, and hands you code with a passing exhaustive check plus a generated regression trace corpus — verified from the moment it's written.\n\nReal result: on a production SaaS subscription-billing machine, this method\nindependently corroborated a genuine double-charge bug\n([ examples/case-study-subscription.md](/jdubray/polygraph/blob/main/examples/case-study-subscription.md)).\nIn the controlled eval, replay alone found 0/5 seeded bugs — model checking\nfound 5/5, with counterexamples\n(\n\n[).](/jdubray/polygraph/blob/main/eval/FINDING-faithful-reproduction.md)\n\n`eval/FINDING-faithful-reproduction.md`\n\nThose tools work — but you have to hand-translate your code into their\nlanguage and keep the translation current, which is why almost nobody does\nit. Specifying even a small-to-midsize system is a multi-month effort, and\nevery code change invalidates the spec. Polygraph's bet is that an LLM can\nproduce that formal model *from your source* cheaply enough to rerun on every\nchange, and that replaying real traces against the model tells you whether to\ntrust it. You stay in JavaScript the whole time. (If you *do* want the real\nthing, an optional `--tla`\n\nflag mechanically transpiles the winning spec to\nTLA+ and runs TLC over it.)\n\nThis bet is no longer a fringe position. Emilie Ma's Berlin Buzzwords talk\n[ Scaling formal methods with LLMs](https://www.youtube.com/watch?v=M2wb2ug2gYg)\npresents the same thesis from the TLA+ side: SysMoBench, a benchmark showing\nfrontier LLMs can't yet write conformant specs unaided, and Specula, an\nagentic pipeline (spec generation + model checking + trace validation — the\nsame three legs Polygraph stands on) that found 160+ bugs in production\nsystems like MongoDB and etcd-raft, over 130 previously unknown, at roughly\n$40 per repository. The convergent finding: the LLM alone is not trustworthy,\nbut an LLM\n\n*held accountable by a model checker and trace validation*is a practical bug-finder.\n\nEverything revolves around three artifacts you can read and diff:\n\n-\n**A contract**(`contract.json`\n\n) — what's observable: the state fields that matter, the actions the machine accepts, the data each action can carry, which states are terminal. This is the scope of the audit. -\n**A spec**— an executable model of your code, written by an LLM from your source. By default it's a strict, self-describing state-machine module (the[SAM pattern](https://sam.js.org)v2 strict profile): every action it ignores must say*why*(`reject(reason)`\n\n), it can't hide bookkeeping state, and it declares its own action/data domains — so the model checker knows exactly what to explore with zero configuration. Several specs are generated independently and vote, so one bad generation doesn't decide the outcome. -\n**Invariants**(`invariants.mjs`\n\n) — your rules, as plain JS functions over a state:*\"never charged without a confirmed transaction\"*,*\"an expired session can't accept input.\"*These encode your**intent**, which is the one thing no tool can derive from the code — code with a bug is a faithful description of the wrong behavior.\n\nThen two checks run:\n\n**Check 1 — replay (is the spec faithful?).** Real execution traces —\ncaptured by wrapping your dispatch/reducer once, so every step logs a\n`{pre, action, data, post}`\n\nwindow — are replayed against each spec. Before\nany generated spec is trusted, controls run: a hand-written reference spec\nmust score 100% and a deliberately mutated one must fail, proving the harness\ncan actually tell good from bad. Disagreements are triaged into\n**spec-errors** (the LLM misread the code), **code-findings** (the code\ndisagrees with an independent reading of itself — investigate), and\n**contract-errors** (the contract mis-scoped the problem).\n\n**Check 2 — model check (where the bugs are).** The faithful spec is iterated\nexhaustively from its initial state against your invariants. This is the step\nthat finds what tests miss: replay can only flag a bug when spec and code\n*disagree*, and a faithful spec reproduces the bug right along with the code.\nModel checking reaches the bad state anyway and prints the shortest path to\nit. Every check also runs a determinism double-pass for free.\n\n**polygen** runs the same machinery in reverse: draft contract → author\nmodule → propose invariants → model-check → self-repair on violations (capped\nat `--repair-max`\n\nrounds; a non-converging run is reported as NOT converged,\nnever silently presented as clean) → synthesize and independently replay a\ntrace corpus. It also cross-checks that contract and code agree on their\naction/data vocabulary, because the two come from independent model calls —\n[ examples/case-study-polygen-domain-gap.md](/jdubray/polygraph/blob/main/examples/case-study-polygen-domain-gap.md)\nshows a real run where a silent enum-spelling mismatch collapsed the\nexplorable state space, and how the check caught it.\n\npolygen output is JS/TS only: the generated module is directly usable in a JS/TS codebase; porting a verified model to another language would need its own differential check and is out of scope.\n\n| step | who does it | how long |\n|---|---|---|\n| Define the contract (observable fields, actions, terminals) | you, or Claude drafts it for your review | minutes |\n| Capture traces (instrument a copy, drive scenarios) | the agent, in Claude Code — it builds test doubles/emulators if needed; standalone, you wrap your dispatch once (snippet below) |\nthe bulk of the work standalone; delegated in Claude Code |\n| Write invariants (your rules as JS predicates) | you — this is your intent; the tool can propose, only you can confirm | minutes per rule |\n| Generate specs, replay, model-check | automatic | minutes |\n| Triage findings | you — every finding is a lead, not a verdict | depends on what it finds |\n\nTrace capture is historically what made this kind of verification expensive, and it's the step the agent now carries: in the origin study, a Claude agent built a payment-terminal emulator and a fault-injection proxy, instrumented a production payment workflow, drove 17 scenarios, and produced a 75-window corpus autonomously. What stays with you is judgment: confirming the contract covers the right state, and sanity-checking any doubles the agent built against reality.\n\nOne hard prerequisite: **the code must be runnable in isolation**, because\ntraces are ground truth captured from the code actually executing. If it has\na clean step boundary (a dispatch, reducer, or handler), you're set. If it\nonly runs against a DB/network/device, stand up doubles first (or let the\nagent do it). If it can't run at all, replay degenerates to checking specs\nagainst your *expectations*, which can't find code bugs.\n\nWrapping the boundary standalone is one line per scenario:\n\n``` js\nimport { withTracing } from '<plugin>/scripts/instrument/trace-emitter.mjs';\n\n// project ONLY the contract's observable keys:\nconst project = () => ({ txState: m.txState, orderId: m.orderId });\nconst dispatch = withTracing(rawDispatch, project, 'traces/s1_normal.ndjson');\n// Redux-style reducer: tapReducer(...)  ·  SAM component: withSamTracing(...) — see scripts/instrument/\n```\n\nOnly spec **generation** and code **authoring** call the Anthropic API.\nEverything that checks, replays, or explores runs locally on Node ≥ 20.\n\n| task | command | API key? | why |\n|---|---|---|---|\n| Try the quickstart / run the test suite | `npm test` |\nno |\nreplays bundled specs against bundled traces |\n| Validate a trace corpus | `validate_corpus.mjs` |\nno |\nlocal schema/shape checks |\n| Replay saved specs against traces | `verify.mjs --specs …` |\nno |\npure local execution |\n| Model-check a spec against invariants | `check.mjs` |\nno |\nexhaustive local exploration |\n| Escalate to TLA+/TLC | `verify.mjs --tla` |\nno |\nmechanical transpile + local TLC (needs Java + `tla2tools.jar` via `POLYGRAPH_JAVA` / `POLYGRAPH_TLA_JAR` ) |\nGenerate specs from source |\n`verify.mjs --source … --model …` |\nyes (`ANTHROPIC_API_KEY` ) |\nthe LLM writes the specs |\nAuthor new code (polygen) |\n`polygen.mjs --intent … --model …` |\nyes (`ANTHROPIC_API_KEY` ) |\nthe LLM drafts contract, code, and invariants |\n\nThis applies inside Claude Code too: the skills and subagents shell out to\nthese same scripts, so the generate and polygen steps need\n`ANTHROPIC_API_KEY`\n\nset in your environment — your Claude Code session\ncredentials are not used for them.\n\nAs a Claude Code plugin (this repo is its own marketplace):\n\n```\n/plugin marketplace add jdubray/polygraph\n/plugin install polygraph@polygraph\n```\n\nOr clone directly: `git clone https://github.com/jdubray/polygraph ~/.claude/plugins/polygraph`\n\n.\nUpdate later with `/plugin marketplace update polygraph`\n\n. Requires **Node ≥ 20**.\n\nThen just ask in plain language — *\"verify this state machine\"*, *\"does this\ncode do what I think it does?\"*, *\"write a verifiable checkout flow\"* — or use\nthe entry points directly:\n\n| you type | what it is | when to use it |\n|---|---|---|\n`/polygraph:polygraph` |\naudit skill |\nguided end-to-end audit — Claude designs the contract, captures traces, runs controls, triages with you |\n`/polygraph:verify` |\naudit command |\nyou already have contract + traces; just run generate + replay |\n`polygraph-verifier` |\naudit subagent |\nhand off the whole audit for an autonomous run |\n`/polygraph:polygen` |\nauthor skill / command |\nwrite a NEW verified state machine from a feature description |\n`polygen` |\nauthor subagent |\nhand off the whole authoring loop |\n\n```\n# replay saved specs (no API key)\nnode scripts/verify.mjs --contract contract.json --traces traces/ --specs specs/ --out out/\n\n# generate + replay (needs ANTHROPIC_API_KEY)\nnode scripts/verify.mjs --contract contract.json --source src/machine.ts \\\n  --traces traces/ --model sonnet-5 --n 5 --out out/\n\n# validate a corpus (no API key)\nnode scripts/validate_corpus.mjs contract.json traces/\n\n# escalate the winning spec to TLC (no API key; needs Java toolchain)\nPOLYGRAPH_JAVA=/path/to/java POLYGRAPH_TLA_JAR=/path/to/tla2tools.jar \\\n  node scripts/verify.mjs --contract contract.json --traces traces/ --specs specs/ --tla --out out/\n\n# author NEW verifiable code (needs ANTHROPIC_API_KEY)\nnode scripts/polygen.mjs --intent \"<feature description>\" --model sonnet-5 --out out/\n```\n\npolygen writes everything to `<out>/`\n\n: `contract.json`\n\n, `next.cjs`\n\n(the\nmodule), `invariants.mjs`\n\n, `traces/*.ndjson`\n\n, and `polygen-report.md`\n\n. The\nhandoff after a polygen run is deliberately manual: review the contract and\ninvariants (they're the model's reading of your intent, not ground truth),\nwire the module into your real handler — call it, don't reimplement it — then\nrun `/polygraph:verify`\n\nagainst traces captured from the *integrated* code.\nThat last step catches drift between the pure model and the glue around it.\n\nThere is **no default model** — pass `--model`\n\n. Recommended:\n\n| alias | resolves to | notes |\n|---|---|---|\n`sonnet-5` |\n`claude-sonnet-5` |\nbalanced choice (speed / intelligence) |\n`fable-5` |\n`claude-fable-5` |\nstrongest in the origin study |\n\nAnything not in the alias table (`scripts/models.mjs`\n\n) is passed to the API\nverbatim, so an exact Anthropic model id always works. Reasoning models spend\noutput tokens on thinking before the answer; if you lower `--max-tokens`\n\nfrom\nthe default 32000 and see empty specs (`stop_reason: max_tokens`\n\n), raise it\nback.\n\n**Quickstart — the turnstile**(`examples/turnstile-v2/`\n\n, no API key):`npm test`\n\nvalidates the corpus and runs the positive/negative controls;`npm run verify:turnstile-v2`\n\nreplays the bundled specs.**Production case study**(`examples/case-study-subscription.md`\n\n): a real end-to-end run on a closed-source SaaS billing machine that corroborated a genuine double-charge bug — plus an honest look at the risks the method*can't*see (external-service boundaries).**polygen — OTP flow**(`examples/polygen-otp/`\n\n): authored from one sentence; 8 states, 0 violations, a 134-window synthesized corpus, 0 independent-replay failures.**polygen — cart checkout, and a bug polygen found in itself**(`examples/polygen-cart-checkout/`\n\n, narrated in`examples/case-study-polygen-domain-gap.md`\n\n): the contract/code vocabulary mismatch that motivated the domain cross-check, including a false positive the fix introduces.**TLC tier reference**(`examples/etcd-raft-v2/`\n\n): a v2 spec + invariants exercising the TLA+ escalation.\n\n```\n.claude-plugin/plugin.json   plugin manifest\nskills/polygraph/            the audit method, as instructions Claude follows\nskills/polygen/              the author method, as instructions Claude follows\ncommands/                    /polygraph:verify and /polygraph:polygen\nagents/                      polygraph-verifier and polygen subagents\nscripts/                     sam-tv.mjs (replayer), check.mjs (model checker),\n                             to-tla.mjs + tla-check.mjs (TLC tier), verify,\n                             generate, polygen, validate_corpus, models,\n                             vendor/sam-pattern.cjs, instrument/*\ntemplates/                   contract.schema.json + contract.example.json\nexamples/                    worked examples and case studies (see above)\neval/                        the seeded-bug A/B eval and findings\ntest/                        npm test — no API key needed\n```\n\nThe method is introduced in:\n\nJean-Jacques Dubray.\n\nCan Code Specify a System Precisely Enough to Formally Verify It?arXiv:2607.05076, July 2026.[https://arxiv.org/abs/2607.05076]\n\nThe 2.0 release gate: a seeded-bug A/B eval passing at parity or better at two\nmodel tiers, with two bugs newly caught at the cheap replay tier and zero dead\nspecs. Reference implementation and full case study:\n[https://github.com/jdubray/SysMoBench-1](https://github.com/jdubray/SysMoBench-1).\n\nRelated work: Emilie Ma,\n[ Scaling formal methods with LLMs](https://www.youtube.com/watch?v=M2wb2ug2gYg)\n(Berlin Buzzwords) — SysMoBench (benchmarking LLM spec generation on\nproduction systems) and Specula (an agentic TLA+ spec-generation and\nbug-discovery pipeline), reaching the same conclusion from the formal-methods\nside: model checking plus trace validation is what makes LLM-generated specs\ntrustworthy.\n\nIf you use Polygraph in your work, please cite the paper (see `CITATION.cff`\n\n).\n\n## Legacy 1.x artifact (`--legacy-bare-next`\n\n)\n\nPolygraph 1.x derived a bare `next(state, action, data)`\n\nfunction instead of\nthe strict SAM v2 module. That pipeline remains available end-to-end behind\n`--legacy-bare-next`\n\nfor one release (`npm run test:legacy`\n\n,\n`npm run verify:turnstile`\n\n). The v2 strict profile replaced it because it\ncloses whole failure classes by construction — silent no-op specs, hidden\nbookkeeping state, vacuous exploration — while the N-spec voting layer absorbs\nwhat v2 gives up. The one discipline sentence the study showed carried\nbare-next's replay robustness is kept verbatim in the v2 prompts. (The repo's\nhistorical disk name, `bare-next-verify`\n\n, records this lineage.)\n\nApache-2.0 — see `LICENSE`\n\n.", "url": "https://wpnews.pro/news/polygraph-now-your-tests-can-t-lie", "canonical_source": "https://github.com/jdubray/polygraph", "published_at": "2026-07-15 20:46:06+00:00", "updated_at": "2026-07-15 20:58:43.158438+00:00", "lang": "en", "topics": ["developer-tools", "ai-tools", "large-language-models", "ai-agents"], "entities": ["Polygraph", "Claude Code", "Emilie Ma", "SysMoBench", "Specula", "MongoDB", "etcd-raft", "TLA+"], "alternates": {"html": "https://wpnews.pro/news/polygraph-now-your-tests-can-t-lie", "markdown": "https://wpnews.pro/news/polygraph-now-your-tests-can-t-lie.md", "text": "https://wpnews.pro/news/polygraph-now-your-tests-can-t-lie.txt", "jsonld": "https://wpnews.pro/news/polygraph-now-your-tests-can-t-lie.jsonld"}}