{"slug": "show-hn-linejudge-an-independent-verification-harness-for-coding-agents", "title": "Show HN: Linejudge – an independent verification harness for coding agents", "summary": "Linejudge, an independent verification harness for coding agents, found that 8 out of 8 agent-authored patches passed machine checks but only 6 out of 8 actually fixed the target issues in sqlite-utils, with two patches failing to address the bugs. The harness, built by Phillip Mex, runs agents on goals and independently verifies results using real commands, diffs, and exit codes, never trusting the agent's own success claims.", "body_md": "**The independent line judge for coding agents. Never trust the player's call.**\n\nCoding agents routinely claim success. linejudge runs your agent on a goal, then\n**independently verifies the result** — real commands, real diffs, real exit\ncodes — and renders its own verdict. The agent's opinion of its own work is\nnever consulted.\n\nEight agent-authored patches for eight real [sqlite-utils](https://github.com/simonw/sqlite-utils)\nissues, put through every gate the harness has:\n\n| evidence | result |\n|---|---|\n| Runs succeeded | 7/8 |\n| Independently verified pass — judged by verifiers, not by the claim | 8/8 |\n| Regression test proven — fails with the fix reverted | 8/8 |\n| Diff reviewed against the issue it claims to close | 6/8 |\n\nThree machine gates, 8/8 on every one. Reading the diffs found two that don't fix the bug:\n\n**#439**— the patch counts bytes as`utf-8-sig`\n\n. Fixes utf-16, regresses the default: a 708-byte plain-utf-8 CSV reports 1011. The added test covers utf-16-le only.**#762**— the constraint parser masks string literals but not SQL comments, so a commented-out`-- CHECK (id > 0)`\n\nis re-emitted as an*active*constraint. A table that accepted`id = -1`\n\nstarts rejecting it.\n\nNeither is reachable by a test suite that doesn't already know about the bug.\n**Machine verification is necessary and it is not sufficient** — so the\nscoreboard reports those levels separately instead of as one number.\n\nFull evidence trail: [PROOF.md](/phillipmex/linejudge/blob/main/PROOF.md). The long version, with both rejects\nwalked through in detail:\n[Eight patches passed every check I had. Two of them were wrong.](https://phillipmex.github.io/linejudge/)\n\n``` bash\n$ python proofs/demo.py --root demo        # 3 tasks, one agent LIES\n20260721T…-issue-101-config-loader…: SUCCESS\n20260721T…-issue-102-add-json-flag…: FAILED     ← claimed success, wrote nothing\n20260721T…-issue-103-docs-quickstart…: SUCCESS\n```\n\nThe failed run's agent reported `## Status: SUCCESS`\n\n. The `files_exist`\n\nverifier checked the filesystem and disagreed. **The verdict, not the claim,\ndecides the run status.** No tokens spent — see the full\n[sample PROOF.md](/phillipmex/linejudge/blob/main/docs/PROOF-sample.md).\n\nEvery agent loop on the market grades its own homework: the model says \"done\" and the harness believes it. linejudge splits the roles. The agent plays the point; the harness calls the lines:\n\n**Independent verification**— a declarative verifier spec (`command`\n\n,`files_exist`\n\n,`diff_constraints`\n\n,`http_check`\n\n) executed by the harness, outside the agent session.[Full spec](/phillipmex/linejudge/blob/main/docs/verifier-spec.md).**Blast-radius guarding**— read-only directories are snapshotted (`git status`\n\n) before and after every run; any unexpected mutation fails the run with diagnostics.**Verified-diff-only writes**— write access goes through a git worktree on an unmerged`linejudge/<run_id>`\n\nbranch. The terminal state is a reviewable, verified diff — never a silent merge into your working tree.**Auditable cross-run learning**— a second, tool-less agent call distills each run into a versioned markdown lesson; future runs retrieve the most relevant lessons by tag. No opaque memory, and a poisoning guard keeps rate-limit garbage out of the pool.**Cost accounting**— per-run token/dollar cost captured from the agent's own telemetry into`run_cost.json`\n\nand an append-only`runs/ledger.jsonl`\n\n.**Local review dashboard**— every run's full evidence trail (prompt, claim, diff, verdict, cost, lesson) with an approve/reject gate. The decision is written*beside*the evidence, never into it.**Zero dependencies**— Python stdlib only, every source file ≤300 lines. You can read the whole engine in an afternoon.\n\nWorks with Claude Code headless (`claude -p`\n\n) today; the\n[adapter interface](/phillipmex/linejudge/blob/main/docs/adapter-guide.md) is agent-agnostic by design —\n`run(prompt, cwd, timeout, …) -> RunResult`\n\nis the entire contract.\n\n```\ngoal.md ─────► runner ─────► adapter ──────► agent (claude -p / mock / yours)\n (task +          │                             │\n  verifiers)      │◄───── REPORT.md ────────────┘   ◄── the CLAIM\n                  │\n                  ├── guard: read_dirs snapshot before/after (trip ⇒ FAILED)\n                  ├── write flow: git worktree ─► write_diff.patch ─► branch\n                  ├── verifiers ─────────► verdict.json             ◄── the CALL\n                  ├── distill (2nd call) ─► learnings/<id>.md\n                  └── ledger ────────────► run_cost.json + runs/ledger.jsonl\n                                                │\n                              linejudge dashboard (review + approve/reject)\n```\n\nNeeds Python 3.10+. Nothing else — no API key required for the mock pipeline.\n\n```\npip install linejudge\n\n# example goals + the mock demo live in the repo\ngit clone https://github.com/phillipmex/linejudge && cd linejudge\n\n# 1. See exactly what the harness would send — zero spend\nlinejudge run goals/examples/hello.md --dry-run\n\n# 2. Full pipeline on a scripted mock agent (one run lies and gets caught)\npython proofs/demo.py --root demo\n\n# 3. Review the evidence\nlinejudge dashboard --root demo        # http://127.0.0.1:8765\n```\n\nTo run against a real agent, install [Claude Code](https://docs.anthropic.com/en/docs/claude-code),\nset `ANTHROPIC_API_KEY`\n\n(directly or in a gitignored `.env.local`\n\n), and drop the\n`--dry-run`\n\n:\n\n```\nlinejudge run goals/examples/hello.md --root .\n```\n\nStep-by-step walkthrough with a write-mode goal: [docs/quickstart.md](/phillipmex/linejudge/blob/main/docs/quickstart.md).\n\nA goal is one markdown file: a fenced header (deliberately *not* YAML — no\nnesting, no quoting rules, nothing to mis-parse) plus the prompt body.\n\n```\n---\nname: fix-config-crash\ntags:\n  - widget\nread_dirs:\n  - /path/to/reference-repo      # guarded read-only\nwrite_repo: /path/to/widget      # optional: write cycle via worktree\nverifiers:\n  - command: python -m pytest -q\n  - files_exist: done.txt\n  - diff_constraints: max_files=5 deny=**/*.env\ntimeout_secs: 1800\n---\nFix the config loader crash on empty YAML files. …\n```\n\n|\n\n[Verifier spec](/phillipmex/linejudge/blob/main/docs/verifier-spec.md)[Adapter guide](/phillipmex/linejudge/blob/main/docs/adapter-guide.md)[Comparison](/phillipmex/linejudge/blob/main/docs/comparison.md)`claude -p`\n\nloops, Aider, OpenHands, hosted agents[Governance templates](/phillipmex/linejudge/blob/main/docs/governance-templates.md)[ADR-0001](/phillipmex/linejudge/blob/main/docs/adr/ADR-0001-lineage-and-open-core.md)[PROOF.md](/phillipmex/linejudge/blob/main/PROOF.md)[Sample PROOF.md](/phillipmex/linejudge/blob/main/docs/PROOF-sample.md)`proofs/`\n\nturns real GitHub issues into goals and renders an honest scoreboard:\n\n```\npython proofs/generate.py --repo owner/name --limit 5 --verifier \"command: pytest -q\"\nlinejudge run proofs/goals/issue-….md --root proofs/root      # per goal\npython proofs/stats.py --root proofs/root                     # → PROOF.md\n```\n\nPROOF.md reports **runs succeeded** and **independently verified pass %** as\nseparate numbers, because they are separate facts.\n\nv0.2.0. 104 tests, CI on Windows + Ubuntu × Python 3.10/3.12, zero runtime dependencies.\n\nApache-2.0", "url": "https://wpnews.pro/news/show-hn-linejudge-an-independent-verification-harness-for-coding-agents", "canonical_source": "https://github.com/phillipmex/linejudge", "published_at": "2026-08-11 01:09:22+00:00", "updated_at": "2026-08-11 01:41:23.995863+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "ai-safety"], "entities": ["Linejudge", "Phillip Mex", "sqlite-utils"], "alternates": {"html": "https://wpnews.pro/news/show-hn-linejudge-an-independent-verification-harness-for-coding-agents", "markdown": "https://wpnews.pro/news/show-hn-linejudge-an-independent-verification-harness-for-coding-agents.md", "text": "https://wpnews.pro/news/show-hn-linejudge-an-independent-verification-harness-for-coding-agents.txt", "jsonld": "https://wpnews.pro/news/show-hn-linejudge-an-independent-verification-harness-for-coding-agents.jsonld"}}