The independent line judge for coding agents. Never trust the player's call.
Coding agents routinely claim success. linejudge runs your agent on a goal, then independently verifies the result — real commands, real diffs, real exit codes — and renders its own verdict. The agent's opinion of its own work is never consulted.
Eight agent-authored patches for eight real sqlite-utils issues, put through every gate the harness has:
| evidence | result |
|---|---|
| Runs succeeded | 7/8 |
| Independently verified pass — judged by verifiers, not by the claim | 8/8 |
| Regression test proven — fails with the fix reverted | 8/8 |
| Diff reviewed against the issue it claims to close | 6/8 |
Three machine gates, 8/8 on every one. Reading the diffs found two that don't fix the bug:
#439— the patch counts bytes asutf-8-sig
. Fixes utf-16, regresses the default: a 708-byte plain-utf-8 CSV reports 1011. The added test covers utf-16-le only.#762— the constraint parser masks string literals but not SQL comments, so a commented-out-- CHECK (id > 0)
is re-emitted as anactiveconstraint. A table that acceptedid = -1
starts rejecting it.
Neither is reachable by a test suite that doesn't already know about the bug. Machine verification is necessary and it is not sufficient — so the scoreboard reports those levels separately instead of as one number.
Full evidence trail: PROOF.md. The long version, with both rejects walked through in detail: Eight patches passed every check I had. Two of them were wrong.
$ python proofs/demo.py --root demo # 3 tasks, one agent LIES
20260721T…-issue-101-config-…: SUCCESS
20260721T…-issue-102-add-json-flag…: FAILED ← claimed success, wrote nothing
20260721T…-issue-103-docs-quickstart…: SUCCESS
The failed run's agent reported ## Status: SUCCESS
. The files_exist
verifier checked the filesystem and disagreed. The verdict, not the claim, decides the run status. No tokens spent — see the full sample PROOF.md.
Every agent loop on the market grades its own homework: the model says "done" and the harness believes it. linejudge splits the roles. The agent plays the point; the harness calls the lines:
Independent verification— a declarative verifier spec (command
,files_exist
,diff_constraints
,http_check
) executed by the harness, outside the agent session.Full spec.Blast-radius guarding— read-only directories are snapshotted (git status
) before and after every run; any unexpected mutation fails the run with diagnostics.Verified-diff-only writes— write access goes through a git worktree on an unmergedlinejudge/<run_id>
branch. The terminal state is a reviewable, verified diff — never a silent merge into your working tree.Auditable cross-run learning— a second, tool-less agent call distills each run into a versioned markdown lesson; future runs retrieve the most relevant lessons by tag. No opaque memory, and a poisoning guard keeps rate-limit garbage out of the pool.Cost accounting— per-run token/dollar cost captured from the agent's own telemetry intorun_cost.json
and an append-onlyruns/ledger.jsonl
.Local review dashboard— every run's full evidence trail (prompt, claim, diff, verdict, cost, lesson) with an approve/reject gate. The decision is writtenbesidethe evidence, never into it.Zero dependencies— Python stdlib only, every source file ≤300 lines. You can read the whole engine in an afternoon.
Works with Claude Code headless (claude -p
) today; the
adapter interface is agent-agnostic by design —
run(prompt, cwd, timeout, …) -> RunResult
is the entire contract.
goal.md ─────► runner ─────► adapter ──────► agent (claude -p / mock / yours)
(task + │ │
verifiers) │◄───── REPORT.md ────────────┘ ◄── the CLAIM
│
├── guard: read_dirs snapshot before/after (trip ⇒ FAILED)
├── write flow: git worktree ─► write_diff.patch ─► branch
├── verifiers ─────────► verdict.json ◄── the CALL
├── distill (2nd call) ─► learnings/<id>.md
└── ledger ────────────► run_cost.json + runs/ledger.jsonl
│
linejudge dashboard (review + approve/reject)
Needs Python 3.10+. Nothing else — no API key required for the mock pipeline.
pip install linejudge
git clone https://github.com/phillipmex/linejudge && cd linejudge
linejudge run goals/examples/hello.md --dry-run
python proofs/demo.py --root demo
linejudge dashboard --root demo # http://127.0.0.1:8765
To run against a real agent, install Claude Code,
set ANTHROPIC_API_KEY
(directly or in a gitignored .env.local
), and drop the
--dry-run
:
linejudge run goals/examples/hello.md --root .
Step-by-step walkthrough with a write-mode goal: docs/quickstart.md.
A goal is one markdown file: a fenced header (deliberately not YAML — no nesting, no quoting rules, nothing to mis-parse) plus the prompt body.
---
name: fix-config-crash
tags:
- widget
read_dirs:
- /path/to/reference-repo # guarded read-only
write_repo: /path/to/widget # optional: write cycle via worktree
verifiers:
- command: python -m pytest -q
- files_exist: done.txt
- diff_constraints: max_files=5 deny=**/*.env
timeout_secs: 1800
---
Fix the config crash on empty YAML files. …
|
Verifier specAdapter guideComparisonclaude -p
loops, Aider, OpenHands, hosted agentsGovernance templatesADR-0001PROOF.mdSample PROOF.mdproofs/
turns real GitHub issues into goals and renders an honest scoreboard:
python proofs/generate.py --repo owner/name --limit 5 --verifier "command: pytest -q"
linejudge run proofs/goals/issue-….md --root proofs/root # per goal
python proofs/stats.py --root proofs/root # → PROOF.md
PROOF.md reports runs succeeded and independently verified pass % as separate numbers, because they are separate facts.
v0.2.0. 104 tests, CI on Windows + Ubuntu × Python 3.10/3.12, zero runtime dependencies.
Apache-2.0