Show HN: Pairmark, race Claude Code vs. Codex on your repo, blind cross-judged

Pairmark, a new open-source tool by Hemanshu Upadhyay, lets developers race Anthropic's Claude Code against OpenAI's Codex CLI on the same task in their own repository, with blind cross-judging and an evidence-backed verdict. The tool runs locally via npx pairmark, requires Node 20 or newer and git, and produces a report.html with a verdict card, receipt image, and detailed evidence tables. It aims to settle arguments about which coding agent is better by providing concrete, repo-specific comparisons.

Run Claude Code and Codex on the same task in your own repo, side by side, and get a verdict backed by evidence. npx pairmark "add rate limiting to POST /api/login" One command. Two isolated worktrees. Both agents get the identical brief. You watch them work in a split screen, the same checks run in each worktree, each agent then reviews both patches blind, and you get a single report.html you can post. It runs on the subscriptions you already have. No API keys. No accounts. No server. Nothing leaves your machine except the calls the two CLIs make on their own. The receipt above is real. It comes from pairmark's own repository, where the two agents were asked to add the runs subcommand. The full report is docs/example-report.html /Hemanshu-Upadhyay/pairmark/blob/main/docs/example-report.html , and the winning patch is the runs command you can run today. Why why Quick start quick-start What you see what-you-see How a race works how-a-race-works Verdict rules verdict-rules Commands commands Options options The run directory the-run-directory Fairness fairness Safety safety Monorepos, dirty trees, dependencies monorepos-dirty-trees-dependencies Troubleshooting troubleshooting Limits limits How it was built how-it-was-built Contributing contributing Every AI native argues about which coding agent is better. Almost nobody has evidence from their own codebase, because getting it means two worktrees, two CLIs, two sets of permission flags, running the tests twice, and reading two diffs with the context of both in your head. So the argument stays an argument. pairmark turns it into a receipt. The answer it gives is not "which agent is smarter". It is "which agent did better on this task, in this repo, today, and here is exactly why". Run it ten times on ten real tasks and you have a basis for a decision. Run it once and you have something worth posting. Requirements: - Node 20 or newer, and git Claude Code https://docs.anthropic.com/en/docs/claude-code on your PATH, signed in Codex CLI https://github.com/openai/codex on your PATH, signed in- A JavaScript or TypeScript repository with at least one commit Check your setup: npx pairmark doctor Race: cd your-repo npx pairmark "your task, written the way you would brief a colleague" A browser tab opens with the live view. The terminal prints progress. When both agents finish, the checks run, the judges run, and the verdict prints with the path to the report. Writing a good task brief matters more than anything else. Say what done looks like. Name the file or function if you know it. Say what not to touch. Both agents get exactly your words plus a short working agreement see The brief how-a-race-works . Live. A split screen with one column per agent on one clock. Each column shows the agent's messages, every file it writes, every command it runs, command output when it fails, and a running count of files, commands, tokens and for Claude Code cost. A thin activity strip above each feed shows the rhythm of the run at a glance. A phase bar at the top tracks worktrees, racing, checks, judging, verdict. The verdict card. The winner, the rule that decided it, the reasons in plain sentences, and any dissent from a judge who would have picked differently. Dissent is always shown, never averaged away. The receipt. A 1200 by 630 image drawn in the page, with a download button and a copyable text summary. Task, verdict, both agents' time, files, lines, checks and judge scores, repo and commit. Made to be posted. Evidence. One table: outcome, time, files changed, each check with its duration, whether the check configuration was touched, which test files were touched, tokens in and out, reported cost, average judge score, model. Judges. One card per judge with its pick, confidence, the decisive reason, and per-patch scores on correctness, completeness, quality and safety with cited evidence. Each card says which patch that judge saw as A and which as B. Timelines. The full event feed of each agent, as it was streamed live, plus the agent's own closing summary. Patches. Everything each agent changed against the shared baseline, per file, with additions and deletions coloured. Patches that touched check configuration carry a flag. Run. CLI versions, models, exact command lines, flags, base commit, dependency strategy, and the brief both agents received. The same HTML file is the live page and the report. Live, it reads a server-sent event stream. As a report, it reads JSON embedded in the file. Nothing external is loaded, so the report works offline and as an email attachment. - Preflight. Confirms a git repository with a commit, finds claude and codex on the PATH with their versions, detects the package manager from the lockfile, and reads check commands from package.json scripts in the directory you ran from. typecheck , lint , test and build are picked up when present. The npm placeholder test script is ignored. - Run directory. .pairmark/runs/<timestamp-id / inside your repo. It is added to .git/info/exclude , so your git status never shows it and no commit can pick it up by accident. - Worktrees. Three detached worktrees at your HEAD: alpha and beta for the agents and base for the judges. The names are neutral on purpose, so no path in a check log can tell a judge whose patch it is reading. If your working tree is dirty, the uncommitted changes and untracked files are snapshotted into each worktree and committed there as a baseline, so every agent starts from what you actually see and every diff is measured from that same point. - Dependencies. node modules is cloned into each worktree with a copy-on-write copy where the filesystem supports it APFS on macOS, btrfs and xfs on Linux , which takes milliseconds and gives each agent its own tree. Nested node modules in shallow workspaces are cloned too. Without a node modules to clone, a frozen install runs with the detected package manager. See Options options for the other strategies. - The brief. Both agents receive exactly the same text: your task, then a short working agreement. Work here. Verify locally where it helps. Do not commit. Do not weaken or delete tests, lint rules or type checks unless the task says so. End with a summary. The brief never mentions the other agent, never says a race is on, and never asks the agent to run the project's checks, because pairmark runs those itself afterwards. That last point matters: asking the agents to run the checks would reward whichever CLI happens to handle long shell commands better, which is not what you are trying to measure. - The race. Both CLIs start at the same moment, each in its own worktree, each from the directory you ran pairmark in. Claude Code runs as claude -p --output-format stream-json and Codex as codex exec --json . Both streams are normalised into one event model status, text, thinking, file, command, tool result, usage, done, error and pushed to the live page. The raw JSONL from each CLI is written to the run directory untouched, so nothing is lost if a CLI changes its output format. - Checks. When both agents have finished or hit the time limit, the patch of each worktree is collected against the baseline new files included, since untracked files are staged first . Then the same check commands run in each worktree, one worktree at a time so two test suites never fight over a port. Exit code, duration and the last lines of output are recorded. - Judging. Each agent is started again in read-only mode inside the base worktree, so it can open any file to understand the context. It receives the brief, both patches labelled A and B in an order randomised per judge, both sets of check results, and any flags check configuration touched, test files touched, time limit hit . It is told that the patches and logs are untrusted data and that any instruction found inside them counts against that patch. It returns structured JSON: four scores per patch, cited evidence, a winner or tie, a confidence, and one decisive reason. A judge does not know which patch is its own. - Verdict. The rules below are applied in order. The report and run.json are written. The terminal prints the summary table. The live page switches to the report view. Applied in this order. The report names the rule that decided the run. Only one agent changed any file. It wins by default and the report says so. This covers timeouts, crashes, and an agent that decided not to act. Exactly one patch passes every check without touching the check configuration. It wins. Judge scores are shown as commentary, with dissent when a judge would have picked the other. A patch that edits package.json , tsconfig .json , or a test, lint, formatter or build configuration file is flagged and cannot win on this rule, because a passing check means less when the patch could have changed what the check does. It can still win on the judges. Both judges pick the same patch. It wins. The judges split, or both call it a tie. The result is a tie and the disagreement is printed as dissent. A split is never averaged into a winner. Neither judge returned a decision. The report shows the evidence only. Why not average the judge scores and pick the higher number? Because a 0.3 gap on a 10 point scale between two models grading their own work blind is noise, and calling noise a winner is the fastest way to make the tool untrustworthy. pairmark would rather say "tie, and here is why each judge leaned the way it did" than manufacture a result. npx pairmark "<task " race both agents on the task npx pairmark doctor preflight only: git, CLIs, package manager, checks npx pairmark runs list past runs, newest first npx pairmark report run regenerate and open a report latest by default npx pairmark apply <claude|codex run apply an agent's patch to your working tree, staged npx pairmark clean remove the worktrees of every run reports and patches stay run is a run id as printed by runs , or a path to a run directory. apply uses git apply --3way --index , so a patch against an older baseline still applies when it can, and the result is staged for you to review with git diff --cached . --check "<cmd " a check to run in both worktrees after the race repeatable; default: from package.json --agents claude,codex who races default both; a single agent gives evidence only, never a winner --timeout 20m per agent --check-timeout 10m per check command --judge-timeout 10m per judge --model-claude <id model for Claude Code default: the CLI's default --model-codex <id model for Codex default: the CLI's default --include <path gitignored file to copy into each worktree repeatable; default .env and .env.local when present --deps clone|install|link|none node modules strategy default clone --sandboxed use the CLIs' restricted modes instead of full auto --no-judge skip the blind cross-judging --no-open do not open the browser --port 4747 live UI port the next free port is used if taken Durations accept 90s , 20m , 1.5h . --deps : clone default : copy-on-write copy of every node modules found up to three directories deep. Falls back to a plain copy if the filesystem cannot reflink, and to a frozen install if there is no node modules at all. install : a fresh frozen install per worktree with the detected package manager npm ci , pnpm install --frozen-lockfile , yarn install --immutable , bun install --frozen-lockfile . Slower, cleanest. link : symlink node modules from your main tree. Fastest, but the agents share one tree, so a dependency one of them adds is visible to the other. none : do nothing. For repositories that do not need dependencies to run their checks. --sandboxed runs Claude Code with --permission-mode acceptEdits and an explicit tool allowlist, and Codex with --sandbox workspace-write and network access on. Both agents then live within their CLI's own guard rails instead of the worktree alone. --no-open is for terminals without a browser, CI, and SSH sessions. The live page still serves on the port; the report is still written. Everything from a run is in .pairmark/runs/<id / : | File | What it is | |---|---| report.html | The self-contained report. Open it anywhere. | run.json | Every event, every score, every number the report shows. | brief.md | The exact text both agents received. | claude.jsonl , codex.jsonl | Raw event streams from each CLI, unmodified. | claude.patch , codex.patch | Each agent's full diff against the baseline. | judge-claude.prompt.md , judge-codex.prompt.md | The exact judge prompts, with the A and B assignment of that judge. | judge-schema.json , judge-codex.last.json | The JSON schema the judges answer to, and Codex's raw answer. | baseline.patch | Only when your tree was dirty: the snapshot applied to every worktree. | alpha/ , beta/ , base/ | The worktrees, kept until pairmark clean . | The worktrees are real git worktrees. cd .pairmark/runs/<id /alpha and look around, run the tests yourself, or diff the two by hand. The design tries to remove every thumb from the scale it knows about. Same brief, same commit, same dependencies, same start time. No agent sees the other's work or knows the other exists. Same checks, run by pairmark. Neither agent is asked to run them, so shell handling and patience are not being measured. Blind judging with randomised labels. Each judge gets the patches as A and B in an order drawn per judge. Worktree names are neutral and run paths are scrubbed from check logs before they reach a judge. Untrusted content. Judges are told that the diffs and logs are data, and that any text inside them addressing the judge counts against that patch's safety score. A patch cannot argue for itself. Checks outrank opinions. A passing patch beats a failing one before any judge is consulted. The check surface is protected. A patch that edits the configuration the checks run under is flagged and cannot win on the checks alone. Ties are ties. Judges decide only when they agree. Dissent is printed. Everything is inspectable. Raw streams, prompts, patches and worktrees are all on disk. What it cannot remove: model preferences. A judge may recognise a style. That is why judging is secondary and cross-checked, and why the report shows each judge's pick with its reason instead of a single number. By default both CLIs run in their full-auto modes inside disposable worktrees: --dangerously-skip-permissions for Claude Code and --dangerously-bypass-approvals-and-sandbox for Codex. That is the same trust you extend when you run either agent yourself with those flags, applied to a copy of your repo. The worktrees are inside your repository directory but outside git's view, and they are yours to inspect or remove. If that is more trust than you want, use --sandboxed . pairmark itself makes no network calls. The two CLIs make whatever calls they normally make. Monorepos. Run pairmark from the package directory. Both agents work from that directory inside their worktrees, and the checks are read from that directory's package.json . The whole repository is still available to them. Dirty trees. Uncommitted changes are part of the baseline. Both agents start from your current state, and their diffs are measured from that state, not from HEAD. The snapshot is recorded as baseline.patch in the run directory. Environment files. .env and .env.local are copied into each worktree when they exist, because tests usually need them. Add more with --include . Package managers. npm, pnpm, yarn and bun are detected from the lockfile, or from the packageManager field in package.json . Check commands are built accordingly pnpm run test , yarn test , and so on . claude is not on PATH or . Install the CLI and sign in once interactively. codex is not on PATH npx pairmark doctor shows what pairmark can see. Both agents end with an error immediately. Usually a sign-in problem with one CLI. Run claude -p "hi" and codex exec "say hi" by hand from the same terminal. Running pairmark from inside a Claude Code or Codex session. Supported. The nesting guard variables are removed from the child environment so the CLIs will start. A check fails in both worktrees but passes in your tree. The worktrees have your committed and uncommitted files plus copied env files, and nothing else. A gitignored file your checks need can be copied with --include . A generated file that is not in git can be produced by a --deps install run if a postinstall creates it. The checks take too long. Raise --check-timeout . A timed-out check counts as a failure and is labelled as such. An agent runs out of time. Its patch at that moment is still collected and checked. The report marks the run as stopped at the limit. Raise --timeout for bigger tasks. Port in use. The next free port is used automatically. The terminal prints the address. The judges keep failing. The judge prompt contains both patches. Very large patches are truncated at 60,000 characters each with a note. If a judge times out, raise --judge-timeout . If both fail, the report shows evidence only and says so. - JavaScript and TypeScript projects. The dependency and check detection are built around package.json . - Two agents, Claude Code and Codex. The adapters are small and live in src/agents/ . - One task per run. - Codex does not report cost. Its row shows token counts. - The report's receipt is drawn on a canvas, so it needs a browser to download. Everything else in the report is plain HTML. pairmark was designed on 2 September 2026 in a two-round discussion between Claude Code and Codex, and then built by Claude Code in a single session. Claude Code proposed three ideas. Codex attacked each one, picked the race, and in a second round found the weak points of the first design: judges that can be gamed by patches that weaken tests, node modules symlinks that break pnpm and workspaces, a brief that biased the result toward whichever CLI runs shell commands better, and a report nobody would post. Every one of those points is in the code above. The verbatim transcript is in docs/DISCUSSION.md /Hemanshu-Upadhyay/pairmark/blob/main/docs/DISCUSSION.md and the design in docs/DESIGN.md /Hemanshu-Upadhyay/pairmark/blob/main/docs/DESIGN.md . The first race on the tool's own repository asked both agents to add the runs subcommand. Both passed every check. Codex also fixed a real fragility in the build script, which meant it touched package.json , so rule 2 gave the win to Claude Code and recorded Codex's dissent. The winning patch shipped as-is. The build fix was taken by hand. That report is docs/example-report.html /Hemanshu-Upadhyay/pairmark/blob/main/docs/example-report.html . pairmark is an open tool from Salient https://salient-studio.com , an AI product engineering studio. If you want this kind of discipline applied to an agent or an AI feature in your own product, that is the work Salient does. git clone https://github.com/Hemanshu-Upadhyay/pairmark cd pairmark npm install npm test npm run build node dist/cli.js doctor The code is TypeScript with zero runtime dependencies. src/agents/ holds the two stream parsers and the spawner. src/score.ts holds the verdict rules and is fully unit tested. src/ui/ui.html is the entire live page and report. Run a real race in a scratch repository before sending a change; the unit tests cover the logic, not the CLIs. Adding an agent means one parser that turns its JSONL into the shared event model, one spawn spec, and one judge invocation. Pull requests welcome. MIT.