Tadashi Shigeoka· Wed, July 22, 2026
Leave a pull request review to a single model and whatever that model structurally overlooks stays overlooked forever. Ask the same model twice and it walks past the same spot twice.
The obvious countermeasure is to have models from different lineages read the same diff. Claude Code and Codex differ in both training data and post-training, so their blind spots sit in different places. Anything both flag independently is very likely real. Anything only one of them raises might be useful or might be noise, and that is exactly the set a human should adjudicate.
I had been doing exactly that already: several terminal tabs, Claude Code and Codex each in their own session, then reading the results side by side. But walking through that by hand on every PR is tedious, and it is easy to forget to run one of them. Collapsing the same routine into a single command is where this started.
So I added a skill called omh-pr-multi-review
to oh-my-hermes (PR #5). This post covers its design and the implementation decisions behind it.
About the oh-my-hermes repository #
For context, oh-my-hermes is a collection of personal customizations for Hermes Agent. The rule is that the agent core is never modified: only skills, plugins, hooks, and config templates are bolted on from outside. Every identifier carries an omh
prefix so nothing collides with other extensions.
Skills are distributable as a tap.
The new omh-pr-multi-review
is one skill under that skills directory. It is only three files.
Don’t let the agent narrate the procedure #
The first decision was to keep the review procedure out of the agent’s judgment entirely.
When people hear “skill,” they usually picture a SKILL.md full of natural-language steps that the agent reads and executes in order. That is certainly easier to write. But for work with strong side effects like this (switch the checkout, invoke four CLIs in sequence, always return to the original branch), that shape is dangerous. Agents skip steps, and when something fails they may helpfully reach for something like git checkout -f
.
So SKILL.md says essentially one thing: run the bundled script.
All the actual control flow lives in review_pr.py
. The only discretion left to the agent is passing the PR URL and optionally overriding the output path or the timeout.
Draw the boundary between the deterministic script and the nondeterministic agent as early as possible. I think that generalizes well beyond this case, to any skill that carries side effects.
The overall flow #
flowchart TD
START[Receive PR URL] --> VAL[Validate URL format, CLI presence, auth,<br/>repository identity, clean working tree]
VAL -->|fail| ABORT[Abort with exit 2]
VAL -->|pass| CO[gh pr checkout --detach]
CO --> R1[claude /review]
CO --> R2[claude /security-review]
CO --> R3[claude /code-review]
CO --> R4[codex review]
R1 --> AGG[Aggregate four sections]
R2 --> AGG
R3 --> AGG
R4 --> AGG
AGG --> SUM[Generate cross-review summary<br/>via codex exec]
SUM --> OUT[Write Markdown report]
AGG --> RESTORE[Restore original checkout]
Front-load the validation #
Before doing anything else, the script checks every precondition it can. A failure caught here is vastly cheaper than the same failure caught after thirty minutes of reviewing.
The PR URL is matched strictly against a regular expression.
It uses fullmatch
, so nothing sneaks through on a partial match. Query strings and fragments (which browsers love to attach when you copy a URL) are tolerated.
Next, every required command must be on PATH
.
Then authentication. Each of the three CLIs gets a login check, and a failure aborts with a concrete remedy attached.
Putting “here is what to do next” into the error message matters more than usual for tools an agent drives. Given an explicit remedy, an agent fixes the problem itself; given a bare exit 1
, it starts guessing and doing strange things.
Two more checks are quiet but important.
The first confirms that the current repository matches the PR’s repository.
Running gh pr checkout
with a PR URL from a different repository leaves you in an incoherent state. GitHub owner and repository names are case-insensitive, so the comparison lowercases both sides.
The second is working tree cleanliness.
With --untracked-files=all
, even a stray untracked file stops the run. That may look overly strict, but this script switches branches, and I did not want to leave any path where a user’s local changes get caught up in that. The script never force-checks-out and never discards changes.
The reviewer lineup #
Once the checkout lands, four reviewers run in sequence. The Claude Code side invokes three slash commands with identical model settings.
-p
runs non-interactively, --effort high
buys deeper reasoning, and --no-session-persistence
leaves no session history behind. The three commands are kept separate because /review
is general review, /security-review
takes a security lens, and /code-review
does a detailed pass over the diff, each with its own prompt and perspective. Even within one model, changing the lens changes what gets caught.
The Codex side injects its settings inline.
Configuration goes through -c
because I do not want to touch the user’s config.toml
. A tool that mutates global configuration changes the state of your environment after a single use, and the accident shows up later, while you are doing something else. service_tier="fast"
selects the Fast tier in the current CLI, which noticeably shortens the wait.
The comparison base passed to --base
resolves to origin/<base>
using the baseRefName
read from gh pr view
. If the fetch fails, it falls back to the local <base>
and warns on stderr.
A stale local base inflates the diff and blurs the review, so a successful fetch is what you want. But degrading is better than refusing to run offline, so it continues in a degraded mode.
Isolate the failures #
Failure handling took the most care in this script.
Consider three reviews succeeding over thirty minutes and the fourth timing out. Failing the whole run and keeping nothing would be the worst possible design. The three successes have value, and the fact that the fourth died is itself information.
So per-reviewer results accumulate as tuples rather than raising.
Timeouts work the same way: partial output is rescued from subprocess.TimeoutExpired
and kept in the report.
Findings that made it out before the cutoff are worth reading on their own. The exit code is 124 to match the timeout(1)
convention.
If the checkout itself fails, all four sections are filled with the same error and the report is still written. A report explaining why it is empty beats an empty report.
Restoration always runs in a finally
block.
The restore target is the branch name recorded before the run. If the script was started from a detached HEAD, symbolic-ref
fails and it falls back to the commit SHA.
Even when restoration fails, the report is written and a warning is printed. Raising there and throwing the report away would just double the user’s loss.
Generating the cross-review summary #
Once all four results are in, the model gets called one last time. This is the point of the whole skill.
Three things matter in that instruction.
- Only pick up items that multiple reviewers flagged independently. Solo findings stay out of the summary.
- Merge duplicates, and name which reviewers agreed.
- If there is no credible overlap, say so.
The third is the critical one. Tell a summarizing model to “summarize” and it will produce something plausible-sounding even with nothing to work from. Unless you explicitly authorize the conclusion “there is no overlap,” the summary becomes a machine that always lies a little.
The call is read-only and ephemeral.
--sandbox read-only
removes file writes and --ephemeral
leaves no session. There is no reason for the summarization phase to touch code, so the capability is dropped entirely. The review results are written to a Markdown file in a temporary directory and only the path is passed, keeping a large blob of text off the command line.
If summary generation fails, the report as a whole is still a success.
The summary is a way in, not the evidence. The evidence is the raw output in each section. That ordering is written into SKILL.md’s output contract as well: do not present the generated summary as a replacement for the individual evidence.
The shape of the report #
The final output looks like this.
The UTC execution timestamp and the exact model options are recorded because rereading a report later is useless if you cannot tell which configuration produced it. Models get replaced within months. Judging the quality of findings in a six-month-old report requires knowing what was running at the time.
Practical impressions and cost #
The obvious cost is four high-effort inference runs per PR, plus one more for the summary. That is not cheap, especially with both --effort high
and model_reasoning_effort="high"
set.
So this is not something to run on every PR. It fits situations like these.
- PRs touching authentication, authorization, billing, or anywhere a mistake is expensive.
- Code that handles external input or spawns subprocesses.
- PRs you did not write yourself, or that an agent wrote a lot of, where you do not fully hold the shape in your head.
Conversely, throwing four models’ worth of reasoning at a dependency bump, a config value change, or a typo fix is simply waste. A single review remains the right amount for those.
There is also a wall-clock issue. The four reviews run serially, so you can wait through the default 1,800-second timeout four times over. I considered parallelizing, but that has four processes reading the same working tree at once, plus tangled output and rate-limit handling, so the first version stays serial. Splitting the working trees with git worktree would resolve it, and that remains on the list.
Conclusion #
A code review from a single model inherits that model’s blind spots wholesale. Have models from different lineages read the same diff independently, and look first at where their findings overlap. omh-pr-multi-review
makes that simple workflow reproducible in one command.
Three things drove the implementation. Confine procedural decisions to a deterministic script rather than the agent. Isolate reviewer failures so successful results always survive. And position the cross-review summary as a way in rather than a substitute for raw output, with explicit permission to report that no overlap exists.
The three prohibitions (never mutate global configuration, never force checkout, never run against a dirty working tree) mattered just as much for a tool an agent invokes. For a human-operated tool, “please be careful” would cover it. When an agent is the caller, the only real guarantee is not implementing the dangerous path at all.
That’s all from building a Hermes Agent skill that has Claude Code and Codex review the same PR independently and cross-checks their findings, from the Gemba.