Your Review Prompt Is a Dependency — Pin It Before It Reaches the Merge Queue

A proposed Python tool, tools/lock_review.py, generates a committed review.lock.json holding SHA-256 hashes of three pinned inputs — the prompt file prompts/review.md, decoding settings in review.settings.json, and the fixture set under fixtures/review/ — and fails CI with a --check flag when the prompt drifts without an explicit lock bump. The approach canonicalizes files before hashing (normalizing line endings, stripping trailing whitespace and blank lines, sorting settings keys) to avoid spurious drift signals, and separates three failure classes: transport errors that warrant a rerun, nondeterminism that warrants k-of-n agreement, and prompt drift that the lockfile catches. The author argues model-backed review jobs fail because prompts, decoding settings and fixtures behave like production code while being versioned like scratch notes.

Your Review Prompt Is a Dependency — Pin It Before It Reaches the Merge Queue Most model-backed review jobs do not fail because the model is bad. They fail because someone edited prompts/review.md at 4pm on a Friday, and nothing in the pipeline noticed. AI-authored diffs keep arriving faster than human review capacity grows, so more teams are wiring a model call into the PR p ath. That is a reasonable move. The problem is that the prompt, the decoding settings, and the fixture set now behave like production code, while being versioned like a scratch note. This article gives you a small, reproducible artifact for that gap: a review lockfile plus a checker that fails CI when the prompt drifts without an explicit lock bump. It is boring on purpose. Boring is what you want sitting between a git push and a merge queue. When a model-backed review job goes red, people reach for the same reflex: rerun it. That reflex is only correct for one of the three failure classes. Symptom Class Wrong fix Right fix 429, timeout, DNS blip Transport none needed — rerun it Backoff, request budget, alerting Same diff, different verdict Nondeterminism Retry until green k-of-n agreement, quarantine on split Verdict flips after an unreviewed prompt edit Drift Usually invisible, so nothing Lockfile + lock-bump PR The third row is the expensive one. It does not produce a flaky red. It produces a confident green on a rule nobody approved. So the goal is not "make the model deterministic." The goal is to make the inputs to the model reviewable, and to make disagreement loud instead of flaky. Pin exactly three things. Anything else is configuration you should not silently inherit. The prompt text — one file, no inlined strings scattered across the codebase. The decoding settings — temperature, max tokens, top-p, stop sequences, in a committed JSON file. The fixture set — a handful of diffs with the verdict label you expect. The model handle itself belongs in settings too, but treat it as a named slot your operator resolves, not a hardcoded identifier you scatter across jobs. When the slot's resolution changes, your lock should change with it — loudly. prompts/review.md the instruction text review.settings.json decoding params + model slot name fixtures/review/ .json diff + expected label review.lock.json generated hashes, committed Hashing raw file bytes is a trap. A line-ending change or a trailing space from your editor will produce a spurious drift signal, and your team will start ignoring the check within a week. Canonicalize first. Normalize line endings, strip trailing whitespace per line, strip the trailing blank lines, then hash. Sort settings keys before serializing. This is the whole trick, and it is the difference between a check people trust and a check people --no-verify around. /usr/bin/env python3 tools/lock review.py import argparse, hashlib, json, pathlib, sys ROOT = pathlib.Path file .resolve .parents 1 LOCK = ROOT / "review.lock.json" def canonical text path: pathlib.Path - str: raw = path.read text encoding="utf-8" .replace "\r\n", "\n" lines = line.rstrip for line in raw.split "\n" return "\n".join lines .strip + "\n" def sha text: str - str: return hashlib.sha256 text.encode "utf-8" .hexdigest def digest prompt - str: return sha canonical text ROOT / "prompts/review.md" def digest settings - str: settings = json.loads canonical text ROOT / "review.settings.json" blob = json.dumps settings, sort keys=True, separators= ",", ":" return sha blob def digest fixtures - str: files = sorted ROOT / "fixtures/review" .glob " .json" blob = "".join f"{f.name}:{sha canonical text f }\n" for f in files return sha blob def current - dict: return { "schema": 1, "prompt sha256": digest prompt , "settings sha256": digest settings , "fixtures sha256": digest fixtures , } def main - int: ap = argparse.ArgumentParser ap.add argument "--check", action="store true" ap.add argument "--update", action="store true" args = ap.parse args now = current if args.update: LOCK.write text json.dumps now, indent=2 + "\n", encoding="utf-8" print "lock written — open a lock-bump PR, do not push this alone" return 2 if not LOCK.exists : print "missing review.lock.json; run --update once and commit it" return 1 locked = json.loads LOCK.read text encoding="utf-8" drift = {k: locked.get k , v for k, v in now.items if locked.get k = v} if not drift: print "review lock OK" return 0 for key, was, is in drift.items : print f"DRIFT {key}: locked={was} current={is }" return 1 if name == " main ": sys.exit main Run it once locally to create the baseline, then treat the output as read-only: python tools/lock review.py --update && git add review.lock.json python tools/lock review.py --check echo $? expect 0 Now edit one word in prompts/review.md and rerun --check. You want exit code 1 and a printed prompt sha256 drift line. If you get a clean pass after editing the prompt, stop and fix the canonicalization before continuing. The CI check is the backstop. The hook is what saves your reviewers from a red build they did not cause. /usr/bin/env bash .git/hooks/pre-push set -euo pipefail if python tools/lock review.py --check; then echo "review.lock.json is stale." echo "Fix: python tools/lock review.py --update, then open a lock-bump PR." exit 1 fi Keep the CI side narrow. You only need it to run when the pinned inputs change: name: review-lock on: pull request: paths: - "prompts/ " - "review.settings.json" - "fixtures/review/ " - "tools/lock review.py" - "review.lock.json" jobs: lock: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-python@v5 with: python-version: "3.12" - run: python tools/lock review.py --check The lock-bump PR is the whole point. A prompt change now shows up as a reviewable diff of hashes plus the prompt body, owned by a human, instead of a silent behavioral change buried in a squash merge. Here is where most teams get flaky. They call the model once, get approve on one run and needs human on the next, and the job flaps. Retrying until green is not a fix; it is a coin flip with extra billing. Run the verdict k times per fixture and require agreement. On a split, emit quarantine — a distinct state that never masquerades as pass or fail. tools/fixture check.py sketch — call model is your own adapter import json, pathlib K = 3 LABELS = {"approve", "block", "needs human"} def verdict for diff: str - str: your transport, your auth, your timeout handling return call model prompt=PINNED PROMPT, diff=diff, settings=SETTINGS rows = for fx in sorted pathlib.Path "fixtures/review" .glob " .json" : data = json.loads fx.read text runs = verdict for data "diff" for in range K bad = r for r in runs if r not in LABELS if bad: state = "quarantine" unparseable output is not a verdict elif len set runs 1: state = "quarantine" split decision, human decides else: state = "pass" if runs 0 == data "expected label" else "fail" rows.append fx.name, state, runs for name, state, runs in rows: print f"{state:10s} {name:32s} {runs}" Two rules keep this honest. First, quarantine must be a first-class state in your reporting, not a relabeled failure. Second, never let the fixture runner write to review.lock.json — generating the lock and validating behavior are separate jobs with separate permissions. Write this down and put it next to the merge queue config. Ambiguity is where broken gates live. Lock state Fixture state Merge queue action drift not run Block. Needs a lock-bump PR. OK any fail Block. The prompt no longer matches your expected labels. OK any quarantine Hold. Route to a human, keep the diff out of the queue. OK all pass, verdict block Block on content, not on infrastructure. OK all pass, verdict approve Admit to queue. OK all pass, verdict needs human Route to a human. Do not treat as green. That last row matters more than it looks. A model that says "I cannot judge this" is not a passing check, and a pipeline that treats it as one has quietly converted uncertainty into approval. All five steps are runnable end to end before you ask anyone for spend, because none of them require a large sample. You are checking lock stability and label agreement on a dozen fixtures, not benchmarking quality. For that bootstrap phase I used MonkeyCode's free model access and its free server option to host the fixture runner, which meant the gate existed before the procurement conversation started. Check the current terms on the official MonkeyCode project page before you plan around them — free-tier scope and availability are exactly the kind of thing that changes, and I am not going to quote you numbers I cannot verify today. Disclosure: This article was prepared as part of MonkeyCode's product outreach. If your team already has a paid endpoint and a runner, skip the product entirely. Nothing above depends on it. The lockfile is the artifact; the vendor is a convenience. The lock proves provenance, not quality. A pinned, reviewed, thoroughly wrong prompt passes the lock check every time. k = 3 is a starting point, not a measurement. I have not measured disagreement rates on your diffs. Log the run counts for two weeks and pick k from your own data. Skip this if your review is fully deterministic linters, type checkers, policy engines . You already have this property for free; a hash lock adds nothing. Skip this if you cannot version prompts in the same repo as the fixtures. A lockfile that hashes files living in three different systems is a maintenance tax with no payoff. Do not use quarantine as a soft pass. If humans never triage the quarantine queue, you have built an approval machine with extra steps. The honest version of the pitch: this adds a small amount of friction to prompt edits and removes a class of silent behavior change. If your prompt changes weekly and nobody has ever been surprised by a verdict flip, you may not need it yet. When you do get surprised, come back — the checker is forty lines and it will still be here. Key Takeaways - •That is a reasonable move. - •This story was reported by Dev.to , covering developments in the dev space. - •AI advancements continue to reshape industries — read the full article on Dev.to for complete coverage. 📖 Continue reading the full article: Read Full Article on Dev.to → https://dev.to/codego 3211/your-review-prompt-is-a-dependency-pin-it-before-it-reaches-the-merge-queue-3mfe