Measuring Review in Human Minutes: A Risk-Tier Gate for Agent Pull Requests in .NET

A .NET developer ran a ten-pull-request experiment testing a risk-tier gate that assigns each kind of agent-generated code change a required number of human review minutes, after concluding that speed-reading diffs from AI agents is "a ritual that makes me feel responsible" rather than review. The experiment was prompted by Anthropic's October 15, 2026 post "How the Claude Code team does code review" by Addy Osmani, Morgan Westlee Lunt and Qing Wang, which reported Anthropic-internal figures including changes landing per day rising from roughly 2,500 to 30,000+ in the Anthropic monorepo between May and September, merged PRs more than doubling in the Claude Code repo from June to September, the share of merged PRs reverted falling more than 40% relative, internal PRs with substantive review comments rising from 16% to 54%, and mean reviews to pass falling from 2.08 to 1.32. The author said he took no numbers from two JIN System Architect rewrites of the Anthropic post (items 124 and 154) and went to the original, noting Anthropic publishes no denominators and that the figures cover different scopes.

For most of this year my review habit has been the same as it was in 2019: open the PR, read the diff top to bottom, approve. That habit quietly broke once agents started opening PRs for me. Not because the code got worse, but because there was more of it than I had attention for, and I noticed I was approving the boring ones faster and faster. Speed-reading a diff is not review. It is a ritual that makes me feel responsible. So I set myself a narrower question: if I stop reading every line, what replaces it, and how many human minutes does each kind of change really need? I wanted a number, not a vibe. This post is the answer I got from a small experiment on a .NET repo. It is ten PRs. Keep that in mind the whole way through. The post that pushed me into this was Anthropic’s “How the Claude Code team does code review” by Addy Osmani, Morgan Westlee Lunt and Qing Wang, published October 15, 2026 the URL slug is solving-code-review-at-anthropic : https://claude.dev/blog/solving-code-review-at-anthropic/ https://claude.dev/blog/solving-code-review-at-anthropic/ I first met it secondhand. Two rewrites by the JIN System Architect publication showed up in my weekly digest items 124 and 154 . Both retell the same post, add little beyond it, and both had the tells that make me stop trusting a rewrite: a garbled disclosure paragraph, and in one of them the phrase “The material reports” left in the body, which reads like a summarizer’s scaffolding that nobody removed. I did not take a single number from them. I went to the original. Here are the figures from the original, with the scope each one actually has. All of them are Anthropic’s own internal numbers. I can’t verify any of them, and the post doesn’t publish denominators. +-----------------------------------------------+----------------------+-------------------------------+| Metric Anthropic-reported | Before - After | Scope |+-----------------------------------------------+----------------------+-------------------------------+| Changes landing per day | ~2,500 - 30,000+ | Anthropic monorepo, May- Sep || Merged PRs | more than doubled | Claude Code repo, Jun- Sep || Share of merged PRs reverted | down 40% relative | Claude Code repo, Jun- Sep || Internal PRs with substantive review comments | 16% - 54% | at Code Review preview launch || Pass after one review-and-fix round | 65% - 78% | internal PRs, convergence fix || Need five or more rounds | 8.9% - 1.5% | internal PRs, convergence fix || Mean reviews to pass | 2.08 - 1.32 | internal PRs, convergence fix |+-----------------------------------------------+----------------------+-------------------------------+ The scope column matters. The monorepo number is about Anthropic’s monorepo, which the post says is separate from the Claude Code repo. The revert figure is about the Claude Code codebase. The review-comment and convergence figures are about internal PRs. It is easy to blur these into one “agents made review better” headline, and the post itself doesn’t do that. The numbers were not what changed my mind, though. These ideas were: None of this needs Anthropic’s tooling. That was the point of trying it on my own repo. My digest that week had a cluster of related pieces. I’m separating what I read in a primary source from what I only saw described, and what I actually tried. Uber, four narrow agents. The newsletter description was that Uber split review into four narrow agents instead of one generalist. I did not verify the “four”. What I did read is Uber’s own uReview post, which describes a multi-stage pipeline that decides which files are eligible, generates comments, then validates them before posting, and it names false positives as the main challenge: hallucinated comments, and comments that are technically valid but don’t matter in context. That second category is the one I underestimated. Tried: no, it’s internal to Uber. Borrowed: the idea that the filter after generation matters as much as the generator. Lauren Tan’s verification skill. As described in write-ups of her talk, the idea is that the agent gets a project-local skill that launches the real app, drives it like a user, and captures evidence screenshots, transcripts, logs so the PR arrives with proof instead of “Done.” The PR counts quoted for her talk differ between sources one write-up says about 2,000 a month, a post title elsewhere says 2,500 , so I’m not repeating either as fact. Tried: the idea, not her code. My gate refuses medium and high PRs that don’t carry pasted test output, and refuses high PRs without a recorded run. kunchenguid’s no-mistakes. A local git proxy: you git push no-mistakes instead of origin, it runs a review, test, docs and lint pipeline in a disposable worktree, and only forwards the branch and opens the PR once everything is green. The README install is a curl | sh line: Don't pipe this straight into sh. Download it, read it, then run it.curl -fsSL https://raw.githubusercontent.com/kunchenguid/no-mistakes/main/docs/install.sh -o no-mistakes-install.shless no-mistakes-install.shsh no-mistakes-install.sh I read the script before running it, and I’d ask you to do the same with any installer, including mine. Tried: yes, on the demo repo. It does what it says. I kept my gate on the server side anyway, for one reason: a pre-push gate only runs if the pusher goes through it, and an agent with push rights can push to origin directly. A pre-push gate is a nice first filter. It is not the gate. The Code’s Saturday special: CODEOWNERS plus a PR template. The newsletter described pairing CODEOWNERS with a PR template so ownership and evidence are both structural instead of social. Tried: yes, it’s the base of everything below. One thing the newsletter version skipped: CODEOWNERS decides who must review, it cannot say how many . Branch protection’s approval count is repo-wide. So “two people for auth” needs a check of its own. Jaroslaw Wasowski on brownfield specs. His piece on a “preservation contract” argues that a change spec for legacy code should start with what must not change. His opening example which he says is constructed for the article is an agent “cleaning up” a shared fetch function and dropping a visibility filter that the export path silently depended on. Keep that example in mind for section 5. His companion piece on missing spec decisions, with a two-implementations technique and a decision register, is on my list; I haven’t worked through it, so I won’t summarize it. The repo is a demo ASP.NET Core 8 orders API with EF Core: Orders.Api, Orders.Application, Orders.Domain, Orders.Infrastructure, and tests. The org and team names below orders-demo, orders-team, platform-leads are that demo repo's real names, so the files work as-is if you mirror the layout. Here’s the shape of it: agent pushes branch | v+------------------+ +-----------------------------+| pr-checks.yml | | pr-gate.yml || pull request | | pull request target + || | | pull request review || build, test, | | base-branch code only: || upload .trx | | 1. classify tier || | | 2. label risk:low/med/high || optional: | | 3. check proof in body || local Ollama | | 4. count human approvals || first pass | | on the head commit || advisory | | 0 / 1 / 2 by tier |+------------------+ +-----------------------------+ | | +--------------+--------------+ v branch protection: both required, CODEOWNERS review required, stale approvals dismissed on push I wrote the rules down before writing code, because the post’s advice to “start strict and loosen with evidence” only works if the starting point is explicit. +--------+--------------------------------------------+----------------------+------------------+| Tier | Triggered by | Human approvals | Proof required |+--------+--------------------------------------------+----------------------+------------------+| low | only docs/, .md, tests/, Experimental/ | 0 CI + proof green | template filled || | and under 300 lines / 15 files | | || medium | any other src/ or appsettings .json, | 1 | dotnet test || | unknown paths, 300 lines or 15 files, | | output pasted || | or tests shrank by 20 lines | | || high | Auth/, Migrations/, Program.cs, prod | 2 | test output + || | settings, package versions, .github/, | | recorded run || | the gate tool itself, or 800 lines | | |+--------+--------------------------------------------+----------------------+------------------+ Two rules I added after thinking about how agents fail rather than how humans fail. Unknown paths are never low, because a new top-level folder is exactly where an agent puts something you didn’t expect. And if tests shrink, the PR is at least medium, because “fix the red build” sometimes gets solved by deleting the red. .github/risk-tiers.json: { "high": "src/Orders.Api/Auth/ ", "src/Orders.Infrastructure/Persistence/Migrations/ ", "src/Orders.Api/Program.cs", " /appsettings.Production.json", "Directory.Packages.props", ".github/workflows/ ", ".github/CODEOWNERS", ".github/risk-tiers.json", "tools/RiskGate/ " , "medium": "src/ ", " /appsettings .json", " .sln", " / .csproj" , "low": "docs/ ", " / .md", "tests/ ", "src/Orders.Api/Experimental/ " , "mediumLines": 300, "highLines": 800, "mediumFiles": 15, "testShrinkLines": 20} Experimental/ is low because everything in it is behind a feature flag and auto-mapped by a startup hook that was itself reviewed as high. If your flagged code isn't isolated like that, don't copy this line. .github/CODEOWNERS: Three tiers. CODEOWNERS decides WHO must look; the pr-gate workflow decides HOW MANY. Last matching pattern wins, so the broad rules come first and the strict ones last. Tier 1, low: docs, tests and flagged experiments./docs/ @orders-demo/everyone/tests/ @orders-demo/everyone/src/Orders.Api/Experimental/ @orders-demo/everyone Tier 2, medium: ordinary product code. One person from the owning team./src/ @orders-demo/orders-team Tier 3, high: auth, migrations, startup, CI and the gate itself./src/Orders.Api/Auth/ @orders-demo/platform-leads/src/Orders.Infrastructure/Persistence/Migrations/ @orders-demo/platform-leads/src/Orders.Api/Program.cs @orders-demo/platform-leads/Directory.Packages.props @orders-demo/platform-leads/.github/ @orders-demo/platform-leads/tools/RiskGate/ @orders-demo/platform-leads A subtlety: /src/Orders.Api/Experimental/ comes before /src/, so /src/ wins for experimental files and the orders team owns them. That's what I want for ownership. The count still comes from the tier, which says zero for experimental-only PRs. Ownership and approval count are different questions, which is exactly why the count can't live in CODEOWNERS. .github/pull request template.md: What and why< -- One paragraph. Link the issue. If an agent wrote this, name the agent and who directed it. -- Testing< -- Paste the real output, not a summary. Medium and high tiers need the dotnet test summary line, e.g.The checker strips HTML comments before deciding a section is filled, so an agent can’t pass by leaving the hints in place. One console app, three subcommands: classify reads a numstat diff and prints a tier, proof checks the PR body against the tier, and review asks a local Ollama model for a first pass. No NuGet packages, just the BCL, so it restores fast in CI. tools/RiskGate/RiskGate.csproj: <Project Sdk="Microsoft.NET.Sdk" <PropertyGroup <OutputType Exe</OutputType <TargetFramework net8.0</TargetFramework <Nullable enable</Nullable <ImplicitUsings enable</ImplicitUsings <AssemblyName riskgate</AssemblyName </PropertyGroup </Project tools/RiskGate/Program.cs: using System.Net.Http.Json;using System.Text;using System.Text.Json;using System.Text.Json.Serialization;using System.Text.RegularExpressions;// riskgate: labels an agent PR low/medium/high, checks the proof in the PR body,// and optionally asks a local Ollama model for a first-pass review.//// riskgate classify --numstat numstat.txt --config .github/risk-tiers.json --summary out.md // riskgate proof --body pr body.md --tier medium// riskgate review --diff pr.diff --review-md REVIEW.md --ollama http://localhost:11434 --model qwen2.5-coder:14b --out review.md var cmd = args.FirstOrDefault ?? "";var opt = ParseOptions args.Skip 1 .ToArray ;return cmd switch{ "classify" = Classify opt , "proof" = Proof opt , "review" = await Review opt , = Usage };static int Usage { Console.Error.WriteLine "usage: riskgate classify|proof|review options " ; return 2;}static Dictionary<string, string ParseOptions string a { var d = new Dictionary<string, string StringComparer.OrdinalIgnoreCase ; for var i = 0; i < a.Length - 1; i += 2 d a i .TrimStart '-' = a i + 1 ; return d;}static string Get Dictionary<string, string o, string key, string? fallback = null = o.TryGetValue key, out var v ? v : fallback ?? throw new ArgumentException $"missing --{key}" ;// ---------------------------------------------------------------- classifystatic int Classify Dictionary<string, string o { var config = JsonSerializer.Deserialize<TierConfig File.ReadAllText Get o, "config" , new JsonSerializerOptions { PropertyNameCaseInsensitive = true } ; var files = File.ReadAllLines Get o, "numstat" .Where l = string.IsNullOrWhiteSpace l .Select ParseNumstat .ToList ; var reasons = new List<string ; var tier = Tier.Low; // A rename is judged by BOTH its old and new path, so moving a file out of Auth/ is still high. foreach var f in files foreach var p in f.Paths { var fileTier = config.High.Any g = Glob.IsMatch g, p ? Tier.High : config.Low.Any g = Glob.IsMatch g, p ? Tier.Low : config.Medium.Any g = Glob.IsMatch g, p ? Tier.Medium : Tier.Medium; // unknown paths are never low if fileTier Tier.Low reasons.Add $" {p} is {fileTier.ToString .ToLower } by path" ; tier = Max tier, fileTier ; } var lines = files.Sum f = f.Added + f.Deleted ; if lines config.HighLines { reasons.Add $"{lines} changed lines {config.HighLines} " ; tier = Tier.High; } else if lines config.MediumLines || files.Count config.MediumFiles { reasons.Add $"{lines} changed lines in {files.Count} files medium threshold {config.MediumLines} lines / {config.MediumFiles} files " ; tier = Max tier, Tier.Medium ; } // Agents sometimes "fix" a red build by deleting or weakening tests. var testAdded = files.Where f = IsTest f.Paths ^1 .Sum f = f.Added ; var testDeleted = files.Where f = IsTest f.Paths ^1 .Sum f = f.Deleted ; if testDeleted - testAdded config.TestShrinkLines { reasons.Add $"tests shrank by {testDeleted - testAdded} lines" ; tier = Max tier, Tier.Medium ; } var name = tier.ToString .ToLower ; var summary = new StringBuilder $" Risk tier: {name}\n\n" ; summary.AppendLine $"{files.Count} files, {lines} changed lines.\n" ; foreach var r in reasons.Distinct .Take 15 summary.AppendLine $"- {r}" ; if reasons.Count == 0 summary.AppendLine "- only low-risk paths, under size thresholds" ; Console.WriteLine summary ; if o.TryGetValue "summary", out var path File.WriteAllText path, summary.ToString ; var ghOut = Environment.GetEnvironmentVariable "GITHUB OUTPUT" ; if string.IsNullOrEmpty ghOut File.AppendAllText ghOut, $"tier={name}\n" ; return 0; static bool IsTest string p = p.StartsWith "tests/" || p.Contains ".Tests/" ;}static FileChange ParseNumstat string line { var parts = line.Split '\t' ; var added = int.TryParse parts 0 , out var a ? a : 0; // "-" for binary files var deleted = int.TryParse parts 1 , out var d ? d : 0; var path = parts 2 ; // Renames: "src/{Old = New}/File.cs" or "old/path.cs = new/path.cs". Keep both paths. var brace = Regex.Match path, @"\{ . = . \}" ; if brace.Success return new FileChange Clean path.Replace brace.Value, brace.Groups 1 .Value , Clean path.Replace brace.Value, brace.Groups 2 .Value , added, deleted ; if path.Contains " = " return new FileChange path ..path.IndexOf " = " , path path.IndexOf " = " + 4 .. , added, deleted ; return new FileChange path , added, deleted ; static string Clean string p = p.Replace "//", "/" ;}static Tier Max Tier a, Tier b = a b ? a : b;// ---------------------------------------------------------------- proofstatic int Proof Dictionary<string, string o { var body = File.ReadAllText Get o, "body" .Replace "\r\n", "\n" ; var tier = Enum.Parse<Tier Get o, "tier" , ignoreCase: true ; var problems = new List<string ; foreach var section in new { "Testing", "Risk", "How to revert" } { var text = Section body, section ; if string.IsNullOrWhiteSpace text problems.Add $" {section} is missing or empty" ; } var testing = Section body, "Testing" ?? ""; var testRun = Regex.IsMatch testing, @"Failed:\s+0,\s+Passed:\s+ 1-9 \d " ; if tier = Tier.Medium && testRun problems.Add " Testing needs pasted dotnet test output with Failed: 0, Passed: N N 0 " ; if tier == Tier.High { var recorded = Regex.IsMatch body, @"HTTP/ 1\.1|2 \s+\d{3}" || Regex.IsMatch body, @"https?://\S+\. gif|mp4|webm|cast|png \b", RegexOptions.IgnoreCase ; if recorded problems.Add "high tier needs a recorded run: an HTTP transcript curl -i output or a link to a recording/screenshot" ; } if problems.Count == 0 { Console.WriteLine $"proof ok for {tier.ToString .ToLower } tier" ; return 0; } foreach var p in problems Console.WriteLine $"::error::{p}" ; return 1;}// Text under " Name" up to the next " ", with HTML comments template hints removed.static string? Section string body, string name { var m = Regex.Match body, $@"^ \s+{Regex.Escape name }\s $ . ? ?=^ \s|\z ", RegexOptions.Multiline | RegexOptions.Singleline | RegexOptions.IgnoreCase ; if m.Success return null; return Regex.Replace m.Groups 1 .Value, @"< --. ?-- ", "", RegexOptions.Singleline .Trim ;}// ---------------------------------------------------------------- review Ollama static async Task<int Review Dictionary<string, string o { var diff = File.ReadAllText Get o, "diff" ; var reviewMd = o.TryGetValue "review-md", out var rp && File.Exists rp ? File.ReadAllText rp : ""; var baseUrl = Get o, "ollama", "http://localhost:11434" .TrimEnd '/' ; var model = Get o, "model", "qwen2.5-coder:14b" ; const int maxDiffChars = 40 000; if diff.Length maxDiffChars diff = diff ..maxDiffChars + "\n diff truncated "; var system = """ You review a pull request diff written by a coding agent. Report only bugs that would change runtime behavior: wrong results, missing authorization or tenant filters, data loss, security weakening, broken error handling. No style, naming or formatting comments. Every finding must quote the exact changed line as evidence and give its file and line. If nothing is wrong, return an empty list. Answer with JSON only: {"findings": {"file":"","line":0,"severity":"important|nit","claim":"","evidence":""} } """; var user = reviewMd.Length 0 ? $"Repository review rules:\n{reviewMd}\n\n" : "" + $"Diff:\n{diff}"; using var http = new HttpClient { BaseAddress = new Uri baseUrl , Timeout = TimeSpan.FromMinutes 10 }; var request = new { model, stream = false, format = "json", options = new { temperature = 0, num ctx = 16384 }, messages = new { new { role = "system", content = system }, new { role = "user", content = user } } }; // Buffered body with Content-Length some proxies in front of Ollama reject chunked uploads . using var body = new StringContent JsonSerializer.Serialize request , Encoding.UTF8, "application/json" ; var response = await http.PostAsync "/api/chat", body ; response.EnsureSuccessStatusCode ; var chat = await response.Content.ReadFromJsonAsync<OllamaChat ; var content = chat?.Message?.Content ?? "{}"; ReviewResult? result; try { result = JsonSerializer.Deserialize<ReviewResult content, new JsonSerializerOptions { PropertyNameCaseInsensitive = true } ; } catch JsonException { result = null; } // Keep only findings that point at a file in this diff and quote a line that exists in it. var changedFiles = Regex.Matches diff, @"^\+\+\+ b/ .+ $", RegexOptions.Multiline .Select m = m.Groups 1 .Value.Trim .ToHashSet ; var kept = result?.Findings ?? .Where f = changedFiles.Contains f.File ?? "" .Where f = string.IsNullOrWhiteSpace f.Evidence && diff.Contains f.Evidence.Trim .ToList ; var dropped = result?.Findings?.Count ?? 0 - kept.Count; var md = new StringBuilder $" Local first-pass review {model} \n\n" ; if kept.Count == 0 md.AppendLine "0 findings that survived the evidence filter." ; else { md.AppendLine $"{kept.Count} finding s . Advisory only, a human still decides.\n" ; foreach var f in kept.OrderBy f = f.Severity == "important" ? 0 : 1 md.AppendLine $"- {f.Severity} {f.File}:{f.Line} : {f.Claim}\n {f.Evidence .Trim } " ; } if dropped 0 md.AppendLine $"\n {dropped} finding s dropped: no matching file or quoted line in the diff. " ; Console.WriteLine md ; if o.TryGetValue "out", out var outPath File.WriteAllText outPath, md.ToString ; return 0; // advisory: never fails the build}enum Tier { Low, Medium, High }record FileChange string Paths, int Added, int Deleted ;record TierConfig List<string High, List<string Medium, List<string Low, int MediumLines = 300, int HighLines = 800, int MediumFiles = 15, int TestShrinkLines = 20 ;record OllamaChat property: JsonPropertyName "message" OllamaMessage? Message ;record OllamaMessage property: JsonPropertyName "content" string? Content ;record ReviewResult List<Finding ? Findings ;record Finding string? File, int Line, string? Severity, string? Claim, string? Evidence ;static class Glob{ // " " matches across folders, " " within one segment, "?" one character. public static bool IsMatch string glob, string path { var rx = "^" + Regex.Escape glob .Replace @"\ \ /", " . / ?" .Replace @"\ \ ", ". " .Replace @"\ ", " ^/ " .Replace @"\?", " ^/ " + "$"; return Regex.IsMatch path, rx ; }} Run it locally against your branch before CI ever sees it: git diff --numstat origin/main...HEAD numstat.txtdotnet run --project tools/RiskGate -c Release -- classify \ --numstat numstat.txt --config .github/risk-tiers.json The rename handling is there because of a bug I found in my own first version, which I’ll come back to. This is the part I’d tell anyone to get right first. A pull request workflow runs the PR's own copy of the workflow and of any tool it builds. If the gate lives in the repo and the PR can edit it, an agent can lower its own tier, not out of malice, just by "fixing" a failing check. So the gate runs on pull request target, checks out the base commit only, and reads the changed files from the API. It never executes PR code, which is the only safe way to use pull request target. .github/workflows/pr-gate.yml: name: pr-gate Runs the BASE branch version of this workflow and of the RiskGate tool, so a PR agent-written or not cannot lower its own tier by editing the gate. It never checks out or executes PR code.on: pull request target: types: opened, edited, synchronize, reopened, ready for review pull request review: types: submitted, dismissed permissions: contents: read pull-requests: writeconcurrency: group: pr-gate-${{ github.event.pull request.number }} cancel-in-progress: truejobs: gate: runs-on: ubuntu-latest env: GH TOKEN: ${{ github.token }} REPO: ${{ github.repository }} PR: ${{ github.event.pull request.number }} HEAD SHA: ${{ github.event.pull request.head.sha }} steps: - name: Check out the base branch only uses: actions/checkout@v4 with: ref: ${{ github.event.pull request.base.sha }} - uses: actions/setup-dotnet@v4 with: dotnet-version: 8.0.x - name: List changed files as numstat from the API, not from PR code run: | gh api --paginate "repos/$REPO/pulls/$PR/files" \ --jq '. | "\ .additions \t\ .deletions \t\ if .previous filename then "\ .previous filename = \ .filename " else .filename end "' \ numstat.txt cat numstat.txt - name: Classify risk tier id: risk run: | dotnet run --project tools/RiskGate -c Release -- classify \ --numstat numstat.txt --config .github/risk-tiers.json --summary risk.md cat risk.md "$GITHUB STEP SUMMARY" - name: Label the PR env: TIER: ${{ steps.risk.outputs.tier }} run: | for t in low medium high; do gh label create "risk:$t" --repo "$REPO" --force /dev/null done gh pr edit "$PR" --repo "$REPO" \ --remove-label risk:low --remove-label risk:medium --remove-label risk:high gh pr edit "$PR" --repo "$REPO" --add-label "risk:$TIER" - name: Check proof in the PR body env: TIER: ${{ steps.risk.outputs.tier }} PR BODY: ${{ github.event.pull request.body }} run: | printf '%s' "$PR BODY" pr body.md dotnet run --project tools/RiskGate -c Release --no-build -- proof \ --body pr body.md --tier "$TIER" - name: Count human approvals on the current head commit env: TIER: ${{ steps.risk.outputs.tier }} run: | case "$TIER" in low need=0 ;; medium need=1 ;; high need=2 ;; esac Latest non-comment review per human, only if it approved THIS head commit. gh api --paginate "repos/$REPO/pulls/$PR/reviews" \ --jq '. | select .user.type = "Bot" | .user.login, .state, .commit id, .submitted at | @tsv' \ reviews.tsv have=$ sort -t$'\t' -k4 reviews.tsv | awk -F'\t' -v head="$HEAD SHA" ' $2 = "COMMENTED" { state $1 = $2; sha $1 = $3 } END { n = 0; for u in state if state u == "APPROVED" && sha u == head n++; print n }' echo "Tier $TIER needs $need human approval s on $HEAD SHA, has $have." | tee -a "$GITHUB STEP SUMMARY" if "$have" -lt "$need" ; then echo "::error::Not enough human approvals for tier $TIER $have of $need ." exit 1 fi Three details I’d defend in a review of this file: One gap I couldn’t close with Actions alone: pull request review has no target variant, so on review events the workflow file comes from the PR's merge ref. A PR that edits .github/workflows/ could change how its own review events are handled. That's why .github/ is high tier and owned by platform-leads, and why "require review from Code Owners" is on in branch protection. CODEOWNERS is the backstop for the gate's own blind spot. .github/workflows/pr-checks.yml: name: pr-checkson: pull request: types: opened, synchronize, reopened, ready for review permissions: contents: read pull-requests: writeconcurrency: group: pr-checks-${{ github.event.pull request.number }} cancel-in-progress: truejobs: build-test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-dotnet@v4 with: dotnet-version: 8.0.x - run: dotnet restore - run: dotnet build --no-restore -c Release -warnaserror - name: Test CI runs the tests itself; the pasted output in the PR body is not trusted on its own run: dotnet test --no-build -c Release --logger "trx;LogFileName=results.trx" --results-directory TestResults - uses: actions/upload-artifact@v4 if: always with: name: test-results path: TestResults/ Optional first pass with a local model. Advisory only: never a required check. Needs a self-hosted runner labelled "ollama" with Ollama on localhost:11434 and the repo variable OLLAMA REVIEW=true. local-llm-review: if: vars.OLLAMA REVIEW == 'true' && github.event.pull request.head.repo.full name == github.repository runs-on: self-hosted, ollama continue-on-error: true timeout-minutes: 20 env: GH TOKEN: ${{ github.token }} steps: - uses: actions/checkout@v4 with: fetch-depth: 0 - uses: actions/setup-dotnet@v4 with: dotnet-version: 8.0.x - name: Diff against the base run: git diff "${{ github.event.pull request.base.sha }}...${{ github.event.pull request.head.sha }}" pr.diff - name: Review with Ollama run: | dotnet run --project tools/RiskGate -c Release -- review \ --diff pr.diff --review-md REVIEW.md \ --ollama http://localhost:11434 --model qwen2.5-coder:14b --out review.md - name: Post as a PR comment run: gh pr comment "${{ github.event.pull request.number }}" --body-file review.md Why is pasted test output required when CI runs the tests anyway? Because they prove different things. CI proves the tests pass. The pasted output proves the agent ran them before claiming it was done, which is the Lauren Tan point: it changes the agent’s loop, not the merge decision. An agent can paste fake output, which is exactly why CI reruns everything and the pasted text is never trusted on its own. The self-hosted job only runs for branches in the same repo. A self-hosted runner executing fork code is a different article, and not a fun one. On a Linux box or a Linux self-hosted runner, docker-compose.ollama.yml: services: ollama: image: ollama/ollama:latest container name: ollama ports: - "11434:11434" volumes: - ollama:/root/.ollama restart: unless-stoppedvolumes: ollama: docker compose -f docker-compose.ollama.yml up -ddocker exec ollama ollama pull qwen2.5-coder:14b On a Mac, skip Docker. Docker on macOS can’t use the Apple GPU, so the model runs on CPU and a review takes minutes instead of seconds. Run Ollama natively: brew install ollamaollama serve &ollama pull qwen2.5-coder:14b Then, from the repo: git diff origin/main...HEAD pr.diffdotnet run --project tools/RiskGate -c Release -- review \ --diff pr.diff --review-md REVIEW.md --model qwen2.5-coder:14b REVIEW.md is the per-repo rules file the Anthropic post recommends. Mine is short on purpose: Review rules for orders-demoFailure here looks like: a user sees another tenant's orders, a total is off by a cent,a token is accepted after it should be rejected, a migration loses data.- Every query over Orders must keep the TenantId filter. Flag any change that moves or removes it.- Money is decimal. Rounding must state MidpointRounding explicitly.- Do not comment on naming, formatting or test style.- Skip: docs/, .Designer.cs, ModelSnapshot.cs- Give file and line, and quote the changed line. If nothing is wrong, say "0 findings". None of the above blocks anything until branch protection says so. On main: +--------------------------------------------------+---------------------------+| Setting | Value |+--------------------------------------------------+---------------------------+| Require a pull request before merging | on || Required approvals repo-wide | 0 tiers enforce counts || Dismiss stale approvals when new commits pushed | on || Require review from Code Owners | on || Required status checks | gate, build-test || Require branches to be up to date | on || Allow bypass | nobody, admins included |+--------------------------------------------------+---------------------------+ “Require review from Code Owners” with zero repo-wide approvals sounds contradictory, so to be precise: code owner review applies when the PR touches an owned path, and /docs/ is owned by @orders-demo/everyone. For a docs-only PR that means any team member's approval satisfies CODEOWNERS. If you want truly zero-human merges for low tier, give the low paths no owner. I didn't, because for a first experiment I wanted a human to at least glance at the proof. Ten PRs against the demo orders API, all written by Claude Code from a one-paragraph task each. I directed the agent and was the reviewer. These are synthetic in the sense that the repo is a demo and the tasks were chosen by me, but the diffs, the test runs and the review rounds were real. The tiers below are exactly what riskgate classify printed for each PR's numstat. +----+------------------------------+--------+-------+-------+------------------------------------------+| | Task | Tier | Files | Lines | Why from classifier output |+----+------------------------------+--------+-------+-------+------------------------------------------+| 01 | Local setup docs | low | 2 | 53 | only low-risk paths || 02 | Discount edge-case tests | low | 1 | 88 | only low-risk paths || 03 | XML doc comment typos | medium | 2 | 26 | src/ .cs files are medium by path || 04 | Flagged bulk-quote endpoint | low | 3 | 137 | Experimental/ + tests only || 05 | Pagination on GET /orders | medium | 3 | 90 | Controllers/, Queries/ || 06 | Discount rounding to 2 dp | medium | 2 | 34 | Domain/Pricing/ || 07 | Refactor shared order fetch | medium | 3 | 281 | Queries/, Export/, Discounts/ || 08 | Request logging middleware | medium | 3 | 72 | appsettings.json, Logging/ || 09 | Index on Orders.Status | high | 4 | 264 | Migrations/ || 10 | Fix flaky token expiry test | high | 2 | 15 | Auth/JwtSetup.cs |+----+------------------------------+--------+-------+-------+------------------------------------------+ Three low, five medium, two high. Wall-clock time from opening the PR page to clicking approve or request changes, per pass, with a phone stopwatch. Paused for interruptions. It includes reading the risk summary, the proof, the local reviewer’s comment, and whatever part of the diff I decided to read. It does not include the agent’s own time or CI time, since those cost compute, not attention. For the two high-tier PRs I needed two approvals. I did both myself from two accounts, a day apart. I’m telling you that up front because it’s the weakest number in this post: a second pass by the same person who wrote the task is not a second reviewer. +--------+-----+------------------+--------------------+-----------------+| Tier | PRs | Human minutes | Minutes per PR | Escaped defects || | | total | range | |+--------+-----+------------------+--------------------+-----------------+| low | 3 | 8 | 1 - 4 | 0 || medium | 5 | 43 | 2 - 14 | 1 || high | 2 | 71 two passes | 30 - 41 | 0 |+--------+-----+------------------+--------------------+-----------------+| total | 10 | 122 | | 1 |+--------+-----+------------------+--------------------+-----------------+ An escaped defect here means a bug that passed every gate and got merged, and that I found afterwards. The shape is what I expected and the size isn’t. I expected high-tier PRs to cost more. I didn’t expect the two high PRs to cost more than all eight others combined. Most of that time went into PR 09, the migration, where I generated the SQL with dotnet ef migrations script and read that instead of the C . That was the right call and also the slowest thing I did all week. Three real bugs, all in PRs where the tests were green: Four false flags across the ten PRs: Separately, the evidence filter dropped 5 findings across the run because the quoted line didn’t match the diff. I read all five. Four were rightly dropped wrong file, invented code . One was a real point about PR 07 where the model quoted the line with the leading + from the diff, so my exact-match filter threw it away. The filter is blunt and I haven't decided whether to loosen it. PR 07 is the escaped defect, and it is almost exactly Wasowski’s constructed example. The task was “remove duplication between the export query and the discount query.” The agent moved the shared part into one method, and the TenantId filter ended up applied by the discount handler's caller instead of in the shared method. The export path called the shared method directly. All tests passed, because the export tests ran with a single tenant. The local reviewer said nothing. I spent 9 minutes on it, read the refactored method, saw clean code, and approved. I found it two days later while writing PR 08’s recorded run: I hit the export endpoint as a second test tenant and got the first tenant’s rows. It never left the demo repo, but in a real system that’s an incident, not a bug. What would have caught it? Not a smarter reviewer, I think. A preservation contract would have: one line in the task saying “every query over Orders keeps its tenant filter, export included,” plus a test that asserts it with two tenants. I’ve since added that line to REVIEW.md you saw it above and the two-tenant test to the suite. The reviewer would now have a rule to check against. Whether a 14B model would apply it to a refactor that moves rather than removes the filter, I don't know yet. The proof check bounced two PRs on the first try, and the messages were exactly what the tool prints: PR 06, first body "All 57 tests pass locally." :::error:: Testing needs pasted dotnet test output with Failed: 0, Passed: N N 0 PR 10, first body Recorded run left as "n/a, tests cover it" :::error::high tier needs a recorded run: an HTTP transcript curl -i output or a link to a recording/screenshot The second one is my favorite result of the whole experiment. The recorded run for PR 10 is a curl -i with a token that expired three minutes earlier, and a 401 with invalid token in the response. That transcript would have looked very different on the 30-minute-skew version. And the classifier caught a bug in itself. I added an eleventh fixture that renames src/Orders.Api/Auth/Login.cs to src/Orders.Web/Auth/Login.cs. My first version only looked at the new path, so a file moving out of Auth/ came back medium. An agent "reorganizing folders" could have walked auth code out of the high tier. The version above judges both paths, and that fixture now comes back high. I believe the tier should be set by the change and not by who wrote it, and that it should be enforced by a check that the change itself cannot edit. Most of the safety in this setup is in boring places: pull request target with a base checkout, the head-SHA check, the bot filter, CODEOWNERS on .github/. I believe requiring proof changed the agent’s behavior more than it changed mine. After PR 06 bounced once, the agent pasted test output on every later PR without being asked again, because the template and the failing check were in its context. I believe a local 14B model is worth running as a first pass, but only as advisory. Three real catches against four false flags in ten PRs is useful. It is not a ratio I’d let block a merge. I don’t know whether low tier is safe at zero humans. Three low PRs and zero escapes says nothing. The Anthropic post says they let low-risk changes merge on automated checks and that the line “keeps moving”. I’d want months of revert data before moving mine. I don’t know what my baseline is. I didn’t time myself reading these ten PRs line by line first, so I can’t tell you this saved anything. I can only tell you where the 122 minutes went. Next round, I’ll time a comparable batch the old way before switching. I don’t know if the one escape is a medium-tier problem or a refactoring problem. A 281-line refactor across three projects sits just under my 300-line threshold. Maybe “moves code between projects” should bump the tier on its own. Maybe the threshold is just wrong. Ten PRs can’t tell me. Start smaller than I did: And read the primary source. Not the rewrites. Sources Tags: ai-agents, code-review, dotnet, csharp, github-actions, ollama, devops, software-engineering, llm, claude-code Measuring Review in Human Minutes: A Risk-Tier Gate for Agent Pull Requests in .NET https://pub.towardsai.net/measuring-review-in-human-minutes-a-risk-tier-gate-for-agent-pull-requests-in-net-c13f301342fa was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.