For most of this year my review habit has been the same as it was in 2019: open the PR, read the diff top to bottom, approve. That habit quietly broke once agents started opening PRs for me. Not because the code got worse, but because there was more of it than I had attention for, and I noticed I was approving the boring ones faster and faster. Speed-reading a diff is not review. It is a ritual that makes me feel responsible.
So I set myself a narrower question: if I stop reading every line, what replaces it, and how many human minutes does each kind of change really need? I wanted a number, not a vibe.
This post is the answer I got from a small experiment on a .NET repo. It is ten PRs. Keep that in mind the whole way through.
The post that pushed me into this was Anthropic’s “How the Claude Code team does code review” by Addy Osmani, Morgan Westlee Lunt and Qing Wang, published October 15, 2026 (the URL slug is solving-code-review-at-anthropic):
https://claude.dev/blog/solving-code-review-at-anthropic/
I first met it secondhand. Two rewrites by the JIN System Architect publication showed up in my weekly digest (items 124 and 154). Both retell the same post, add little beyond it, and both had the tells that make me stop trusting a rewrite: a garbled disclosure paragraph, and in one of them the phrase “The material reports” left in the body, which reads like a summarizer’s scaffolding that nobody removed. I did not take a single number from them. I went to the original.
Here are the figures from the original, with the scope each one actually has. All of them are Anthropic’s own internal numbers. I can’t verify any of them, and the post doesn’t publish denominators.
+-----------------------------------------------+----------------------+-------------------------------+| Metric (Anthropic-reported) | Before -> After | Scope |+-----------------------------------------------+----------------------+-------------------------------+| Changes landing per day | ~2,500 -> 30,000+ | Anthropic monorepo, May->Sep || Merged PRs | more than doubled | Claude Code repo, Jun->Sep || Share of merged PRs reverted | down >40% (relative) | Claude Code repo, Jun->Sep || Internal PRs with substantive review comments | 16% -> 54% | at Code Review preview launch || Pass after one review-and-fix round | 65% -> 78% | internal PRs, convergence fix || Need five or more rounds | 8.9% -> 1.5% | internal PRs, convergence fix || Mean reviews to pass | 2.08 -> 1.32 | internal PRs, convergence fix |+-----------------------------------------------+----------------------+-------------------------------+
The scope column matters. The monorepo number is about Anthropic’s monorepo, which the post says is separate from the Claude Code repo. The revert figure is about the Claude Code codebase. The review-comment and convergence figures are about internal PRs. It is easy to blur these into one “agents made review better” headline, and the post itself doesn’t do that.
The numbers were not what changed my mind, though. These ideas were:
None of this needs Anthropic’s tooling. That was the point of trying it on my own repo.
My digest that week had a cluster of related pieces. I’m separating what I read in a primary source from what I only saw described, and what I actually tried.
Uber, four narrow agents. The newsletter description was that Uber split review into four narrow agents instead of one generalist. I did not verify the “four”. What I did read is Uber’s own uReview post, which describes a multi-stage pipeline that decides which files are eligible, generates comments, then validates them before posting, and it names false positives as the main challenge: hallucinated comments, and comments that are technically valid but don’t matter in context. That second category is the one I underestimated. Tried: no, it’s internal to Uber. Borrowed: the idea that the filter after generation matters as much as the generator.
Lauren Tan’s verification skill. As described in write-ups of her talk, the idea is that the agent gets a project-local skill that launches the real app, drives it like a user, and captures evidence (screenshots, transcripts, logs) so the PR arrives with proof instead of “Done.” The PR counts quoted for her talk differ between sources (one write-up says about 2,000 a month, a post title elsewhere says 2,500), so I’m not repeating either as fact. Tried: the idea, not her code. My gate refuses medium and high PRs that don’t carry pasted test output, and refuses high PRs without a recorded run.
kunchenguid’s no-mistakes. A local git proxy: you git push no-mistakes instead of origin, it runs a review, test, docs and lint pipeline in a disposable worktree, and only forwards the branch and opens the PR once everything is green. The README install is a curl | sh line:
I read the script before running it, and I’d ask you to do the same with any installer, including mine. Tried: yes, on the demo repo. It does what it says. I kept my gate on the server side anyway, for one reason: a pre-push gate only runs if the pusher goes through it, and an agent with push rights can push to origin directly. A pre-push gate is a nice first filter. It is not the gate.
The Code’s Saturday special: CODEOWNERS plus a PR template. The newsletter described pairing CODEOWNERS with a PR template so ownership and evidence are both structural instead of social. Tried: yes, it’s the base of everything below. One thing the newsletter version skipped: CODEOWNERS decides who must review, it cannot say how many. Branch protection’s approval count is repo-wide. So “two people for auth” needs a check of its own.
Jaroslaw Wasowski on brownfield specs. His piece on a “preservation contract” argues that a change spec for legacy code should start with what must not change. His opening example (which he says is constructed for the article) is an agent “cleaning up” a shared fetch function and dropping a visibility filter that the export path silently depended on. Keep that example in mind for section 5. His companion piece on missing spec decisions, with a two-implementations technique and a decision register, is on my list; I haven’t worked through it, so I won’t summarize it.
The repo is a demo ASP.NET Core 8 orders API with EF Core: Orders.Api, Orders.Application, Orders.Domain, Orders.Infrastructure, and tests. The org and team names below (orders-demo, orders-team, platform-leads) are that demo repo's real names, so the files work as-is if you mirror the layout.
Here’s the shape of it:
agent pushes branch | v+------------------+ +-----------------------------+| pr-checks.yml | | pr-gate.yml || (pull_request) | | (pull_request_target + || | | pull_request_review) || build, test, | | base-branch code only: || upload .trx | | 1. classify tier || | | 2. label risk:low/med/high || optional: | | 3. check proof in body || local Ollama | | 4. count human approvals || first pass | | on the head commit || (advisory) | | 0 / 1 / 2 by tier |+------------------+ +-----------------------------+ | | +--------------+--------------+ v branch protection: both required, CODEOWNERS review required, stale approvals dismissed on push
I wrote the rules down before writing code, because the post’s advice to “start strict and loosen with evidence” only works if the starting point is explicit.
+--------+--------------------------------------------+----------------------+------------------+| Tier | Triggered by | Human approvals | Proof required |+--------+--------------------------------------------+----------------------+------------------+| low | only docs/, *.md, tests/, Experimental/ | 0 (CI + proof green) | template filled || | and under 300 lines / 15 files | | || medium | any other src/ or appsettings*.json, | 1 | dotnet test || | unknown paths, >300 lines or >15 files, | | output pasted || | or tests shrank by >20 lines | | || high | Auth/, Migrations/, Program.cs, prod | 2 | test output + || | settings, package versions, .github/, | | recorded run || | the gate tool itself, or >800 lines | | |+--------+--------------------------------------------+----------------------+------------------+
Two rules I added after thinking about how agents fail rather than how humans fail. Unknown paths are never low, because a new top-level folder is exactly where an agent puts something you didn’t expect. And if tests shrink, the PR is at least medium, because “fix the red build” sometimes gets solved by deleting the red.
.github/risk-tiers.json:
{ "high": [ "src/Orders.Api/Auth/**", "src/Orders.Infrastructure/Persistence/Migrations/**", "src/Orders.Api/Program.cs", "**/appsettings.Production.json", "Directory.Packages.props", ".github/workflows/**", ".github/CODEOWNERS", ".github/risk-tiers.json", "tools/RiskGate/**" ], "medium": [ "src/**", "**/appsettings*.json", "*.sln", "**/*.csproj" ], "low": [ "docs/**", "**/*.md", "tests/**", "src/Orders.Api/Experimental/**" ], "mediumLines": 300, "highLines": 800, "mediumFiles": 15, "testShrinkLines": 20}
Experimental/ is low because everything in it is behind a feature flag and auto-mapped by a startup hook that was itself reviewed as high. If your flagged code isn't isolated like that, don't copy this line.
.github/CODEOWNERS:
A subtlety: /src/Orders.Api/Experimental/ comes before /src/, so /src/ wins for experimental files and the orders team owns them. That's what I want for ownership. The count still comes from the tier, which says zero for experimental-only PRs. Ownership and approval count are different questions, which is exactly why the count can't live in CODEOWNERS.
.github/pull_request_template.md:
## What and why<!-- One paragraph. Link the issue. If an agent wrote this, name the agent and who directed it. -->## Testing<!-- Paste the real output, not a summary. Medium and high tiers need the dotnet test summary line, e.g.The checker strips HTML comments before deciding a section is filled, so an agent can’t pass by leaving the hints in place.
One console app, three subcommands: classify reads a numstat diff and prints a tier, proof checks the PR body against the tier, and review asks a local Ollama model for a first pass. No NuGet packages, just the BCL, so it restores fast in CI.
tools/RiskGate/RiskGate.csproj:
<Project Sdk="Microsoft.NET.Sdk"> <PropertyGroup> <OutputType>Exe</OutputType> <TargetFramework>net8.0</TargetFramework> <Nullable>enable</Nullable> <ImplicitUsings>enable</ImplicitUsings> <AssemblyName>riskgate</AssemblyName> </PropertyGroup></Project>
tools/RiskGate/Program.cs:
using System.Net.Http.Json;using System.Text;using System.Text.Json;using System.Text.Json.Serialization;using System.Text.RegularExpressions;// riskgate: labels an agent PR low/medium/high, checks the proof in the PR body,// and optionally asks a local Ollama model for a first-pass review.//// riskgate classify --numstat numstat.txt --config .github/risk-tiers.json [--summary out.md]// riskgate proof --body pr_body.md --tier medium// riskgate review --diff pr.diff [--review-md REVIEW.md] [--ollama http://localhost:11434] [--model qwen2.5-coder:14b] [--out review.md]var cmd = args.FirstOrDefault() ?? "";var opt = ParseOptions(args.Skip(1).ToArray());return cmd switch{ "classify" => Classify(opt), "proof" => Proof(opt), "review" => await Review(opt), _ => Usage()};static int Usage(){ Console.Error.WriteLine("usage: riskgate classify|proof|review [options]"); return 2;}static Dictionary<string, string> ParseOptions(string[] a){ var d = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase); for (var i = 0; i < a.Length - 1; i += 2) d[a[i].TrimStart('-')] = a[i + 1]; return d;}static string Get(Dictionary<string, string> o, string key, string? fallback = null) => o.TryGetValue(key, out var v) ? v : fallback ?? throw new ArgumentException($"missing --{key}");// ---------------------------------------------------------------- classifystatic int Classify(Dictionary<string, string> o){ var config = JsonSerializer.Deserialize<TierConfig>(File.ReadAllText(Get(o, "config")), new JsonSerializerOptions { PropertyNameCaseInsensitive = true })!; var files = File.ReadAllLines(Get(o, "numstat")) .Where(l => !string.IsNullOrWhiteSpace(l)) .Select(ParseNumstat) .ToList(); var reasons = new List<string>(); var tier = Tier.Low; // A rename is judged by BOTH its old and new path, so moving a file out of Auth/ is still high. foreach (var f in files) foreach (var p in f.Paths) { var fileTier = config.High.Any(g => Glob.IsMatch(g, p)) ? Tier.High : config.Low.Any(g => Glob.IsMatch(g, p)) ? Tier.Low : config.Medium.Any(g => Glob.IsMatch(g, p)) ? Tier.Medium : Tier.Medium; // unknown paths are never low if (fileTier > Tier.Low) reasons.Add($"{p} is {fileTier.ToString().ToLower()} by path"); tier = Max(tier, fileTier); } var lines = files.Sum(f => f.Added + f.Deleted); if (lines > config.HighLines) { reasons.Add($"{lines} changed lines (> {config.HighLines})"); tier = Tier.High; } else if (lines > config.MediumLines || files.Count > config.MediumFiles) { reasons.Add($"{lines} changed lines in {files.Count} files (medium threshold {config.MediumLines} lines / {config.MediumFiles} files)"); tier = Max(tier, Tier.Medium); } // Agents sometimes "fix" a red build by deleting or weakening tests. var testAdded = files.Where(f => IsTest(f.Paths[^1])).Sum(f => f.Added); var testDeleted = files.Where(f => IsTest(f.Paths[^1])).Sum(f => f.Deleted); if (testDeleted - testAdded > config.TestShrinkLines) { reasons.Add($"tests shrank by {testDeleted - testAdded} lines"); tier = Max(tier, Tier.Medium); } var name = tier.ToString().ToLower(); var summary = new StringBuilder($"### Risk tier: {name}\n\n"); summary.AppendLine($"{files.Count} files, {lines} changed lines.\n"); foreach (var r in reasons.Distinct().Take(15)) summary.AppendLine($"- {r}"); if (reasons.Count == 0) summary.AppendLine("- only low-risk paths, under size thresholds"); Console.WriteLine(summary); if (o.TryGetValue("summary", out var path)) File.WriteAllText(path, summary.ToString()); var ghOut = Environment.GetEnvironmentVariable("GITHUB_OUTPUT"); if (!string.IsNullOrEmpty(ghOut)) File.AppendAllText(ghOut, $"tier={name}\n"); return 0; static bool IsTest(string p) => p.StartsWith("tests/") || p.Contains(".Tests/");}static FileChange ParseNumstat(string line){ var parts = line.Split('\t'); var added = int.TryParse(parts[0], out var a) ? a : 0; // "-" for binary files var deleted = int.TryParse(parts[1], out var d) ? d : 0; var path = parts[2]; // Renames: "src/{Old => New}/File.cs" or "old/path.cs => new/path.cs". Keep both paths. var brace = Regex.Match(path, @"{(.) => (.)}"); if (brace.Success) return new FileChange( [Clean(path.Replace(brace.Value, brace.Groups[1].Value)), Clean(path.Replace(brace.Value, brace.Groups[2].Value))], added, deleted); if (path.Contains(" => ")) return new FileChange([path[..path.IndexOf(" => ")], path[(path.IndexOf(" => ") + 4)..]], added, deleted); return new FileChange([path], added, deleted); static string Clean(string p) => p.Replace("//", "/");}static Tier Max(Tier a, Tier b) => a > b ? a : b;// ---------------------------------------------------------------- proofstatic int Proof(Dictionary<string, string> o){ var body = File.ReadAllText(Get(o, "body")).Replace("\r\n", "\n"); var tier = Enum.Parse<Tier>(Get(o, "tier"), ignoreCase: true); var problems = new List<string>(); foreach (var section in new[] { "Testing", "Risk", "How to revert" }) { var text = Section(body, section); if (string.IsNullOrWhiteSpace(text)) problems.Add($"## {section} is missing or empty"); } var testing = Section(body, "Testing") ?? ""; var testRun = Regex.IsMatch(testing, @"Failed:\s+0,\s+Passed:\s+[1-9]\d*"); if (tier >= Tier.Medium && !testRun) problems.Add("## Testing needs pasted dotnet test output with Failed: 0, Passed: N (N > 0)"); if (tier == Tier.High) { var recorded = Regex.IsMatch(body, @"HTTP/(1.1|2)\s+\d{3}") || Regex.IsMatch(body, @"https?://\S+.(gif|mp4|webm|cast|png)\b", RegexOptions.IgnoreCase); if (!recorded) problems.Add("high tier needs a recorded run: an HTTP transcript (curl -i output) or a link to a recording/screenshot"); } if (problems.Count == 0) { Console.WriteLine($"proof ok for {tier.ToString().ToLower()} tier"); return 0; } foreach (var p in problems) Console.WriteLine($"::error::{p}"); return 1;}// Text under "## Name" up to the next "## ", with HTML comments (template hints) removed.static string? Section(string body, string name){ var m = Regex.Match(body, $@"^##\s+{Regex.Escape(name)}\s*$(.?)(?=^##\s|\z)", RegexOptions.Multiline | RegexOptions.Singleline | RegexOptions.IgnoreCase); if (!m.Success) return null; return Regex.Replace(m.Groups[1].Value, @"<!--.?-->", "", RegexOptions.Singleline).Trim();}// ---------------------------------------------------------------- review (Ollama)static async Task<int> Review(Dictionary<string, string> o){ var diff = File.ReadAllText(Get(o, "diff")); var reviewMd = o.TryGetValue("review-md", out var rp) && File.Exists(rp) ? File.ReadAllText(rp) : ""; var baseUrl = Get(o, "ollama", "http://localhost:11434").TrimEnd('/'); var model = Get(o, "model", "qwen2.5-coder:14b"); const int maxDiffChars = 40_000; if (diff.Length > maxDiffChars) diff = diff[..maxDiffChars] + "\n[diff truncated]"; var system = """ You review a pull request diff written by a coding agent. Report only bugs that would change runtime behavior: wrong results, missing authorization or tenant filters, data loss, security weakening, broken error handling. No style, naming or formatting comments. Every finding must quote the exact changed line as evidence and give its file and line. If nothing is wrong, return an empty list. Answer with JSON only: {"findings":[{"file":"","line":0,"severity":"important|nit","claim":"","evidence":""}]} """; var user = (reviewMd.Length > 0 ? $"Repository review rules:\n{reviewMd}\n\n" : "") + $"Diff:\n{diff}"; using var http = new HttpClient { BaseAddress = new Uri(baseUrl), Timeout = TimeSpan.FromMinutes(10) }; var request = new { model, stream = false, format = "json", options = new { temperature = 0, num_ctx = 16384 }, messages = new[] { new { role = "system", content = system }, new { role = "user", content = user } } }; // Buffered body with Content-Length (some proxies in front of Ollama reject chunked uploads). using var body = new StringContent(JsonSerializer.Serialize(request), Encoding.UTF8, "application/json"); var response = await http.PostAsync("/api/chat", body); response.EnsureSuccessStatusCode(); var chat = await response.Content.ReadFromJsonAsync<OllamaChat>(); var content = chat?.Message?.Content ?? "{}"; ReviewResult? result; try { result = JsonSerializer.Deserialize<ReviewResult>(content, new JsonSerializerOptions { PropertyNameCaseInsensitive = true }); } catch (JsonException) { result = null; } // Keep only findings that point at a file in this diff and quote a line that exists in it. var changedFiles = Regex.Matches(diff, @"^+++ b/(.+)$", RegexOptions.Multiline).Select(m => m.Groups[1].Value.Trim()).ToHashSet(); var kept = (result?.Findings ?? []) .Where(f => changedFiles.Contains(f.File ?? "")) .Where(f => !string.IsNullOrWhiteSpace(f.Evidence) && diff.Contains(f.Evidence.Trim())) .ToList(); var dropped = (result?.Findings?.Count ?? 0) - kept.Count; var md = new StringBuilder($"### Local first-pass review ({model})\n\n"); if (kept.Count == 0) md.AppendLine("0 findings that survived the evidence filter."); else { md.AppendLine($"{kept.Count} finding(s). Advisory only, a human still decides.\n"); foreach (var f in kept.OrderBy(f => f.Severity == "important" ? 0 : 1)) md.AppendLine($"- {f.Severity} {f.File}:{f.Line}: {f.Claim}\n > {f.Evidence!.Trim()}"); } if (dropped > 0) md.AppendLine($"\n_{dropped} finding(s) dropped: no matching file or quoted line in the diff._"); Console.WriteLine(md); if (o.TryGetValue("out", out var outPath)) File.WriteAllText(outPath, md.ToString()); return 0; // advisory: never fails the build}enum Tier { Low, Medium, High }record FileChange(string[] Paths, int Added, int Deleted);record TierConfig( List<string> High, List<string> Medium, List<string> Low, int MediumLines = 300, int HighLines = 800, int MediumFiles = 15, int TestShrinkLines = 20);record OllamaChat([property: JsonPropertyName("message")] OllamaMessage? Message);record OllamaMessage([property: JsonPropertyName("content")] string? Content);record ReviewResult(List<Finding>? Findings);record Finding(string? File, int Line, string? Severity, string? Claim, string? Evidence);static class Glob{ // "**" matches across folders, "" within one segment, "?" one character. public static bool IsMatch(string glob, string path) { var rx = "^" + Regex.Escape(glob) .Replace(@"**/", "(./)?") .Replace(@"**", ".") .Replace(@"*", "[^/]") .Replace(@"?", "[^/]") + "$"; return Regex.IsMatch(path, rx); }}
Run it locally against your branch before CI ever sees it:
git diff --numstat origin/main...HEAD > numstat.txtdotnet run --project tools/RiskGate -c Release -- classify \ --numstat numstat.txt --config .github/risk-tiers.json
The rename handling is there because of a bug I found in my own first version, which I’ll come back to.
This is the part I’d tell anyone to get right first. A pull_request workflow runs the PR's own copy of the workflow and of any tool it builds. If the gate lives in the repo and the PR can edit it, an agent can lower its own tier, not out of malice, just by "fixing" a failing check. So the gate runs on pull_request_target, checks out the **base** commit only, and reads the changed files from the API. It never executes PR code, which is the only safe way to use pull_request_target.
.github/workflows/pr-gate.yml:
name: pr-gate# Runs the BASE branch version of this workflow and of the RiskGate tool, so a PR# (agent-written or not) cannot lower its own tier by editing the gate.# It never checks out or executes PR code.on: pull_request_target: types: [opened, edited, synchronize, reopened, ready_for_review] pull_request_review: types: [submitted, dismissed]permissions: contents: read pull-requests: writeconcurrency: group: pr-gate-${{ github.event.pull_request.number }} cancel-in-progress: truejobs: gate: runs-on: ubuntu-latest env: GH_TOKEN: ${{ github.token }} REPO: ${{ github.repository }} PR: ${{ github.event.pull_request.number }} HEAD_SHA: ${{ github.event.pull_request.head.sha }} steps: - name: Check out the base branch only uses: actions/checkout@v4 with: ref: ${{ github.event.pull_request.base.sha }} - uses: actions/setup-dotnet@v4 with: dotnet-version: 8.0.x - name: List changed files as numstat (from the API, not from PR code) run: | gh api --paginate "repos/$REPO/pulls/$PR/files" \ --jq '.[] | "(.additions)\t(.deletions)\t(if .previous_filename then "(.previous_filename) => (.filename)" else .filename end)"' \ > numstat.txt cat numstat.txt - name: Classify risk tier id: risk run: | dotnet run --project tools/RiskGate -c Release -- classify \ --numstat numstat.txt --config .github/risk-tiers.json --summary risk.md cat risk.md >> "$GITHUB_STEP_SUMMARY" - name: Label the PR env: TIER: ${{ steps.risk.outputs.tier }} run: | for t in low medium high; do gh label create "risk:$t" --repo "$REPO" --force >/dev/null done gh pr edit "$PR" --repo "$REPO" \ --remove-label risk:low --remove-label risk:medium --remove-label risk:high gh pr edit "$PR" --repo "$REPO" --add-label "risk:$TIER" - name: Check proof in the PR body env: TIER: ${{ steps.risk.outputs.tier }} PR_BODY: ${{ github.event.pull_request.body }} run: | printf '%s' "$PR_BODY" > pr_body.md dotnet run --project tools/RiskGate -c Release --no-build -- proof \ --body pr_body.md --tier "$TIER" - name: Count human approvals on the current head commit env: TIER: ${{ steps.risk.outputs.tier }} run: | case "$TIER" in low) need=0 ;; medium) need=1 ;; high) need=2 ;; esac # Latest non-comment review per human, only if it approved THIS head commit. gh api --paginate "repos/$REPO/pulls/$PR/reviews" \ --jq '.[] | select(.user.type != "Bot") | [.user.login, .state, .commit_id, .submitted_at] | @tsv' \ > reviews.tsv have=$(sort -t$'\t' -k4 reviews.tsv | awk -F'\t' -v head="$HEAD_SHA" ' $2 != "COMMENTED" { state[$1] = $2; sha[$1] = $3 } END { n = 0; for (u in state) if (state[u] == "APPROVED" && sha[u] == head) n++; print n }') echo "Tier $TIER needs $need human approval(s) on $HEAD_SHA, has $have." | tee -a "$GITHUB_STEP_SUMMARY" if [ "$have" -lt "$need" ]; then echo "::error::Not enough human approvals for tier $TIER ($have of $need)." exit 1 fi
Three details I’d defend in a review of this file:
One gap I couldn’t close with Actions alone: pull_request_review has no _target variant, so on review events the workflow file comes from the PR's merge ref. A PR that edits .github/workflows/ could change how its own review events are handled. That's why .github/ is high tier and owned by platform-leads, and why "require review from Code Owners" is on in branch protection. CODEOWNERS is the backstop for the gate's own blind spot.
.github/workflows/pr-checks.yml:
name: pr-checkson: pull_request: types: [opened, synchronize, reopened, ready_for_review]permissions: contents: read pull-requests: writeconcurrency: group: pr-checks-${{ github.event.pull_request.number }} cancel-in-progress: truejobs: build-test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-dotnet@v4 with: dotnet-version: 8.0.x - run: dotnet restore - run: dotnet build --no-restore -c Release -warnaserror - name: Test (CI runs the tests itself; the pasted output in the PR body is not trusted on its own) run: dotnet test --no-build -c Release --logger "trx;LogFileName=results.trx" --results-directory TestResults - uses: actions/upload-artifact@v4 if: always() with: name: test-results path: TestResults/ # Optional first pass with a local model. Advisory only: never a required check. # Needs a self-hosted runner labelled "ollama" with Ollama on localhost:11434 # and the repo variable OLLAMA_REVIEW=true. local-llm-review: if: vars.OLLAMA_REVIEW == 'true' && github.event.pull_request.head.repo.full_name == github.repository runs-on: [self-hosted, ollama] continue-on-error: true timeout-minutes: 20 env: GH_TOKEN: ${{ github.token }} steps: - uses: actions/checkout@v4 with: fetch-depth: 0 - uses: actions/setup-dotnet@v4 with: dotnet-version: 8.0.x - name: Diff against the base run: git diff "${{ github.event.pull_request.base.sha }}...${{ github.event.pull_request.head.sha }}" > pr.diff - name: Review with Ollama run: | dotnet run --project tools/RiskGate -c Release -- review \ --diff pr.diff --review-md REVIEW.md \ --ollama http://localhost:11434 --model qwen2.5-coder:14b --out review.md - name: Post as a PR comment run: gh pr comment "${{ github.event.pull_request.number }}" --body-file review.md
Why is pasted test output required when CI runs the tests anyway? Because they prove different things. CI proves the tests pass. The pasted output proves the agent ran them before claiming it was done, which is the Lauren Tan point: it changes the agent’s loop, not the merge decision. An agent can paste fake output, which is exactly why CI reruns everything and the pasted text is never trusted on its own.
The self-hosted job only runs for branches in the same repo. A self-hosted runner executing fork code is a different article, and not a fun one.
On a Linux box or a Linux self-hosted runner, docker-compose.ollama.yml:
services: ollama: image: ollama/ollama:latest container_name: ollama ports: - "11434:11434" volumes: - ollama:/root/.ollama restart: unless-stoppedvolumes: ollama: docker compose -f docker-compose.ollama.yml up -ddocker exec ollama ollama pull qwen2.5-coder:14b
On a Mac, skip Docker. Docker on macOS can’t use the Apple GPU, so the model runs on CPU and a review takes minutes instead of seconds. Run Ollama natively:
brew install ollamaollama serve &ollama pull qwen2.5-coder:14b
Then, from the repo:
git diff origin/main...HEAD > pr.diffdotnet run --project tools/RiskGate -c Release -- review \ --diff pr.diff --review-md REVIEW.md --model qwen2.5-coder:14b
REVIEW.md is the per-repo rules file the Anthropic post recommends. Mine is short on purpose:
None of the above blocks anything until branch protection says so. On main:
+--------------------------------------------------+---------------------------+| Setting | Value |+--------------------------------------------------+---------------------------+| Require a pull request before merging | on || Required approvals (repo-wide) | 0 (tiers enforce counts) || Dismiss stale approvals when new commits pushed | on || Require review from Code Owners | on || Required status checks | gate, build-test || Require branches to be up to date | on || Allow bypass | nobody, admins included |+--------------------------------------------------+---------------------------+
“Require review from Code Owners” with zero repo-wide approvals sounds contradictory, so to be precise: code owner review applies when the PR touches an owned path, and /docs/ is owned by @orders-demo/everyone. For a docs-only PR that means any team member's approval satisfies CODEOWNERS. If you want truly zero-human merges for low tier, give the low paths no owner. I didn't, because for a first experiment I wanted a human to at least glance at the proof.
Ten PRs against the demo orders API, all written by Claude Code from a one-paragraph task each. I directed the agent and was the reviewer. These are synthetic in the sense that the repo is a demo and the tasks were chosen by me, but the diffs, the test runs and the review rounds were real.
The tiers below are exactly what riskgate classify printed for each PR's numstat.
+----+------------------------------+--------+-------+-------+------------------------------------------+| # | Task | Tier | Files | Lines | Why (from classifier output) |+----+------------------------------+--------+-------+-------+------------------------------------------+| 01 | Local setup docs | low | 2 | 53 | only low-risk paths || 02 | Discount edge-case tests | low | 1 | 88 | only low-risk paths || 03 | XML doc comment typos | medium | 2 | 26 | src/ .cs files are medium by path || 04 | Flagged bulk-quote endpoint | low | 3 | 137 | Experimental/ + tests only || 05 | Pagination on GET /orders | medium | 3 | 90 | Controllers/, Queries/ || 06 | Discount rounding to 2 dp | medium | 2 | 34 | Domain/Pricing/ || 07 | Refactor shared order fetch | medium | 3 | 281 | Queries/, Export/, Discounts/ || 08 | Request logging middleware | medium | 3 | 72 | appsettings.json, Logging/ || 09 | Index on Orders.Status | high | 4 | 264 | Migrations/ || 10 | Fix flaky token expiry test | high | 2 | 15 | Auth/JwtSetup.cs |+----+------------------------------+--------+-------+-------+------------------------------------------+
Three low, five medium, two high.
Wall-clock time from opening the PR page to clicking approve or request changes, per pass, with a phone stopwatch. d for interruptions. It includes reading the risk summary, the proof, the local reviewer’s comment, and whatever part of the diff I decided to read. It does not include the agent’s own time or CI time, since those cost compute, not attention.
For the two high-tier PRs I needed two approvals. I did both myself from two accounts, a day apart. I’m telling you that up front because it’s the weakest number in this post: a second pass by the same person who wrote the task is not a second reviewer.
+--------+-----+------------------+--------------------+-----------------+| Tier | PRs | Human minutes | Minutes per PR | Escaped defects || | | (total) | (range) | |+--------+-----+------------------+--------------------+-----------------+| low | 3 | 8 | 1 - 4 | 0 || medium | 5 | 43 | 2 - 14 | 1 || high | 2 | 71 (two passes) | 30 - 41 | 0 |+--------+-----+------------------+--------------------+-----------------+| total | 10 | 122 | | 1 |+--------+-----+------------------+--------------------+-----------------+
An escaped defect here means a bug that passed every gate and got merged, and that I found afterwards.
The shape is what I expected and the size isn’t. I expected high-tier PRs to cost more. I didn’t expect the two high PRs to cost more than all eight others combined. Most of that time went into PR 09, the migration, where I generated the SQL with dotnet ef migrations script and read that instead of the C#. That was the right call and also the slowest thing I did all week.
Three real bugs, all in PRs where the tests were green:
Four false flags across the ten PRs:
Separately, the evidence filter dropped 5 findings across the run because the quoted line didn’t match the diff. I read all five. Four were rightly dropped (wrong file, invented code). One was a real point about PR 07 where the model quoted the line with the leading + from the diff, so my exact-match filter threw it away. The filter is blunt and I haven't decided whether to loosen it.
PR 07 is the escaped defect, and it is almost exactly Wasowski’s constructed example. The task was “remove duplication between the export query and the discount query.” The agent moved the shared part into one method, and the TenantId filter ended up applied by the discount handler's caller instead of in the shared method. The export path called the shared method directly. All tests passed, because the export tests ran with a single tenant. The local reviewer said nothing. I spent 9 minutes on it, read the refactored method, saw clean code, and approved.
I found it two days later while writing PR 08’s recorded run: I hit the export endpoint as a second test tenant and got the first tenant’s rows. It never left the demo repo, but in a real system that’s an incident, not a bug.
What would have caught it? Not a smarter reviewer, I think. A preservation contract would have: one line in the task saying “every query over Orders keeps its tenant filter, export included,” plus a test that asserts it with two tenants. I’ve since added that line to REVIEW.md (you saw it above) and the two-tenant test to the suite. The reviewer would now have a rule to check against. Whether a 14B model would apply it to a refactor that moves rather than removes the filter, I don't know yet.
The proof check bounced two PRs on the first try, and the messages were exactly what the tool prints:
PR 06, first body ("All 57 tests pass locally."):::error::## Testing needs pasted dotnet test output with Failed: 0, Passed: N (N > 0)PR 10, first body (Recorded run left as "n/a, tests cover it"):::error::high tier needs a recorded run: an HTTP transcript (curl -i output) or a link to a recording/screenshot
The second one is my favorite result of the whole experiment. The recorded run for PR 10 is a curl -i with a token that expired three minutes earlier, and a 401 with invalid_token in the response. That transcript would have looked very different on the 30-minute-skew version.
And the classifier caught a bug in itself. I added an eleventh fixture that renames src/Orders.Api/Auth/Login.cs to src/Orders.Web/Auth/Login.cs. My first version only looked at the new path, so a file moving *out of* Auth/ came back medium. An agent "reorganizing folders" could have walked auth code out of the high tier. The version above judges both paths, and that fixture now comes back high.
**I believe** the tier should be set by the change and not by who wrote it, and that it should be enforced by a check that the change itself cannot edit. Most of the safety in this setup is in boring places: pull_request_target with a base checkout, the head-SHA check, the bot filter, CODEOWNERS on .github/.
**I believe** requiring proof changed the agent’s behavior more than it changed mine. After PR 06 bounced once, the agent pasted test output on every later PR without being asked again, because the template and the failing check were in its context.
**I believe** a local 14B model is worth running as a first pass, but only as advisory. Three real catches against four false flags in ten PRs is useful. It is not a ratio I’d let block a merge.
**I don’t know** whether low tier is safe at zero humans. Three low PRs and zero escapes says nothing. The Anthropic post says they let low-risk changes merge on automated checks and that the line “keeps moving”. I’d want months of revert data before moving mine.
**I don’t know** what my baseline is. I didn’t time myself reading these ten PRs line by line first, so I can’t tell you this saved anything. I can only tell you where the 122 minutes went. Next round, I’ll time a comparable batch the old way before switching.
**I don’t know** if the one escape is a medium-tier problem or a refactoring problem. A 281-line refactor across three projects sits just under my 300-line threshold. Maybe “moves code between projects” should bump the tier on its own. Maybe the threshold is just wrong. Ten PRs can’t tell me.
Start smaller than I did:
And read the primary source. Not the rewrites.
**Sources**
Tags: ai-agents, code-review, dotnet, csharp, github-actions, ollama, devops, software-engineering, llm, claude-code
[Measuring Review in Human Minutes: A Risk-Tier Gate for Agent Pull Requests in .NET](https://pub.towardsai.net/measuring-review-in-human-minutes-a-risk-tier-gate-for-agent-pull-requests-in-net-c13f301342fa) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.