I asked one model to review another, and it found a bias I would not have

A developer built an autonomous vulnerability remediation system and discovered that a cross-family AI reviewer rejected 65% of triage proposals, revealing a systematic bias in the primary model toward inflating severity ratings. The reviewer, running on Gemma, cited unsupported severity escalations in half of its rejections, a finding the developer says would have been missed without the architecture's design.

I spent five days building an autonomous system that owns the vulnerability remediation lifecycle — the six weeks after a scan, which is where one-person security programs actually die. Finding vulnerabilities is solved. Chasing the owner who never opened the ticket is not. The part worth writing about is not the pipeline. It is a number I did not expect and initially misread as a bug. Every triage decision in this system is challenged before it becomes state. A reasoning agent on Gemini 3.5 Flash proposes a severity, an SLA, and a remediation path, with cited evidence. A reviewer then either ratifies it or rejects it with a stated reason. The reviewer runs on Gemma — deliberately a different model family. The argument for that is easy to state and hard to verify: a model auditing its own reasoning shares its own blind spots. I believed it when I designed it. I did not have evidence for it. The reviewer rejects 65% of the proposals it sees — 91 rejections across 139 verdicts, measured on 31 August 2026. About half of the findings that reach a decision are ratified. Those denominators matter and I will not blur them. A rejected proposal is re-proposed once with the feedback attached, so one finding can produce more than one verdict: 65% is against verdicts, and the roughly-half figure is against decisions. Reporting only the friendlier of the two would have been the easier thing to do. My first reaction was that something was broken. A 65% disagreement rate looks like a defect rate. It is not. A reviewer that ratifies everything is indistinguishable from having no reviewer at all, so the rate is a health metric — and the direction that should worry you is the one approaching zero. I categorised every rejection reason expecting a spread. Instead: | What the reviewer objected to | Share of rejections | |---|---| | Severity escalated beyond what the CVSS evidence supports | 50% | | Remediation text naming no specific version | 32% | | Proposed SLA conflicting with the CISA KEV due date | 20% | | Prompt-injection text inside the scanner's own comment field | 12% | A rejection can cite more than one of these, so the shares total more than 100%. The last row was not a category I went looking for. Half of all rejections say the same thing. Verbatim, from the record: "The severity is escalated to critical without evidence supporting such a jump from the CVSS base of 7.8, and the remediation is vague." "The severity is rated as critical despite the scanner's CVSS being 5.4 and the NVD description indicating a high 7.8 severity, creating a mismatch between evidence and proposal." The triage model has a consistent bias toward inflating severity past its own cited evidence. The split confirms it: rejected proposals skew critical 24 critical to 24 high — 55% critical , ratified ones lean the other way 22 to 32 — 41% critical . This is the part I want to be careful about, because it is easy to overclaim. I cannot prove a same-family reviewer would have missed it — that experiment is one I did not run. What I can say is narrower and still worth something: the cross-family reviewer did catch it, it caught it systematically rather than occasionally, and it wrote down its reasoning every time in a form I could count. I did not find this bias by reading outputs. The architecture found it and filed it. The system runs itself on a schedule. Cloud Scheduler publishes a tick, Pub/Sub fans it to two workers, each executing as its own service account. At 09:01 UTC on August 28th, unattended, both workers woke on cycle 30693 and did nothing. No ticket, no nudge, no escalation. Being precise about why, because the honest version is less tidy than the one I first wrote: the exception agent swept its acceptances and correctly found none to act on. The chase agent had an empty set of SLA clocks to walk, so its restraint that morning was trivial rather than considered. When there is something to weigh it does log the weighing — cycle 9003 logged wait=9 , nine findings evaluated and nine deliberately left alone because their deadlines had not arrived. I could have dressed the first version up and left it. The point survives the correction and is better for it: on the mornings there is something to weigh, the autonomous loop's answer is "not yet," and it says so rather than manufacturing activity. A system that invents work to look busy is worse The duplicate-delivery guard I did have to go and provoke, because production had not obliged. Publishing the same tick a second time gets this — one line per worker, both from the same cycle: tick already ran cycle=9004 rz-worker-chase tick already ran cycle=9004 rz-worker-exception Pub/Sub delivers at least once, so a redelivered tick is a real possibility rather than a hypothetical. The second copy is recognised and does nothing. Every authoritative state-changing tool takes a key derived from the finding, the action, and the cycle, so a resumed agent cannot open a second ticket or send a second nudge. The obvious argument for that design is "don't duplicate work." The stronger one only appeared under test: models are not deterministic. Running the same cycle twice produces different decisions for the same finding — one finding was ratified on the first run and rejected twice on the second. Without the guard, the second run would not merely have duplicated work. It would have silently overwritten a decision a human may already have acted on, with a contradictory one. Each agent has its own service account. What that buys is worth stating precisely, because the obvious version of the claim is not true: Firestore Native has no collection-scoped IAM permission, and Security Rules are bypassed entirely by a server SDK authenticating as a service account. Per-collection separation cannot be enforced by IAM here, and claiming it would be claiming a control that does not exist. What is enforceable is a per-database boundary, and that is where the reporting agent's limit lives: read-only on the operational database, write access conditioned to a separate reports database. It is structurally incapable of writing a ticket. Asserting that in a README is free. So the check performs the forbidden action: a Cloud Run job whose identity is the reporting agent attempts the write and reports what happened. expect DENIED got DENIED PermissionDenied write a ticket as rz-reporting expect ALLOWED got ALLOWED write a report as rz-reporting expect DENIED got DENIED PermissionError read the tracker token as rz-exception expect ALLOWED got ALLOWED read a finding as rz-exception Half of those expect ALLOWED on purpose. An identity that can do nothing proves only that it is broken; the control is that the boundary falls in a specific place. The first version of that check was wrong in an instructive way. It used --impersonate-service-account from my laptop, and both identities returned PERMISSION DENIED . It looked like proof. It was proof that I cannot impersonate anyone — which says nothing whatsoever about the secret. Running as the identity rather than borrowing it is the whole difference. Late in the build I had the control suite audited. One check — the one verifying that the reviewer catches a prompt injection with the input guardrail switched off — had not called a model since the previous day. It ran a cycle against a fixed probe number. The cycle skips any finding whose idempotency record already exists. So after its first successful run it returned immediately, and the check re-read the decision that first run had written, found the expected text, and reported PASS. In 0.3 seconds. For 37.8 hours. The evidence was in my own timings table the entire time: three checks measured 17.2 seconds, while one triage-and-review round costs 15 to 22 seconds on its own. The number was too good and I read it as good news. It now clears its prior record and accepts a verdict only if it was written during that invocation. It takes 31 seconds, because it does the work. The lesson is not "write better tests." It is that cheap-and-green is the most comfortable way for a verification suite to fail , and the suite you trust most Two gaps that every reviewer found within an hour, so I would rather name them: There is no cross-asset finding normalization : real scanner output repeats one CVE across hundreds of hosts, and the synthetic corpus here has 400 findings with 400 distinct CVEs, so the problem never arises. That is the next thing. And the closure loop is newer than the rest of the system, which shows. A rescan closes what it confirms fixed and refuses to close what it could not examine — but the scan file's coverage manifest is taken on trust, and a regression reopening a finding resets that ticket's episode counters rather than preserving the whole trail. Both are known and neither is exercised by the committed corpus, which is exactly the kind of thing worth saying out loud about a feature that is four days old. ADK 2.8 on Vertex AI. Gemini 3.5 Flash for reasoning, Gemma for review, Firestore for state, Agent Engine Memory Bank across sessions, Pub/Sub with a dead-letter queue proven by poisoning it, Model Armor on untrusted ingress, Cloud Run, Cloud Trace, Secret Manager, Terraform. Seven service accounts, of which the two scheduled workers genuinely execute as their own identity; the reasoning graph currently runs under a single client, and the remaining boundaries are proven by running the checks as those identities rather than assumed at runtime. 623 tests. I built Remediation Zero for the All Things Agentic Hackathon , in the Fortified Enterprise Fleet track, over five days in August 2026. Everything in the repository was written during the submission period. The corpus is entirely synthetic — reserved ranges only, .invalid hostnames and addresses drawn from 192.0.2.0/24 , 198.51.100.0/24 and 203.0.113.0/24 — with real CVE identifiers. Code, architecture diagram, and a runbook with every step timed: github.com/Ayliea/remediation-zero