Metaharness for cross-agent adversarial reviews

A single AI reviewer can produce a polished, authoritative comment that is completely wrong. In CI, that failure mode is worse than silence: once a bogus finding looks plausible enough to block a pull request, teams either waste time chasing fiction or stop trusting the reviewer at all.

One pattern worth copying immediately is to make routing policy explicit and conservative instead of letting the review system imply certainty. The useful move here is simple: only auto-route when confidence is high enough, cost is still inside budget, and your benchmark signal is not statistically ambiguous. If any of those checks fail, send the change to a human reviewer with the reasons attached.

export function routeReview(
  ctx: ReviewContext,
  policy: ReviewGatePolicy = defaultReviewPolicy(),
): ReviewDecision {
  const reasons: string[] = [];

  if (ctx.highRiskFileTouched) reasons.push('high-risk-file');
  if (ctx.securitySensitiveChange) reasons.push('security-sensitive');
  if (ctx.costUnits > policy.costBudget) reasons.push('over-budget');
  if (ctx.confidence < policy.confidenceThreshold) reasons.push('low-confidence');
  if (ctx.bootstrap.lower95 <= 0 && ctx.bootstrap.upper95 >= 0) {
    reasons.push('ambiguous-benchmark');
  }

  return { route: reasons.length > 0 ? 'human' : 'auto', reasons };
}

The article’s default policy uses a confidence threshold of 0.7, escalates when cost units exceed 20, and also escalates when the benchmark interval includes zero. That last check is the subtle one: if your evaluation cannot tell “better than baseline” from noise, you do not have evidence strong enough to grant an automatic pass. Encoding that as code instead of reviewer intuition gives you deterministic behavior, better auditability, and a safer failure mode under uncertainty.

This also keeps the system honest about what it knows. A metaharness should reduce human review load without pretending every model judgment is gate-worthy. The article’s framing is to trust validated signals and route uncertainty, not to smooth uncertainty away with nicer prose.

The full write-up also covers: