# Canary Agentic Autofix With Failure Classes and Reliability Gates

> Source: <https://dev.to/robinzzz/canary-agentic-autofix-with-failure-classes-and-reliability-gates-id0>
> Published: 2026-07-16 04:00:39+00:00

GitHub announced agentic autofix for code scanning alerts in public preview on July 10, 2026.

Primary source: [GitHub Changelog, July 10, 2026](https://github.blog/changelog/2026-07-10-agentic-autofix-for-code-scanning-alerts-in-public-preview/).

The wrong metric is “percentage of alerts with a generated patch.” Generation is only the first transition:

``` php
alert -> candidate -> build -> tests -> security oracle
      -> human review -> merge -> post-merge observation
```

This is an evaluation proposal, not a benchmark or assessment of GitHub's preview.

Start with repositories that have active owners, deterministic builds, relevant isolated tests, reversible releases, and no automatic production deployment from candidate patches. Exclude abandoned code, safety-critical paths, and repositories with unreliable tests.

Assign the canary deterministically—for example, hash a stable alert ID into a fixed percentage. Do not move difficult results out of the cohort after seeing them.

Record every attempt, including abstentions and failures:

```
attempt_id: "<id>"
alert_class: "<normalized class>"
base_revision: "<commit>"
outcome:
  generated: true
  applied_cleanly: true
  build_passed: true
  tests_passed: false
  security_oracle_passed: false
  human_decision: "rejected"
failure_class: "semantic_incomplete"
escaped_to_default_branch: false
```

The schema is local evaluation metadata; it does not imply that GitHub exposes these fields.

| Class | Meaning |
|---|---|
| No candidate | Tool abstained |
| Scope violation | Unrelated or forbidden paths changed |
| Apply failure | Patch does not apply to recorded base |
| Build failure | Patched revision cannot build |
| Regression | Existing behavior broke |
| Semantic incomplete | Alert changed but security property remains broken |
| Overcorrection | Valid behavior was blocked |
| Test manipulation | Validation was weakened or removed |
| Stale base | Result targeted another revision |
| Review ambiguity | Human cannot establish why the patch is safe |
| Infrastructure | Evaluation could not complete |
| Post-merge escape | Later evidence disproved acceptance |

Use one primary class and optional secondary classes. Otherwise one attempt contaminates several denominators.

Existing unit tests may not encode the alert's security property. For authorization, require unauthorized denial, authorized success, alternate-entry coverage, and preserved audit behavior. Run the oracle on the unpatched baseline first; if it cannot expose the problem, a patched pass proves little.

Predeclare gates such as zero critical scope violations, zero test-manipulation events, zero automatic merges, bounded infrastructure failure, and a minimum completed sample. Segment results by alert class, language, repository tier, and patch size. Report confidence intervals rather than promoting after a lucky week.

Stop immediately for privilege expansion, weakened policy, revision mismatch, secret exposure, or unexplained production impact. Rollback includes disabling new attempts, freezing open candidates, reviewing already merged canary patches, and preserving evidence.

Offline fixtures are imperfect, alerts are correlated, and escapes may surface late. That is why wider deployment should be earned through bounded evidence, explicit denominators, and safety events that outweigh an attractive average.