Canary Agentic Autofix With Failure Classes and Reliability Gates

GitHub announced agentic autofix for code scanning alerts in public preview on July 10, 2026. A developer proposed an evaluation framework using canary testing with failure classes and reliability gates to measure the tool's effectiveness beyond simple patch generation rates.

GitHub announced agentic autofix for code scanning alerts in public preview on July 10, 2026. Primary source: GitHub Changelog, July 10, 2026 https://github.blog/changelog/2026-07-10-agentic-autofix-for-code-scanning-alerts-in-public-preview/ . The wrong metric is “percentage of alerts with a generated patch.” Generation is only the first transition: php alert - candidate - build - tests - security oracle - human review - merge - post-merge observation This is an evaluation proposal, not a benchmark or assessment of GitHub's preview. Start with repositories that have active owners, deterministic builds, relevant isolated tests, reversible releases, and no automatic production deployment from candidate patches. Exclude abandoned code, safety-critical paths, and repositories with unreliable tests. Assign the canary deterministically—for example, hash a stable alert ID into a fixed percentage. Do not move difficult results out of the cohort after seeing them. Record every attempt, including abstentions and failures: attempt id: "<id " alert class: "<normalized class " base revision: "<commit " outcome: generated: true applied cleanly: true build passed: true tests passed: false security oracle passed: false human decision: "rejected" failure class: "semantic incomplete" escaped to default branch: false The schema is local evaluation metadata; it does not imply that GitHub exposes these fields. | Class | Meaning | |---|---| | No candidate | Tool abstained | | Scope violation | Unrelated or forbidden paths changed | | Apply failure | Patch does not apply to recorded base | | Build failure | Patched revision cannot build | | Regression | Existing behavior broke | | Semantic incomplete | Alert changed but security property remains broken | | Overcorrection | Valid behavior was blocked | | Test manipulation | Validation was weakened or removed | | Stale base | Result targeted another revision | | Review ambiguity | Human cannot establish why the patch is safe | | Infrastructure | Evaluation could not complete | | Post-merge escape | Later evidence disproved acceptance | Use one primary class and optional secondary classes. Otherwise one attempt contaminates several denominators. Existing unit tests may not encode the alert's security property. For authorization, require unauthorized denial, authorized success, alternate-entry coverage, and preserved audit behavior. Run the oracle on the unpatched baseline first; if it cannot expose the problem, a patched pass proves little. Predeclare gates such as zero critical scope violations, zero test-manipulation events, zero automatic merges, bounded infrastructure failure, and a minimum completed sample. Segment results by alert class, language, repository tier, and patch size. Report confidence intervals rather than promoting after a lucky week. Stop immediately for privilege expansion, weakened policy, revision mismatch, secret exposure, or unexplained production impact. Rollback includes disabling new attempts, freezing open candidates, reviewing already merged canary patches, and preserving evidence. Offline fixtures are imperfect, alerts are correlated, and escapes may surface late. That is why wider deployment should be earned through bounded evidence, explicit denominators, and safety events that outweigh an attractive average.