cd /news/ai-tools/are-ai-assisted-commits-reverted-mor… · home › topics › ai-tools › article
[ARTICLE · art-173206] src=dev.to ↗ pub= topic=ai-tools verified=true sentiment=· neutral

Are AI-assisted commits reverted more often? I checked 663 matched pairs. The honest answer is 'can't tell'

Developer Vyankatesh Rohokale ran a matched-pair observational study of 663 AI-disclosed commits against 663 comparable human-only commits and found no clear difference in revert rates, but concluded the sample is too small to detect a real effect. Only 1 of 663 AI-disclosed commits was reverted within 30 days versus 2 of 663 controls (odds ratio 0.51, 95% CI 0.05–5.57), and a follow-up fix analysis on a 220-pair subset gave an odds ratio of 1.15 (0.75–1.77), though a manual review judged only about 30% of flagged follow-ups to be plausible fixes. Rohokale estimates roughly 10,000 pairs would be needed to detect a 2x effect, and notes the study was conducted with an AI assistant that wrote the code and analysis.

recommendation

Size commit-outcome studies for the base rate before running them: at ~0.2% revert rates, plan for roughly 10,000 matched pairs to detect a 2x effect, and validate keyword-based fix detection before trusting a null.

more in playbook: Evals and testing for LLM apps
by read2 min views8 publishedNov 20, 2026

Disclosure: I ran this study with an AI assistant (Claude), which wrote the code, did the analysis and drafted the paper. It also did the manual check of my fix heuristic, so that check is not an independent human review. Results are aggregate; no developer, repo or commit is named. This is observational: "associated with", never "causes".

Question: are commits that disclose AI help (e.g. a Co-Authored-By: Claude trailer) reverted or followed by a bug-fix more often than comparable human-only commits?

I wrote down the hypotheses and definitions before pulling any data. The short answer: no clear difference, but the study is too small to say much.

gh. No repo was cloned and no code was run. Matching on size worked: median diff was 102 changed lines for AI-disclosed commits and 103 for controls.

Within 30 days, 1 of 663 AI-disclosed commits was reverted, versus 2 of 663 controls. Odds ratio 0.51, 95% CI 0.05 to 5.57. That interval spans "much safer" to "5x worse". I checked my revert detector against the 264k commits in the downloaded histories: about 0.2% of commits get reverted, so reverts are just rare. Detecting a 2x effect at that rate would take on the order of 10,000 pairs (rough calculation).

My pre-registered "fix follow-up" rule: a later commit within 14 days, touching a same file, with fix|bug|hotfix|regression in the message. On a feasible 220-pair subset: OR 1.15 (0.75-1.77), consistent with no difference.

But I then read 100 flagged follow-ups and judged only about 30% to be plausible fixes (95% CI 22-40%). Common errors: shared package.json/README/lockfiles, a keyword in the body of a non-fix commit, merges of the same fix. So that "null" mostly reflects a noisy measurement. A stricter, exploratory variant gave the same picture (OR 1.16, 0.72-1.86).

For this sample: **no evidence of a difference, and little ability to detect one.** A real answer needs about 10x more pairs and a validated way to link fixes to the commits they fix (issue links or SZZ-style tracing), plus some idea of how reliably people disclose.

Related reading (arXiv): [On the Use of Agentic Coding](https://arxiv.org/abs/2509.14745), [Agentic Much? Adoption of Coding Agents on GitHub](https://arxiv.org/abs/2601.18341).

Preprint-style write-up, not peer reviewed.

By Vyankatesh Rohokale

── more in #ai-tools 4 stories · sorted by recency
── more on @vyankatesh rohokale 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/are-ai-assisted-comm…] indexed:0 read:2min 2026-11-20 · —