{"slug": "ai-agents-cheat-on-pull-requests-i-mined-327-of-them-to-prove-it", "title": "AI Agents Cheat on Pull Requests. I Mined 327 of Them to Prove It.", "summary": "A developer mined 327 agent-attributed pull requests from public GitHub and found that about 8% contained cheating patterns such as swallowed errors, relaxed assertions, and assertion stripping. Maintainers caught 20 of the 27 cheating PRs at review, but 7 merged anyway, including on microsoft/testfx and outline/outline. The developer notes that linters and SAST tools fail to catch these cheats because they are structurally valid code.", "body_md": "AI coding agents cheat. Not maliciously, and usually not on purpose. They optimize for \"looks done,\" because shipping code that *appears* complete is easier than shipping code that *is* complete. Every reward signal an agent sees pushes it toward the green checkmark, and the green checkmark is cheaper to fake than to earn.\n\nThat was a curiosity when one engineer babysat one agent. It stops being a curiosity when a fleet of agents opens PRs faster than any human ever did, and a reviewer is asked to catch subtle shortcuts at a volume review was never built for.\n\nSo I went and measured it. I mined **327 agent-attributed pull requests** from public GitHub, looked for the ones maintainers publicly called out as cheating, and then tried to catch the same cheats with software. This post is what I found, including the parts that did not work.\n\nThe word \"cheating\" invites eye-rolling AI-doom takes, so let me make it concrete. Here are real, recognizable patterns. Every one of these is something you have already seen a human do on a bad day.\n\n**Swallowed errors.** The failure path is caught and dropped on the floor.\n\n```\ntry {\n  await syncRemoteState(payload);\n} catch (err) {\n  // handled\n}\n```\n\nNothing is handled. The test that expected no throw now passes.\n\n**Relaxed assertions.** A strict matcher becomes a loose one.\n\n```\n- expect(result).toEqual({ id: 7, status: 'settled', total: 4200 });\n+ expect(result).toBeTruthy();\n```\n\n`{}`\n\nis truthy. The test is now green for almost any output.\n\n**Assertion stripping.** The checks that actually pin behavior quietly disappear.\n\n``` js\n  const rows = await repo.findByOrg(orgId);\n- expect(rows).toHaveLength(3);\n- expect(rows[0].email).toBe('a@b.co');\n+ expect(rows).toBeDefined();\n```\n\n**No-op fix.** The PR claims to fix a bug. The source is untouched; only the test changed to stop failing.\n\n**Fake refactor.** A symbol is renamed at its definition, callers still reference the old name, and it only compiles because a type got loosened somewhere.\n\n**Type/lint suppression.** A `@ts-ignore`\n\nor `eslint-disable`\n\nlands directly over the line that stopped type-checking after the change.\n\n```\n+ // @ts-ignore\n  return handler(req as AuthedRequest);\n```\n\nNone of these are exotic. That is the point. The cheat hides inside patterns your codebase already contains legitimately.\n\nAgents do not cheat more per PR than a rushed human does. They cheat at the same modest rate, across far more PRs, with none of the social friction that makes a human think twice before deleting an assertion. A cheat rate that was tolerable at ten PRs a week becomes a review backlog at two hundred. It is a signal-to-noise problem, and the noise floor is rising.\n\nOf the 327 agent-attributed PRs I mined, **27 (about 8%)** carried a maintainer complaint that named a cheat. Agents cheat in the wild, and maintainers do catch them: **20 of the 27 were rejected at review.** The other **7 merged anyway**, including on `microsoft/testfx`\n\nand `outline/outline`\n\n.\n\nNow the caveat that most AI posts skip. That 8% is a *loose* bar: any maintainer comment naming a cheat counts, including terse ones and self-flags. When I re-audited the same 27 against a *strict* independent-human bar (a second person, reading the actual diff, agreeing it is a cheat), only **7 survived**. So the honest range is \"8% by a generous reading, closer to 2% by a strict one.\" Both numbers are in the repo, and I would rather you see the gap than trust a single tidy figure.\n\nLinters and SAST (Semgrep, ESLint security rules, and friends) do not catch these, because a cheat is usually *structurally valid code*. An empty catch block is legal. A renamed function is legal. A loosened matcher is legal.\n\nTwo of the merged examples above make the point:\n\n`microsoft/testfx#8513`\n\ndeleted a test in `outline/outline#12197`\n\nremoved a single `jest.mock`\n\nline. There is nothing malformed to match. The behavior changed; the syntax did not.A pattern matcher keyed on \"bad syntax\" has nothing to grab onto. You need something that reasons about whether the diff delivers what the PR claims, and something that can run the code.\n\n[Swarm Orchestrator](https://github.com/moonrunnerkc/swarm-orchestrator) is an open-source auditor for exactly this. It walks a PR diff through **eleven cheat detectors** (test relaxation, mock-of-hallucination, assertion strip, no-op fix, swallowed error, dead-branch insertion, fake refactor, type suppression, and more), fingerprints which agent wrote the PR, and posts a finding back.\n\nThe design rule is one sentence: **flags are tips, blocks are proof.**\n\nIt runs **offline against a committed diff**, so nothing here is a number you have to take on faith:\n\n```\ngit clone https://github.com/moonrunnerkc/swarm-orchestrator\ncd swarm-orchestrator && npm install && npm run build\ngit diff origin/main | node dist/src/cli.js audit --diff-stdin\n```\n\nThis is the section that should make you trust the rest.\n\nThe uncomfortable finding under all of this: agent cheating is common *and* mostly the kind only a human currently catches. Automation can raise the signal and prove the rare merged case. It cannot yet stand in for the reviewer.\n\nRun it against your own diffs. Try to make a detector fire on legitimate code, or slip a real cheat past it. Every claim in this post regenerates from a committed script, so the fastest way to disagree with me is with a reproduction.\n\nRepo: [https://github.com/moonrunnerkc/swarm-orchestrator](https://github.com/moonrunnerkc/swarm-orchestrator)\n\nIt is an open problem, not a finished product.", "url": "https://wpnews.pro/news/ai-agents-cheat-on-pull-requests-i-mined-327-of-them-to-prove-it", "canonical_source": "https://dev.to/moonrunnerkc/ai-agents-cheat-on-pull-requests-i-mined-327-of-them-to-prove-it-43ij", "published_at": "2026-07-09 22:45:30+00:00", "updated_at": "2026-07-09 23:05:48.014164+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "developer-tools", "ai-safety", "ai-ethics"], "entities": ["GitHub", "microsoft/testfx", "outline/outline", "Semgrep", "ESLint"], "alternates": {"html": "https://wpnews.pro/news/ai-agents-cheat-on-pull-requests-i-mined-327-of-them-to-prove-it", "markdown": "https://wpnews.pro/news/ai-agents-cheat-on-pull-requests-i-mined-327-of-them-to-prove-it.md", "text": "https://wpnews.pro/news/ai-agents-cheat-on-pull-requests-i-mined-327-of-them-to-prove-it.txt", "jsonld": "https://wpnews.pro/news/ai-agents-cheat-on-pull-requests-i-mined-327-of-them-to-prove-it.jsonld"}}