{"slug": "i-reviewed-3-ai-written-prs-from-public-repos-here-s-what-i-d-have-blocked", "title": "I reviewed 3 AI-written PRs from public repos. Here's what I'd have blocked.", "summary": "A developer reviewed three AI-written pull requests from public repositories, including a merged GitHub Copilot agent PR to microsoft/testfx and an open Copilot PR to trimble-oss/modus-wc-2.0, applying an eight-point STOP checklist to each diff. The reviewer concluded the testfx binlog-deduplication PR was safe to merge with a minor documentation nit, but flagged the modus-wc-2.0 dependency-pin PR for splitting because it bundled a security override refresh with a jump of @stencil/react-output-target from 1.2.0 to 1.6.2. The review argues agent PRs need a human second pass focused on blast radius, mixed concerns, and rollback clarity rather than summaries.", "body_md": "I sell a human second pass on one AI-written PR ([Riven Desk](https://chopragunji.gumroad.com/l/byoyi)). Before pitching that, I wanted to do the work in public: pick three recent agent PRs from real repos, read the diffs (not just the summaries), and apply the same STOP checklist I give away for free.\n\nMethod, briefly:\n\n`copilot-swe-agent` and bodies/trailers mentioning Claude Code (`Co-Authored-By: Claude` / `claude.com/claude-code`).\nThese are outsider reviews. I don't maintain these projects. Maintainers may have context I don't. I'm grading the *diff as written*, not the people.\n\n**PR:** [Deduplicate sample binlog argument construction](https://github.com/microsoft/testfx/pull/11740)\n\n**Author signal:** GitHub Copilot coding agent (`copilot-swe-agent`)\n\n**Size:** ~27 changed lines across `eng/build-samples.ps1`, `eng/samples-tools.ps1`, `eng/test-samples.ps1`\n\n**State when reviewed:** merged\n\nExtracts repeated “build a `-bl:` / `/bl:` path under `$BinaryLogDirectory`” into `Get-SampleBinlogArgument` in `eng/samples-tools.ps1`, then calls it from the sample build/test scripts. The call sites already dot-source `samples-tools.ps1`, so the helper is in scope.\n\n| STOP | Fires? | Notes | \n|---|---|---|\n| 1 Secrets | No | No credentials or env files | \n| 2 Blast radius / no boundary | No | One clear intent: dedupe binlog arg construction | \n| 3 Mixed concerns | No | Script-only, no lockfile/infra hitchhikers | \n| 4 “No behavior change” while surface moved | Borderline | Behavior should match; see nit below | \n| 5 Rollback story | Fine | One revert undoes it | \n| 6 Security-sensitive paths | No | Build helper only | \n| 7 Prompt/tool surface | No |  | \n| 8 CI / tests | N/A from diff alone | Trivial pure helper; no new failing assertion added | \n\n`-bl:`` Get-SampleBinlogArgument`). Call sites that need MSBuild-style `/bl:` pass `-ArgumentPrefix \"/bl:\"` explicitly. That looks correct in the diff — just something a human should eyeball once so a future caller doesn’t assume the wrong flag.`.binlog` to `$LogName`. Call sites that previously built `\"$name.binlog\"` now pass `$name` (or `\"$name.restore\"`). Consistent in this PR; don’t re-add **Would merge.** Nothing on the STOP list fires hard. This is the kind of agent PR that should land with a short human glance, not a drama review.\n\n**What I’d fix before merge (optional):** one sentence in the PR body: “Default prefix `-bl:`; MSBuild restore/build paths pass `/bl:`.” Saves the next reviewer two minutes.\n\n**PR:** [Update vulnerable dependency pins](https://github.com/trimble-oss/modus-wc-2.0/pull/1569)\n\n**Author signal:** GitHub Copilot coding agent\n\n**Size:** `package.json` + `package-lock.json` (~280 line churn, mostly lockfile)\n\n**State when reviewed:** open\n\nUpdates npm overrides / pins for `brace-expansion@1|2|5` and `fast-uri`, and bumps `@stencil/react-output-target` from **1.2.0 → 1.6.2** (lockfile follows, including `@lit/react`, `ts-morph`, nested `minimatch`, etc.).\n\n| STOP | Fires? | Notes | \n|---|---|---|\n| 1 Secrets | No |  | \n| 2 Blast radius / no boundary | **Yes — ask/split** | Title says vulnerable pins; diff also jumps a codegen package several minors | \n| 3 Mixed concerns | **Yes — split** | Security pin refresh + Stencil React output-target upgrade in one PR | \n| 4 Surface moved | **Ask** | React wrapper generation can change across 1.2→1.6 with no app source in the diff | \n| 5 Rollback | Partial | Revert works; “why these versions” isn’t written | \n| 6 Security paths | Skimmed | Dependency pins are security-adjacent — need the CVE/advisory names in the PR | \n| 7 Prompt/tool | No |  | \n| 8 Tests that catch the regression | **Ask** | Lockfile-only PRs often go green without proving consumers still build | \n\n`@stencil/react-output-target` 1.2.0 → 1.6.2`brace-expansion` / `fast-uri` overrides`npm run` of those packages.\n**Would not merge as written — request changes / split.**\n\nSmallest clear path:\n\nThis is a classic agent shape: honest security cleanup, then a larger upgrade rides along because the agent “fixed versions” broadly.\n\n**PR:** [feat(lenses): built-in MCP Calls lens](https://github.com/Asymptote-Labs/agent-beacon/pull/723)\n\n**Author signal:** human opener + `Co-Authored-By: Claude` / `claude.com/claude-code` markers\n\n**Size:** ~387 changed lines — new `mcp.lens.html`, Playwright e2e, fixture lines, docs\n\n**State when reviewed:** open\n\nAdds a built-in dashboard “MCP Calls” lens: group MCP tool calls by server/tool, show args/results, mark failures, document it in `docs/concepts/lenses.mdx`, and cover it with Playwright (` builtin-mcp.spec.ts`) including an XSS-shaped payload in fixture args.\n\n| STOP | Fires? | Notes | \n|---|---|---|\n| 1 Secrets | No |  | \n| 2 Blast radius | No | Matches “add MCP lens” intent | \n| 3 Mixed concerns | No | Feature + tests + docs for the same lens | \n| 4 Surface claim | No | Docs say seven built-in lenses now | \n| 5 Rollback | Fine | Revert removes the lens file + docs line | \n| 6 Security-sensitive | Reviewed | Renders untrusted trace payloads in the browser | \n| 7 Prompt/tool / rendered AI output | **Watched closely — clears** | Uses `textContent` /`el()` helpers; e2e asserts markup in args does not execute | \n| 8 Tests | Strong | Playwright checks grouping, failure flag, XSS non-execution, empty state | \n\n`<img src=x onerror=...>`; the test opens Arguments and expects `window.pwned` to stay undefined. That’s the right bar for a lens that prints agent/MCP payloads.`arguments` / `result` under `gen_ai.tool.call`, while the lens JS reads `event.tool.arguments` / `event.tool.result` and `event.tool_call_id`. If `window.beacon.getTrace()` normalizes those fields before the lens runs, fine — and the e2e implies it does. If someone later feeds raw JSONL into the lens, args/results would silently go missing. Worth one maintainer sentence: “getTrace maps gen_ai.call → tool.*.”`mcp__server__tool`, `MCP:tool`, `event.type === \"mcp\"`) are documented enough in empty-state copy. Edge cases (weird tool names) are acceptable for v1.\n**Would merge after confirming the e2e job that runs `builtin-mcp.spec.ts` is green on the PR.** I would not block on style. I would leave the field-mapping note as a non-blocking comment.\n\nThis is closer to what you want from an agent: feature-sized, security-aware rendering, and a test that would fail if someone “helpfully” switched to `innerHTML`.\n\nAcross these three:\n\n| PR | Block? | Why | \n|---|---|---|\n| testfx#11740 | No | Small, matched intent, easy revert | \n| modus-wc#1569 | **Yes (as packaged)** | Security pins mixed with a multi-minor Stencil React target bump; missing advisory + verification notes | \n| agent-beacon#723 | No (pending green e2e) | Untrusted output handled; tests watch the scary path | \n\nThe interesting failure mode wasn’t “AI can’t code.” It was **scope creep inside a true-sounding title** (vuln pins) and **whether security-sensitive UI proves it doesn’t execute untrusted text**.\n\nCI green is not a STOP clear. Title confidence isn’t either.\n\nI used the free STOP one-pager while writing this: [STOP conditions before you merge an AI agent PR](https://chopragunji.gumroad.com/l/zpnmdn).\n\nI’m running a founding price on a single human review of one AI-written PR: **$49 for the first 5** (normally $99) → [Agent PR Audit — founding offer](https://chopragunji.gumroad.com/l/byoyi/FOUNDING).\n\nYou get concrete findings with file names, a merge stance, and what to fix — same shape as the sections above, on *your* PR. No fake “bugs down X%” claims. The point is fewer merges you can’t explain.\n\n— Gunjit / Riven Desk", "url": "https://wpnews.pro/news/i-reviewed-3-ai-written-prs-from-public-repos-here-s-what-i-d-have-blocked", "canonical_source": "https://dev.to/rivendesk/i-reviewed-3-ai-written-prs-from-public-repos-heres-what-id-have-blocked-50ah", "published_at": "2026-10-05 07:34:44+00:00", "updated_at": "2026-10-05 07:48:26.313227+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "ai-tools"], "entities": ["GitHub Copilot", "microsoft/testfx", "trimble-oss/modus-wc-2.0", "@stencil/react-output-target", "Claude Code", "brace-expansion", "fast-uri", "Riven Desk"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/i-reviewed-3-ai-written-prs-from-public-repos-here-s-what-i-d-have-blocked", "markdown": "https://wpnews.pro/news/i-reviewed-3-ai-written-prs-from-public-repos-here-s-what-i-d-have-blocked.md", "text": "https://wpnews.pro/news/i-reviewed-3-ai-written-prs-from-public-repos-here-s-what-i-d-have-blocked.txt", "jsonld": "https://wpnews.pro/news/i-reviewed-3-ai-written-prs-from-public-repos-here-s-what-i-d-have-blocked.jsonld"}}