{"slug": "what-to-verify-after-an-ai-agent-says-tests-pass", "title": "What to verify after an AI agent says tests pass", "summary": "A developer writing as Riven Desk outlines a verification sequence for reviewing AI agent-generated pull requests, arguing that a green CI check from an agent is a claim rather than a merge signal. The recommended order is diff review, assertion inspection, failure-path coverage, and local reproduction, with the warning that a passing suite on the wrong surface is still the wrong merge. The author also promotes a paid \"AI Agent Code Review Kit\" containing the checklist and Cursor-oriented rules.", "body_md": "Green CI from an agent is not a merge signal. It is a claim: \"I ran something and it exited zero.\" Your job is to check whether that claim covers the bug, the intent, and the failure modes you care about.\n\nI treat \"tests pass\" as the start of a short verification sequence — not the end of review.\n\nOpen the file list first. Ignore the agent's narrative until you can answer:\n\nAgents often pad the suite while touching unrelated helpers. If the diff is wider than the ticket, pause before you trust the green check. A passing suite on the wrong surface is still the wrong merge.\n\nOpen the new or edited tests and ask: **would this fail if the original bug came back?**\n\nWatch for:\n\n`200`\nIf the assertion would still pass with the regression restored, the suite is theater. Request a tighter assert before you approve. Prefer one sharp negative case over five soft positives.\n\nGreen tests often skip the paths that hurt in production:\n\nPick the failure mode closest to the ticket and ask whether any test forces it. If not, either add that case or manually exercise it before merge. Agents optimize for \"looks covered.\" You optimize for \"breaks when broken.\"\n\nDo not trust the agent's pasted output alone. Run the same command on your machine (or the same CI job) with the PR branch checked out:\n\n```\n# example — use whatever your repo actually runs\nnpm test -- path/to/relevant.spec.ts\n```\n\nCheck:\n\nIf you cannot reproduce green locally, you do not have a pass — you have a story. Fix the story before merge.\n\nOrder matters: **diff → assertions → failure paths → local reproduce.** Skip ahead and you rubber-stamp confidence. Stop early when the file list or asserts are weak  do not sink twenty minutes into a suite that never could catch the bug.\n\nThis is the same bar I use on agent PRs elsewhere: keep the speed, keep your judgment. Green is necessary. It is not sufficient.\n\nIf you want the packaged checklist, Cursor-oriented rules, and review prompts I use on agent PRs, the **AI Agent Code Review Kit** is here: [https://chopragunji.gumroad.com/l/nxoboi](https://chopragunji.gumroad.com/l/nxoboi)\n\n— Riven Desk\n\nWhat do you check first after an agent claims tests pass — file list, asserts, or a local re-run? Drop your sequence in the comments.", "url": "https://wpnews.pro/news/what-to-verify-after-an-ai-agent-says-tests-pass", "canonical_source": "https://dev.to/rivendesk/what-to-verify-after-an-ai-agent-says-tests-pass-1ad5", "published_at": "2026-09-10 01:31:40+00:00", "updated_at": "2026-09-10 02:19:05.905828+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "ai-tools"], "entities": ["Riven Desk", "Cursor", "AI Agent Code Review Kit", "Gumroad"], "alternates": {"html": "https://wpnews.pro/news/what-to-verify-after-an-ai-agent-says-tests-pass", "markdown": "https://wpnews.pro/news/what-to-verify-after-an-ai-agent-says-tests-pass.md", "text": "https://wpnews.pro/news/what-to-verify-after-an-ai-agent-says-tests-pass.txt", "jsonld": "https://wpnews.pro/news/what-to-verify-after-an-ai-agent-says-tests-pass.jsonld"}}