{"slug": "your-ai-agent-wrote-the-tests-too", "title": "Your AI agent wrote the tests too", "summary": "A developer argues that coding agents which write both a feature and its tests can produce code and tests that agree with each other while both misread the original requirement, leaving gaps such as cross-workspace authorization checks untested. The developer proposes a requirement-by-requirement review table mapping what needs to work, what was checked, and what remains missing, and suggests a separate reviewing agent read the requirements and actual tests rather than the first agent's summary.", "body_md": "An agent builds a feature, writes the tests, and tells you they all passed. The thing I want to know is whether those tests checked what we actually asked it to build.\n\nIf the agent misunderstood a requirement, that misunderstanding can end up in both the code and the tests. They agree with each other, but we still have a problem.\n\nI think this needs more attention as we hand more work to coding agents. I want to be able to look at a requirement and see what was checked, what the result was, and what we still don't know.\n\nLet's say you're adding a CSV export to a product with multiple workspaces. Members should be able to export reports from their own workspace.\n\nThe agent builds the endpoint and adds a test that creates a member, creates a report, and checks the CSV response. That test passes, but we haven't checked what happens if the member requests a report from a different workspace.\n\nNow make the export large enough that it needs a background job. You've got an endpoint to request it, another to check its status, and a way to download the file.\n\nI'd want the agent to follow that whole flow. Checking permissions when someone requests an export doesn't tell us whether another user can download the result.\n\n[OWASP's authorization guidance](https://cheatsheetseries.owasp.org/cheatsheets/Authorization_Cheat_Sheet.html) calls for checking permissions on every request. In this example, that means looking at all the ways someone can get to the exported data.\n\nThe CSV can contain exactly the right columns and still be available to the wrong person.\n\nFor this example, a review could look like this:\n\n| What needs to work | What was checked | What's still missing | \n|---|---|---|\n| A member can export a report from their workspace | An integration test requests the export and checks the CSV | We haven't tried a large export | \n| A member can't request another workspace's export | A test tries the request using a member from a different workspace | We haven't checked access to the job status or download | \n| Removing a member prevents future downloads | Nothing yet | We haven't decided what should happen to links that were already issued | \n\nThat last one is something I'd want worked out before the agent picks an implementation. If I remove someone from a workspace, should a download link they already have keep working? For how long?\n\nThose answers affect what we build and how we test it. A passing test can't settle a product decision we haven't made.\n\nThis gives me enough information to decide what still needs work. A single “verified” label would hide most of it.\n\nI'd want the reviewing agent to read the requirements and check the work against them. If we only give it the first agent's summary, we're asking it to find problems using an explanation that may have already left those problems out.\n\nRunning the tests in CI is useful too. We get a recorded run in a particular environment, but an authorization case we never tested is still missing.\n\nI'd give the reviewer a prompt like this:\n\n```\nGo through the requirements and show me what was checked for each one.\n\nRead the actual tests and results. Explain what they cover\nand which important cases are still missing.\n\nIf a requirement is unclear, explain the situation we need\nto decide on. Don't choose an interpretation and then\nreport that the requirement passed.\n\nInclude the commit and environment each result applies to.\n```\n\nI wouldn't do the same amount of review for a wording change as I would for a feature that exports customer data. But for the latter, I want to know that someone checked the access rules across the whole flow.\n\nHow are you doing this with your agents? Can you get from a requirement to the result of a check, or are you reconstructing that while reviewing the PR?", "url": "https://wpnews.pro/news/your-ai-agent-wrote-the-tests-too", "canonical_source": "https://dev.to/aylonarchdocs/your-ai-agent-wrote-the-tests-too-3k1p", "published_at": "2026-11-04 20:19:55+00:00", "updated_at": "2026-11-04 20:23:54.127398+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "ai-safety"], "entities": ["OWASP"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/your-ai-agent-wrote-the-tests-too", "markdown": "https://wpnews.pro/news/your-ai-agent-wrote-the-tests-too.md", "text": "https://wpnews.pro/news/your-ai-agent-wrote-the-tests-too.txt", "jsonld": "https://wpnews.pro/news/your-ai-agent-wrote-the-tests-too.jsonld"}}