{"slug": "your-ai-agent-writes-playwright-tests-that-pass-while-your-api-returns-500", "title": "Your AI agent writes Playwright tests that pass while your API returns 500", "summary": "A developer released @vedhajs/playwright-test, an Apache-2.0 package that re-exports Playwright's test and expect with an automatic \"guard mode\" fixture that watches the browser context during each test and fails a run when the app reports a server-side problem, such as a POST returning 500, even if the page's own assertions pass. The tool targets agent-written Playwright suites, which the developer says rarely add network assertions; in a small pilot on held-out tasks (one model, three trials per task), an agent driving the browser live missed 9 seeded bugs with Vedha's MCP server versus 26 with Playwright MCP.", "body_md": "AI agents now write a lot of Playwright tests. You ask Claude Code or Copilot to \"add a test for the checkout\", or let an agent explore your app through Playwright MCP and turn what it did into a spec. Healer agents even fix failing tests for you.\n\nHere is the kind of test they write:\n\n``` js\nimport { test, expect } from \"@playwright/test\";\n\ntest(\"places an order\", async ({ page }) => {\n  await page.goto(\"/\");\n  await page.getByLabel(\"Name\").fill(\"Ada\");\n  await page.getByRole(\"button\", { name: \"Place order\" }).click();\n  await expect(page.getByRole(\"status\")).toContainText(\"Order placed\");\n});\n```\n\nIt's a good test: role-based locators and a web-first assertion. It passes.\n\nIt also passes when the order API returns **500 and stores nothing**, as long as the page still shows \"Order placed\". The agent checked what it could *see*. Nobody told it to check what the server did, and agents almost never add a network assertion on their own.\n\nThat matters more with agents than with people. A human who wrote the checkout code might remember the API can fail. An agent generating fifty tests from what's on screen won't, and a healer agent that \"fixes\" a red test by updating the expected text can make things worse. The suite goes green while the backend is broken.\n\nChange the import, in the tests the agent wrote or in the template it writes from:\n\n``` js\nimport { test, expect } from \"@vedhajs/playwright-test\";\n```\n\nNothing else changes. Same test, same broken app:\n\n``` python\n  ✓  1 1-playwright.spec.ts › places an order (import from @playwright/test)\n  ✘  2 2-vedha.spec.ts › places an order (import from @vedhajs/playwright-test)\n\n    Error: Vedha guard: the test's own checks passed, but the app reported a problem:\n      - POST /api/orders returned 500\n```\n\nOn a healthy app, both pass.\n\nThe failure message is also written for whoever reads it next, human or agent: what the app did, and how to accept it if it was expected. A healer agent gets a real reason instead of a green run.\n\n`@vedhajs/playwright-test` re-exports Playwright's `test` and `expect` with an automatic fixture (\"guard mode\") that watches the browser context while each test runs:\n\n`expect` passed.`vedha-guard.json`.\nWhen a finding is expected, opt out per test or per endpoint:\n\n```\ntest.use({ vedhaGuard: \"strict\" }); // 4xx writes and page errors fail too\ntest.use({ vedhaGuard: { ignore: [\"/api/flaky\", /\\/v2\\//], hosts: [\"api.other-domain.com\"] } });\ntest.use({ vedhaGuard: false }); // off, for a test that expects a 500\n```\n\nIt is still Playwright's runner, so retries, sharding, reporters and traces work as before.\n\nPut it where the agent will pick it up:\n\n`CLAUDE.md`, `AGENTS.md`, Copilot instructions): \"Import `test` and `@vedhajs/playwright-test`, not `@playwright/test`.\"\nGuard mode is for the tests an agent writes. When the agent *is* the tester, driving the browser live and reporting pass or fail, Vedha also ships an MCP server that checks every step the same way. In a small pilot on held-out tasks (one model, 3 trials per task), the agent missed **9** seeded bugs with Vedha's MCP server against **26** with Playwright MCP. That's a separate setup from guard mode, covered in [an earlier post](https://dev.to/arun_kumart_22e9a9959b10/is-playwright-alone-enough-for-agentic-testing-433p).\n\n`waitForResponse` assertion catches this too. Guard mode is the safety net for the many tests, agent-written or not, that don't have one.\n\n```\nnpm install -D @vedhajs/playwright-test\n```\n\nChange the import in one agent-written spec and run it. If a test that used to pass now fails, look at the request it names. Either the agent's test was hiding a bug, or it's an expected error to add to `ignore`.\n\nRepo, demo and benchmark data: **[https://github.com/HMAKT99/VEDHA](https://github.com/HMAKT99/VEDHA)** (Apache-2.0)\n\nI'd like to hear what it finds in agent-written suites, and about any false alarm.", "url": "https://wpnews.pro/news/your-ai-agent-writes-playwright-tests-that-pass-while-your-api-returns-500", "canonical_source": "https://dev.to/arun_kumart_22e9a9959b10/your-ai-agent-writes-playwright-tests-that-pass-while-your-api-returns-500-1p0d", "published_at": "2026-11-01 03:08:58+00:00", "updated_at": "2026-11-01 03:22:44.551929+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "ai-tools", "ai-products"], "entities": ["Vedha", "@vedhajs/playwright-test", "Playwright", "Claude Code", "GitHub Copilot", "HMAKT99"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/your-ai-agent-writes-playwright-tests-that-pass-while-your-api-returns-500", "markdown": "https://wpnews.pro/news/your-ai-agent-writes-playwright-tests-that-pass-while-your-api-returns-500.md", "text": "https://wpnews.pro/news/your-ai-agent-writes-playwright-tests-that-pass-while-your-api-returns-500.txt", "jsonld": "https://wpnews.pro/news/your-ai-agent-writes-playwright-tests-that-pass-while-your-api-returns-500.jsonld"}}