Your AI agent writes Playwright tests that pass while your API returns 500

AI agents now write a lot of Playwright tests. You ask Claude Code or Copilot to "add a test for the checkout", or let an agent explore your app through Playwright MCP and turn what it did into a spec. Healer agents even fix failing tests for you.

Here is the kind of test they write:

import { test, expect } from "@playwright/test";

test("places an order", async ({ page }) => {
  await page.goto("/");
  await page.getByLabel("Name").fill("Ada");
  await page.getByRole("button", { name: "Place order" }).click();
  await expect(page.getByRole("status")).toContainText("Order placed");
});

It's a good test: role-based locators and a web-first assertion. It passes.

It also passes when the order API returns 500 and stores nothing, as long as the page still shows "Order placed". The agent checked what it could see. Nobody told it to check what the server did, and agents almost never add a network assertion on their own.

That matters more with agents than with people. A human who wrote the checkout code might remember the API can fail. An agent generating fifty tests from what's on screen won't, and a healer agent that "fixes" a red test by updating the expected text can make things worse. The suite goes green while the backend is broken.

Change the import, in the tests the agent wrote or in the template it writes from:

import { test, expect } from "@vedhajs/playwright-test";

Nothing else changes. Same test, same broken app:

  ✓  1 1-playwright.spec.ts › places an order (import from @playwright/test)
  ✘  2 2-vedha.spec.ts › places an order (import from @vedhajs/playwright-test)

    Error: Vedha guard: the test's own checks passed, but the app reported a problem:
      - POST /api/orders returned 500

On a healthy app, both pass.

The failure message is also written for whoever reads it next, human or agent: what the app did, and how to accept it if it was expected. A healer agent gets a real reason instead of a green run.

@vedhajs/playwright-test re-exports Playwright's test and expect with an automatic fixture ("guard mode") that watches the browser context while each test runs:

expect passed.vedha-guard.json. When a finding is expected, opt out per test or per endpoint:

test.use({ vedhaGuard: "strict" }); // 4xx writes and page errors fail too
test.use({ vedhaGuard: { ignore: ["/api/flaky", /\/v2\//], hosts: ["api.other-domain.com"] } });
test.use({ vedhaGuard: false }); // off, for a test that expects a 500

It is still Playwright's runner, so retries, sharding, reporters and traces work as before.

Put it where the agent will pick it up:

CLAUDE.md, AGENTS.md, Copilot instructions): "Import test and @vedhajs/playwright-test, not @playwright/test." Guard mode is for the tests an agent writes. When the agent is the tester, driving the browser live and reporting pass or fail, Vedha also ships an MCP server that checks every step the same way. In a small pilot on held-out tasks (one model, 3 trials per task), the agent missed 9 seeded bugs with Vedha's MCP server against 26 with Playwright MCP. That's a separate setup from guard mode, covered in an earlier post.

waitForResponse assertion catches this too. Guard mode is the safety net for the many tests, agent-written or not, that don't have one.

npm install -D @vedhajs/playwright-test

Change the import in one agent-written spec and run it. If a test that used to pass now fails, look at the request it names. Either the agent's test was hiding a bug, or it's an expected error to add to ignore.

Repo, demo and benchmark data: https://github.com/HMAKT99/VEDHA (Apache-2.0)

I'd like to hear what it finds in agent-written suites, and about any false alarm.