AI agents are good at clicking through a web app. They're bad at knowing whether it actually worked. An agent sees "Order placed" on the screen and reports success, even when the API behind it returned 500 and saved nothing.
This tutorial sets up an AI agent that tests your app in a real browser and checks what the server did. You'll go from zero to a test you can keep in CI in about five minutes.
We'll use Vedha, an open-source MCP server built on Playwright. It works with Claude Code, Cursor, VS Code and any other MCP client.
In Claude Code:
claude mcp add vedha -- npx -y @vedhajs/mcp --allow-domain localhost --out ./vedha-out
For Cursor, VS Code or Claude Desktop, add this to your MCP config:
{
"mcpServers": {
"vedha": {
"command": "npx",
"args": ["-y", "@vedhajs/mcp", "--allow-domain", "localhost", "--out", "./vedha-out"]
}
}
}
Two things to note:
--allow-domain localhost is required: the agent can only browse the domains you list. Point it at your dev server, not the open web.npx playwright install chromium once.
Optional: install the agent skill, which teaches the agent how to test with Vedha (declare checks, mark writes, never retry an unknown outcome):
npx skills add HMAKT99/VEDHA
Start your app, then ask:
Test the checkout on localhost:3000: place an order for Ada and confirm it worked.
The agent opens the page and works through the flow with vedha_act. Each call is one browser action, and it can say what success looks like. Here is the call an agent sends to fill the form and submit it:
{
"op": "fill", "intent": "enter name", "target": { "label": "Name" }, "value": "Ada",
"then": [{
"op": "click", "intent": "place the order",
"target": { "role": "button", "name": "Place order" },
"mutating": true,
"expect": { "text_contains": "Order placed" }
}]
}
mutating: true tells Vedha the click changes data, so it should check what the server did, not just the page.
On a healthy app, this is what the agent gets back (real output, trimmed):
step 1 (step-2): verified
result: verified
effect: server confirmed (POST /api/orders 201; read-back /api/orders confirms POST /api/orders)
changed on the page (3 added, 3 removed):
+ status: Order placed
Vedha saw the 201, then read the order list back to confirm the order is really there.
Now the same test against a broken backend: the API returns 500, but the frontend still shows "Order placed". This is what the agent gets back (real output, trimmed):
step 1 (step-2): verified
result: failed (health_check_failed): declared checks passed, but the page reported: POST /api/orders returned 500
hint: treat this action as failed: the server or page reported an error even if the page looks fine
requests: POST /api/orders 500
+ status: Order placed
open failure: "place the order" - declared checks passed, but the page reported:
POST /api/orders returned 500. Do not report success unless a later step fixed it.
"Order placed" is on the page, and the agent's own check passed. Vedha still reports the step as failed, says why, and reminds the agent not to claim success. A browser tool that only checks the screen would have reported a pass.
Two more things Vedha does here without being asked:
outcome_unknown and vedha_probe (for example GET /api/orders).
Once a flow is verified, ask the agent:
Save that as a Playwright test.
It calls vedha_promote, which writes a plain @playwright/test spec from the verified steps. This is the real output for the order flow above:
// Generated by Vedha from an explored session. Review before committing.
import { test, expect } from "@playwright/test";
test("places an order", async ({ page }) => {
await test.step("open the shop", async () => {
await page.goto("/");
});
await test.step("enter name", async () => {
await page.getByLabel("Name", { exact: true }).fill("Ada");
});
await test.step("place the order", async () => {
await page.getByRole("button", { name: "Place order", exact: true }).click();
await expect(page.locator("body")).toContainText("Order placed");
await expect.poll(async () => {
const res = await page.request.get("/api/orders");
const body = await res.json();
const items = Array.isArray(body) ? body : [body];
return items.some((it) =>
Object.entries({ name: "Ada", qty: 1, product: "Widget" }).every(([k, val]) => it[k] === val),
);
}).toBe(true);
});
});
Notice the last assertion: because Vedha confirmed the order on the server during the run, the generated test checks the server too, not just the text. It also lists its assumptions at the top, so review it before committing.
Commit the spec and run it with your normal Playwright setup. One more line makes the rest of your suite safer too:
npm install -D @vedhajs/playwright-test
js
import { test, expect } from "@vedhajs/playwright-test"; // instead of "@playwright/test"
With that import, any test whose own checks pass while the app answers with a 5xx, or a write request fails, fails instead of going green. That covers tests written by your agent and by people. (More on guard mode here.)
How much difference does it make? In a small pilot (one model, held-out tasks, 3 trials each), the agent missed 9 seeded bugs with Vedha against 26 with Playwright MCP, using about 30% fewer tokens. Details and limits here.
git clone https://github.com/HMAKT99/VEDHA && cd VEDHA
pnpm install && pnpm build
npx tsx examples/demo/run.ts
Repo, docs and benchmark data: https://github.com/HMAKT99/VEDHA (Apache-2.0)
If you try it on your app, I'd like to hear what it caught, and anything it got wrong.