My agent has two Playwrights

A developer at Appwrite analyzed 1,929 Playwright MCP browser calls across 62 Claude Code sessions from 17 September to 17 October and found that almost none ran the existing test suite; instead, 1,209 calls in 34 sessions were used for visual verification of UI, docs and release changes, with screenshots (281) and localhost navigations (209) dominating. The agent used the browser to catch issues no assertion covered, including a black logo on a near-black OAuth consent card and AVIF cover images that X would not render as share cards, fixing the latter with a JPEG conversion route for og:image and twitter:image. The developer argues the test suite and Playwright MCP answer different questions — regression versus whether newly built work looks right — and should be treated as two separate tools.

My agent made 1,929 browser calls through Playwright MCP last month, and almost none of them were tests. That surprised me, because I do have tests. The Appwrite console has a playwright.config.ts , CI runs four Console E2E jobs on every pull request, and I spent part of the month fixing flakes in them. None of the 1,929 calls, spread over 62 sessions in my Claude Code transcripts between 17 September and 17 October, ran that suite. They used the browser for something else. So my agent has two Playwrights, and I've stopped treating them as one tool. The test suite answers one question: did something that used to work stop working? It runs headless in CI, it's mostly deterministic, and it doesn't care what anything looks like. Playwright MCP answers a different one: does the thing I just built look right? Nobody has written an assertion for that, because an hour ago the thing didn't exist. The agent opens a real browser, goes to the page, clicks through it, reads it back and takes a picture. Interactive version on the original post https://www.chiragaggarwal.tech/blog/my-agent-has-two-playwrights . Screenshots are the second most common call overall. In the sessions where the agent was checking its own work, they're the most common: 281 screenshots against 226 navigations, and 209 of those navigations went to localhost or a local file. It isn't browsing the web. It's looking at what it just made. browser evaluate is mostly reading, not poking. Of its 304 calls plus 188 run code unsafe calls, 175 use querySelectorAll and 151 read innerText . A handful flip the dark class on <html so the next screenshot shows the other theme. This is what one of those pictures looks like. When Appwrite added function intervals as an alternative to cron, the agent built the console picker, then opened it in the browser and screenshotted every state before the PR went up. The interval picker, checked by the agent in a local console before review 14319 | What the session was for | Sessions | Calls | |---|---|---| | Checking a UI, docs or release change | 34 | 1,209 | | Chores in logged-in web apps | 23 | 553 | | Research | 5 | 167 | Nearly two thirds is verification. The rest is a different story, which I'll come back to. I pasted a screenshot of the Appwrite CLI's device sign-in screen and typed four words: "the logo looks off, debug why". The logo was a black mark sitting on a near-black card. The asset was fine. Every official app logo in that storage bucket is black on transparent, on purpose. The marketplace already knew this: its logo component puts the image on a muted tile and inverts it in dark mode. The OAuth consent screens just didn't use that component. The fix was to move it into a shared AppLogo and use it in both places. The part I care about came after. I asked the agent to check the other OAuth clients too, and it used the console's debug preview route to render the consent and result screens for Claude Code, Cursor, Codex, Claude Desktop and Gemini CLI in dark mode, then did one pass in light mode. That took 19 Playwright calls, with no errors, and produced eight screenshots. No test in the suite would have caught this, because no test checks that a logo is visible against its background. If I wrote one now, it would only protect against the last bug. The passkeys announcement showed a grey placeholder when shared on X. The agent found the cause quickly: the cover image was AVIF, and X only renders JPG, PNG, WEBP and GIF for cards. Every blog and changelog cover in the repo is AVIF, all 538 of them. The fix was a route that converts the cover to JPEG for og:image and twitter:image only. Then I asked for "pure reproduction and fix verification screenshots in pr desc". The agent built two small pages showing the share card before and after, resized the browser to the card's size and took a screenshot of each. Its first attempt failed because Playwright MCP blocks file:// URLs, so it served the pages over local HTTP instead. The two screenshots that went into the PR description Then the suite took over. The review bot flagged two problems, a type-baseline check broke because of the new route, and the four Console E2E jobs ran. Once all 20 checks were green, 14622 https://github.com/appwrite/appwrite/pull/14622 was ready. Both Playwrights were in that PR: one showed the reviewer what changed, the other showed that nothing else did. The biggest session of the month used 183 Playwright calls. I'd asked the agent to "do end to end testing of release notes, especially frontend side" before shipping Appwrite 2.3.0. It built the 2.3.0 image locally, installed 2.2.0 in a separate Docker Compose project so my dev data stayed safe, seeded it, and upgraded. migrate failed: projects that already existed never got the new pushLedger collection. After fixing that it found a second gap: existing topics were missing the new qos , expiry and sequence fields. Both fixes shipped with regression tests and went into the release branch before anyone upgraded. To be fair, the browser didn't find the migration bug. migrate exiting with code 1 did. What Playwright did was walk the console against the upgraded install and check each claim in the release notes, one screen at a time. Still, nothing in the test suite runs an upgrade. Release QA used to be manual QA. Now it's manual QA I hand to an agent. Where the browser does earn its keep is reproducing what users report. After 2.4.0, someone in the community said the upgrade wizard ignored their custom ports. I pasted the feedback and asked the agent to confirm each issue before fixing anything. It installed a stack on ports 8080 and 8443 with HTTPS on, opened the upgrade wizard, and screenshotted it offering 80, 443 and HTTPS off. Then it fixed the bug and took the same screenshot again. Nobody writes a test for this kind of bug until it bites someone. Same install, two upgrade wizards 14345 This is the habit I care most about. A reviewer can't run your branch, but they can look at it. The loop has four steps: passkeys-off.png , passkeys-on.png , cli-device-light.png . 317 of the month's 332 screenshots had a name like that. Before ./before.png and runs gh pr create --attach or gh pr edit --attach . Recent versions of gh upload each file to GitHub and rewrite the link to point at the upload. .playwright-mcp/ and any loose PNGs so none of it gets committed. A good pair needs no explanation. When self-hosted consoles stopped offering OAuth and passkey sign-in, the PR showed the page before and after, and the reviewer could see the change in a second: Self-hosted console sign-in, before and after 14434 Dark mode gets its own screenshot, because that's where things like the invisible logo hide: Passkey sign-in in dark mode, with the last-used badge 14400 I didn't start with --attach . Over the month, screenshots reached my PRs three ways: committed to an orphan branch on my fork and linked through raw.githubusercontent.com , committed to a repo as docs assets, and attached. A reviewer caught the problem with the first one: "Screenshot evidence disappears after merge. Screenshots reference an assets branch that the instructions then delete." If the screenshots live on a branch you delete, the evidence goes with it. Attachments stay. 32 of the 53 PRs with screenshots now use them. And 53 out of 904 is the honest number. The rate depends on the repo. In the two public Appwrite repos where most of my UI work lands: | Repo | PRs | With screenshots | |---|---|---| | appwrite/vibes | 39 | 11 28% | | appwrite/appwrite | 224 | 25 11% | Backend-heavy repos sit far lower, which is partly fair, since there's often nothing to look at. But there's a less flattering reason too: I asked for screenshots by hand about 30 times this month "add screenshots in pr desc", "before and after screenshot in pr desc", "show screenshots of ur testing in pr desc" . The habit only happened when I remembered to type it. So I made it a rule. This is now in my global CLAUDE.md : If a PR changes anything a reviewer can see UI, rendered docs, emails, OG cards, dashboards , verify it with Playwright MCP and put screenshots in the PR description without being asked: before and after for fixes, light and dark for UI. Name each file for what it shows, reference it as alt ./name.png in the body, and upload with gh pr create --attach / gh pr edit --attach . Never host screenshots on a branch that gets deleted, and never commit .playwright-mcp/ or stray PNGs. These had nothing to do with UI work. Help Scout, the university's Brightspace, the Google Cloud and AWS consoles, an SMS provider's dashboard. These are tools I use daily that either have no API or none I have access to. The pattern is always the same: "use playwright to login to helpscout, i want you to check some tickets". I log in myself in the browser window, and the agent reads. The browser becomes an API for software that doesn't offer one. That's a different post, but it's why the server stays installed even in weeks when I'm not building UI. file:// is blocked. The suite catches what broke. The screenshots show the reviewer what's new. Originally published on chiragaggarwal.tech https://www.chiragaggarwal.tech/blog/my-agent-has-two-playwrights .