{"slug": "my-agent-has-two-playwrights", "title": "My agent has two Playwrights", "summary": "A developer at Appwrite analyzed 1,929 Playwright MCP browser calls across 62 Claude Code sessions from 17 September to 17 October and found that almost none ran the existing test suite; instead, 1,209 calls in 34 sessions were used for visual verification of UI, docs and release changes, with screenshots (281) and localhost navigations (209) dominating. The agent used the browser to catch issues no assertion covered, including a black logo on a near-black OAuth consent card and AVIF cover images that X would not render as share cards, fixing the latter with a JPEG conversion route for og:image and twitter:image. The developer argues the test suite and Playwright MCP answer different questions — regression versus whether newly built work looks right — and should be treated as two separate tools.", "body_md": "My agent made 1,929 browser calls through Playwright MCP last month, and almost none of them were tests.\n\nThat surprised me, because I do have tests. The Appwrite console has a `playwright.config.ts`, CI runs four Console E2E jobs on every pull request, and I spent part of the month fixing flakes in them. None of the 1,929 calls, spread over 62 sessions in my Claude Code transcripts between 17 September and 17 October, ran that suite. They used the browser for something else.\n\nSo my agent has two Playwrights, and I've stopped treating them as one tool.\n\nThe test suite answers one question: did something that used to work stop working? It runs headless in CI, it's mostly deterministic, and it doesn't care what anything looks like.\n\nPlaywright MCP answers a different one: does the thing I just built look right? Nobody has written an assertion for that, because an hour ago the thing didn't exist. The agent opens a real browser, goes to the page, clicks through it, reads it back and takes a picture.\n\n*Interactive version on [the original post](https://www.chiragaggarwal.tech/blog/my-agent-has-two-playwrights).*\n\nScreenshots are the second most common call overall. In the sessions where the agent was checking its own work, they're the most common: 281 screenshots against 226 navigations, and 209 of those navigations went to `localhost` or a local file. It isn't browsing the web. It's looking at what it just made.\n\n`browser_evaluate` is mostly reading, not poking. Of its 304 calls plus 188 `run_code_unsafe` calls, 175 use `querySelectorAll` and 151 read `innerText`. A handful flip the `dark` class on `<html>` so the next screenshot shows the other theme.\n\nThis is what one of those pictures looks like. When Appwrite added function intervals as an alternative to cron, the agent built the console picker, then opened it in the browser and screenshotted every state before the PR went up.\n\n*The interval picker, checked by the agent in a local console before review (#14319)*\n\n| What the session was for | Sessions | Calls | \n|---|---|---|\n| Checking a UI, docs or release change | 34 | 1,209 | \n| Chores in logged-in web apps | 23 | 553 | \n| Research | 5 | 167 | \n\nNearly two thirds is verification. The rest is a different story, which I'll come back to.\n\nI pasted a screenshot of the Appwrite CLI's device sign-in screen and typed four words: \"the logo looks off, debug why\". The logo was a black mark sitting on a near-black card.\n\nThe asset was fine. Every official app logo in that storage bucket is black on transparent, on purpose. The marketplace already knew this: its logo component puts the image on a muted tile and inverts it in dark mode. The OAuth consent screens just didn't use that component.\n\nThe fix was to move it into a shared `AppLogo` and use it in both places. The part I care about came after. I asked the agent to check the other OAuth clients too, and it used the console's debug preview route to render the consent and result screens for Claude Code, Cursor, Codex, Claude Desktop and Gemini CLI in dark mode, then did one pass in light mode. That took 19 Playwright calls, with no errors, and produced eight screenshots.\n\nNo test in the suite would have caught this, because no test checks that a logo is visible against its background. If I wrote one now, it would only protect against the last bug.\n\nThe passkeys announcement showed a grey placeholder when shared on X. The agent found the cause quickly: the cover image was AVIF, and X only renders JPG, PNG, WEBP and GIF for cards. Every blog and changelog cover in the repo is AVIF, all 538 of them.\n\nThe fix was a route that converts the cover to JPEG for `og:image` and `twitter:image` only. Then I asked for \"pure reproduction and fix verification screenshots in pr desc\". The agent built two small pages showing the share card before and after, resized the browser to the card's size and took a screenshot of each. Its first attempt failed because Playwright MCP blocks `file://` URLs, so it served the pages over local HTTP instead.\n\n*The two screenshots that went into the PR description*\n\nThen the suite took over. The review bot flagged two problems, a type-baseline check broke because of the new route, and the four Console E2E jobs ran. Once all 20 checks were green, [#14622](https://github.com/appwrite/appwrite/pull/14622) was ready. Both Playwrights were in that PR: one showed the reviewer what changed, the other showed that nothing else did.\n\nThe biggest session of the month used 183 Playwright calls. I'd asked the agent to \"do end to end testing of release notes, especially frontend side\" before shipping Appwrite 2.3.0.\n\nIt built the 2.3.0 image locally, installed 2.2.0 in a separate Docker Compose project so my dev data stayed safe, seeded it, and upgraded. `migrate` failed: projects that already existed never got the new `pushLedger` collection. After fixing that it found a second gap: existing topics were missing the new `qos`, `expiry` and `sequence` fields. Both fixes shipped with regression tests and went into the release branch before anyone upgraded.\n\nTo be fair, the browser didn't find the migration bug. `migrate` exiting with code 1 did. What Playwright did was walk the console against the upgraded install and check each claim in the release notes, one screen at a time. Still, nothing in the test suite runs an upgrade. Release QA used to be manual QA. Now it's manual QA I hand to an agent.\n\nWhere the browser does earn its keep is reproducing what users report. After 2.4.0, someone in the community said the upgrade wizard ignored their custom ports. I pasted the feedback and asked the agent to confirm each issue before fixing anything. It installed a stack on ports 8080 and 8443 with HTTPS on, opened the upgrade wizard, and screenshotted it offering 80, 443 and HTTPS off. Then it fixed the bug and took the same screenshot again. Nobody writes a test for this kind of bug until it bites someone.\n\n*Same install, two upgrade wizards (#14345)*\n\nThis is the habit I care most about. A reviewer can't run your branch, but they can look at it.\n\nThe loop has four steps:\n\n`passkeys-off.png`, `passkeys-on.png`, `cli-device-light.png`. 317 of the month's 332 screenshots had a name like that.`![Before](./before.png)` and runs `gh pr create --attach` or `gh pr edit --attach`. Recent versions of `gh` upload each file to GitHub and rewrite the link to point at the upload.`.playwright-mcp/` and any loose PNGs so none of it gets committed.\nA good pair needs no explanation. When self-hosted consoles stopped offering OAuth and passkey sign-in, the PR showed the page before and after, and the reviewer could see the change in a second:\n\n*Self-hosted console sign-in, before and after (#14434)*\n\nDark mode gets its own screenshot, because that's where things like the invisible logo hide:\n\n*Passkey sign-in in dark mode, with the last-used badge (#14400)*\n\nI didn't start with `--attach`. Over the month, screenshots reached my PRs three ways: committed to an orphan branch on my fork and linked through `raw.githubusercontent.com`, committed to a repo as docs assets, and attached. A reviewer caught the problem with the first one: *\"Screenshot evidence disappears after merge. Screenshots reference an assets branch that the instructions then delete.\"* If the screenshots live on a branch you delete, the evidence goes with it. Attachments stay. 32 of the 53 PRs with screenshots now use them.\n\nAnd 53 out of 904 is the honest number. The rate depends on the repo. In the two public Appwrite repos where most of my UI work lands:\n\n| Repo | PRs | With screenshots | \n|---|---|---|\n| `appwrite/vibes` | 39 | 11 (28%) | \n| `appwrite/appwrite` | 224 | 25 (11%) | \n\nBackend-heavy repos sit far lower, which is partly fair, since there's often nothing to look at. But there's a less flattering reason too: I asked for screenshots by hand about 30 times this month (\"add screenshots in pr desc\", \"before and after screenshot in pr desc\", \"show screenshots of ur testing in pr desc\"). The habit only happened when I remembered to type it.\n\nSo I made it a rule. This is now in my global `CLAUDE.md`:\n\nIf a PR changes anything a reviewer can see (UI, rendered docs, emails, OG cards, dashboards), verify it with Playwright MCP and put screenshots in the PR description without being asked: before and after for fixes, light and dark for UI. Name each file for what it shows, reference it as `![alt](./name.png)` in the body, and upload with `gh pr create --attach` / `gh pr edit --attach`. Never host screenshots on a branch that gets deleted, and never commit `.playwright-mcp/` or stray PNGs.\n\nThese had nothing to do with UI work. Help Scout, the university's Brightspace, the Google Cloud and AWS consoles, an SMS provider's dashboard. These are tools I use daily that either have no API or none I have access to.\n\nThe pattern is always the same: \"use playwright to login to helpscout, i want you to check some tickets\". I log in myself in the browser window, and the agent reads. The browser becomes an API for software that doesn't offer one. That's a different post, but it's why the server stays installed even in weeks when I'm not building UI.\n\n`file://` is blocked.\nThe suite catches what broke. The screenshots show the reviewer what's new.\n\n*Originally published on [chiragaggarwal.tech](https://www.chiragaggarwal.tech/blog/my-agent-has-two-playwrights).*", "url": "https://wpnews.pro/news/my-agent-has-two-playwrights", "canonical_source": "https://dev.to/chiragagg5k/my-agent-has-two-playwrights-5e6g", "published_at": "2026-10-26 12:36:32+00:00", "updated_at": "2026-10-26 12:43:24.118067+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools"], "entities": ["Appwrite", "Playwright", "Claude Code", "Cursor", "Codex", "Claude Desktop", "Gemini CLI", "X"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/my-agent-has-two-playwrights", "markdown": "https://wpnews.pro/news/my-agent-has-two-playwrights.md", "text": "https://wpnews.pro/news/my-agent-has-two-playwrights.txt", "jsonld": "https://wpnews.pro/news/my-agent-has-two-playwrights.jsonld"}}