360 green runs, and one test that would pass with the server down

A developer gave an AI a Playwright + TypeScript test repository and ten test scenarios, producing 13 tests that passed 360 of 360 runs with retries disabled. Running the same UI tests against SauceDemo's deliberately buggy demo users exposed failures tied to each seeded bug, though one generated test would still pass with the server down, showing stability alone doesn't prove a test can fail.

I gave an AI a Playwright + TypeScript repo and ten test scenarios. It wrote 13 tests. I ran every test in the repo 20 times with retries off: 360 runs, 360 passed. Then I read the tests. One of them would still pass if the server were down. This article covers: Everything is in a public repo: flakelab-dev/playwright-structure https://github.com/flakelab-dev/playwright-structure . The AI's raw output is committed before any fix, so you can check every claim here. Two public demo apps: SauceDemo https://www.saucedemo.com for UI and Restful Booker https://restful-booker.herokuapp.com/apidoc/index.html for API. config/environments.ts URLs per environment, picked with TEST ENV src/ pages/ Page objects: locators and actions, no business assertions api/ Typed API client + Zod schemas fixtures/ Wires pages, clients and auth together data/ Test data builders tests/ ui/ SauceDemo tests api/ Restful Booker tests playwright.config.ts Two projects, "ui" and "api"; retries off Four decisions matter most. 1. Fixtures do the setup. Tests only declare what they need. A test that asks for loggedInPage gets a browser that is already logged in: js test 'adding a product updates the cart badge', async { loggedInPage } = { await loggedInPage.addToCart 'Sauce Labs Backpack' ; await expect loggedInPage.cartBadge .toHaveText '1' ; } ; 2. The API client checks every response. Methods like createBooking validate the body with a Zod schema, and a failed call reports the URL, status and body: async function jsonOrThrow response: APIResponse : Promise<unknown { if response.ok { throw new Error ${response.url } returned ${response.status }: ${await response.text } ; } return response.json ; } Methods ending in Raw return the response untouched, for tests about status codes. 3. Test data is unique per run. buildBooking adds a random suffix to the last name, so parallel tests can't collide on a shared public API. 4. Retries are off. Retries hide flakiness, and measuring flakiness is the point. The CI workflow can repeat the whole suite from the Actions tab. One more thing about structure: it is also what you give the AI. The AI doesn't read your mind, it reads your repo. Keep that in mind for later. The prompt listed five UI scenarios sorting, cart, checkout, form errors, logout and five API scenarios PUT, PATCH, filter, invalid payload, wrong credentials . It ended with: Rules: no hard waits, no retries, each test independent. Do not run the tests; only write them. Why "do not run"? If the AI runs its tests, it fixes them until they pass, and I lose the first draft. I wanted to measure that draft. npx playwright test --repeat-each=20 The 13 generated tests plus the 5 already in the repo, 20 times each: 360 runs, 360 passed. That proves the tests are stable. It doesn't prove they are useful. A test that never fails might simply be unable to fail. SauceDemo has demo users with deliberate bugs. The loggedInPage fixture reads the user from an environment variable, so I ran the same 7 UI tests as each user, 3 times each: UI USER=problem user npx playwright test tests/ui/ai- --project=ui --repeat-each=3 | User | Passed | What failed | |---|---|---| | standard user | 21/21 | Nothing control | | problem user | 9/21 | Sorting doesn't sort; the last name field ignores input, so checkout breaks | | error user | 12/21 | Sorting doesn't sort; checkout can't finish; a missing last name isn't reported | | visual user | 18/21 | The prices shown after sorting are not the prices shown before | | performance glitch user | 17/21 | Not a bug in the app's logic: login is slow see below | I checked every failure: each one came from the seeded bug, not from a mistake in the test. The tests that stayed green did so because the bug wasn't on their path logout works for every user . This is the result I trust more than 360/360. The generated UI tests catch real bugs. performance glitch user takes over 5 seconds to log in. Look at what that did: | Test | Passed | |---|---| | Sorting | 0/3 | | Logout | 2/3 | | Cart and checkout 5 | 15/15 | Same slowness, three different outcomes. The difference is the first step after login: expect loggedInPage.title .toHaveText 'Products' . Assertions give up after 5 seconds by default. addToCart . Actions have no timeout of their own by default, so they keep waiting. Logout passed 2 times out of 3. That's what a flaky test looks like the first time you meet it. With retries on, it would have been green and the slow login would have gone unnoticed. Before the mistakes, credit where it's due. Some of these I didn't ask for. Raw naming, and a data builder like the one already there. Flake-3f9a1c2e , so other bookings on the shared public API can't match. // Guard: if the default order were already ascending, the test would prove nothing. expect pricesBefore.length .toBeGreaterThan 1 ; expect pricesBefore .not.toEqual expected ; Scenario: "Creating a booking with an invalid payload is rejected." Restful Booker answers a booking with missing fields with 500 . A validation error should be 400; a 500 means the server failed. The AI knew this, and wrote: // Restful Booker answers 500 here rather than 400; the test checks rejection, not the code. expect response.ok , expected rejection, got ${response.status } .toBe false ; expect response.status .toBeGreaterThanOrEqual 400 ; This passes with 400, with 500, and with 503 Service Unavailable. If the service were down, this test would still pass. That's the test in the title. The AI didn't hide the bug, since it left a comment. But it wrote the assertion so the bug fits, and the test can no longer fail for the reason that matters. Scenario: "Authenticating with wrong credentials fails." await expect bookingClient.getToken apiAdmin.username, 'wrong-password' .rejects.toThrow ; toThrow with no message accepts any error: a network failure, a timeout, a typo in the URL. And here's why it passed. Restful Booker answers wrong credentials with 200 and {"reason":"Bad credentials"} . The client tried to read a token from that body, the schema check failed, and that error made the test pass. It passed because of a parsing error, not because the API rejected the login. The PUT, PATCH and filter tests create bookings and never delete them. That's about 60 abandoned bookings per 20-run session, on a public service that other people use. The test I wrote by hand had the same gap. The AI followed the pattern it was given, including the bad part. That's the lesson I keep from this experiment: AI multiplies your repo's conventions, good and bad. Fix the base before you generate on top of it. The same locators page title, item names, cart badge are defined in three page objects, and four checkout tests repeat the same setup. They aren't bugs, so I left them for a later refactor. SauceDemo has no documentation. Every expected value in the UI tests the error messages, the empty cart after checkout, the text after logout came from what the app does today. When I reviewed them, I got stuck on one question: if the app has a bug today, should the test adapt to it, or fail? And based on what? Without a spec, these tests are characterization tests : they detect change , not correctness . That's still useful the seeded bugs prove it , but it's worth naming. I marked all 7 UI tests "⚠️ Oracle": the expected behavior isn't backed by a specification. The API has docs, so the review was clearer. PUT, PATCH and filter match the docs. The wrong-credentials response 200 with a reason isn't described in the docs at all, and the test treats it as correct. Final count for the 13 tests: | Verdict | Tests | |---|---| | ✅ Good as is | 3 | | ⚠️ Oracle: expected value not backed by a spec | 8 | | ⚠️ Weak: wouldn't catch the failure | 1 | | ❌ Wrong | 1 | The AI did what we all do in a hurry: it looked at current behavior and wrote it down as expected behavior. At least it listed its assumptions. Read that list: it's where the oracle problems are. Each fix is a separate commit on top of the AI's output, so you can compare before and after. Invalid payload bed0c5d https://github.com/flakelab-dev/playwright-structure/commit/bed0c5d : assert the exact status and record the bug. test.info .annotations.push { type: 'known bug', description: 'Returns 500 instead of 400 for missing fields', } ; expect response.status .toBe 500 ; Now the test fails if the service is down 503 , and it fails if the bug gets fixed 400 , which is exactly when you want to update it. The annotation shows up in the HTML report, so the bug stays visible. I considered test.fail and rejected it: it treats any failure as expected, including an outage. Same problem as before. Wrong credentials c272114 https://github.com/flakelab-dev/playwright-structure/commit/c272114 : the client now throws Authentication failed: <reason when the API rejects the login, and the test asserts that exact message. Cleanup 5b5ce0b https://github.com/flakelab-dev/playwright-structure/commit/5b5ce0b : the bookingClient fixture deletes every booking the test created, after the test ends. To check it, a temporary test created a booking; after the run, a GET for that booking returned 404. Ask these four questions about every test: In this experiment, questions 1 and 2 found every real problem. Question 4 is the one the AI already handles well. One tip from my own review: read the support code before judging a test. I flagged a missing status check in the API tests, and it was already there, in the client. git clone https://github.com/flakelab-dev/playwright-structure cd playwright-structure npm ci npx playwright install chromium npm test Run npm run test:repeat to measure it yourself, or set UI USER to one of the broken users and watch the tests catch the bugs. How do you review AI-generated tests today? I'd like to hear what you check first. I publish one measured experiment like this at a time. If you want the next one by email: flakelab.dev/subscribe https://flakelab.dev/subscribe/