{"slug": "360-green-runs-and-one-test-that-would-pass-with-the-server-down", "title": "360 green runs, and one test that would pass with the server down", "summary": "A developer gave an AI a Playwright + TypeScript test repository and ten test scenarios, producing 13 tests that passed 360 of 360 runs with retries disabled. Running the same UI tests against SauceDemo's deliberately buggy demo users exposed failures tied to each seeded bug, though one generated test would still pass with the server down, showing stability alone doesn't prove a test can fail.", "body_md": "I gave an AI a Playwright + TypeScript repo and ten test scenarios. It wrote 13 tests. I ran every test in the repo 20 times with retries off: **360 runs, 360 passed.**\n\nThen I read the tests. One of them would still pass if the server were down.\n\nThis article covers:\n\nEverything is in a public repo: [flakelab-dev/playwright-structure](https://github.com/flakelab-dev/playwright-structure). The AI's raw output is committed before any fix, so you can check every claim here.\n\nTwo public demo apps: [SauceDemo](https://www.saucedemo.com) for UI and [Restful Booker](https://restful-booker.herokuapp.com/apidoc/index.html) for API.\n\n```\nconfig/environments.ts   URLs per environment, picked with TEST_ENV\nsrc/\n  pages/                 Page objects: locators and actions, no business assertions\n  api/                   Typed API client + Zod schemas\n  fixtures/              Wires pages, clients and auth together\n  data/                  Test data builders\ntests/\n  ui/                    SauceDemo tests\n  api/                   Restful Booker tests\nplaywright.config.ts     Two projects, \"ui\" and \"api\"; retries off\n```\n\nFour decisions matter most.\n\n**1. Fixtures do the setup. Tests only declare what they need.** A test that asks for `loggedInPage` gets a browser that is already logged in:\n\n``` js\ntest('adding a product updates the cart badge', async ({ loggedInPage }) => {\n  await loggedInPage.addToCart('Sauce Labs Backpack');\n  await expect(loggedInPage.cartBadge).toHaveText('1');\n});\n```\n\n**2. The API client checks every response.** Methods like `createBooking()` validate the body with a Zod schema, and a failed call reports the URL, status and body:\n\n```\nasync function jsonOrThrow(response: APIResponse): Promise<unknown> {\n  if (!response.ok()) {\n    throw new Error(`${response.url()} returned ${response.status()}: ${await response.text()}`);\n  }\n  return response.json();\n}\n```\n\nMethods ending in `Raw` return the response untouched, for tests about status codes.\n\n**3. Test data is unique per run.** `buildBooking()` adds a random suffix to the last name, so parallel tests can't collide on a shared public API.\n\n**4. Retries are off.** Retries hide flakiness, and measuring flakiness is the point. The CI workflow can repeat the whole suite from the Actions tab.\n\nOne more thing about structure: it is also what you give the AI. The AI doesn't read your mind, it reads your repo. Keep that in mind for later.\n\nThe prompt listed five UI scenarios (sorting, cart, checkout, form errors, logout) and five API scenarios (PUT, PATCH, filter, invalid payload, wrong credentials). It ended with:\n\n```\nRules: no hard waits, no retries, each test independent.\nDo not run the tests; only write them.\n```\n\nWhy \"do not run\"? If the AI runs its tests, it fixes them until they pass, and I lose the first draft. I wanted to measure that draft.\n\n```\nnpx playwright test --repeat-each=20\n```\n\nThe 13 generated tests plus the 5 already in the repo, 20 times each: 360 runs, 360 passed.\n\nThat proves the tests are stable. It doesn't prove they are useful. A test that never fails might simply be unable to fail.\n\nSauceDemo has demo users with deliberate bugs. The `loggedInPage` fixture reads the user from an environment variable, so I ran the same 7 UI tests as each user, 3 times each:\n\n```\nUI_USER=problem_user npx playwright test tests/ui/ai- --project=ui --repeat-each=3\n```\n\n| User | Passed | What failed | \n|---|---|---|\n| standard_user | 21/21 | Nothing (control) | \n| problem_user | 9/21 | Sorting doesn't sort; the last name field ignores input, so checkout breaks | \n| error_user | 12/21 | Sorting doesn't sort; checkout can't finish; a missing last name isn't reported | \n| visual_user | 18/21 | The prices shown after sorting are not the prices shown before | \n| performance_glitch_user | 17/21 | Not a bug in the app's logic: login is slow (see below) | \n\nI checked every failure: each one came from the seeded bug, not from a mistake in the test. The tests that stayed green did so because the bug wasn't on their path (logout works for every user).\n\nThis is the result I trust more than 360/360. **The generated UI tests catch real bugs.**\n\n`performance_glitch_user` takes over 5 seconds to log in. Look at what that did:\n\n| Test | Passed | \n|---|---|\n| Sorting | 0/3 | \n| Logout | 2/3 | \n| Cart and checkout (5) | 15/15 | \n\nSame slowness, three different outcomes. The difference is the first step after login:\n\n`expect(loggedInPage.title).toHaveText('Products')`. Assertions give up after 5 seconds by default.` addToCart()`. Actions have no timeout of their own by default, so they keep waiting.\nLogout passed 2 times out of 3. That's what a flaky test looks like the first time you meet it. With retries on, it would have been green and the slow login would have gone unnoticed.\n\nBefore the mistakes, credit where it's due. Some of these I didn't ask for.\n\n`Raw` naming, and a data builder like the one already there.`Flake-3f9a1c2e`, so other bookings on the shared public API can't match.\n\n```\n// Guard: if the default order were already ascending, the test would prove nothing.\nexpect(pricesBefore.length).toBeGreaterThan(1);\nexpect(pricesBefore).not.toEqual(expected);\n```\n\nScenario: \"Creating a booking with an invalid payload is rejected.\"\n\nRestful Booker answers a booking with missing fields with **500**. A validation error should be 400; a 500 means the server failed. The AI knew this, and wrote:\n\n```\n// Restful Booker answers 500 here rather than 400; the test checks rejection, not the code.\nexpect(response.ok(), `expected rejection, got ${response.status()}`).toBe(false);\nexpect(response.status()).toBeGreaterThanOrEqual(400);\n```\n\nThis passes with 400, with 500, and with 503 Service Unavailable. **If the service were down, this test would still pass.** That's the test in the title.\n\nThe AI didn't hide the bug, since it left a comment. But it wrote the assertion so the bug fits, and the test can no longer fail for the reason that matters.\n\nScenario: \"Authenticating with wrong credentials fails.\"\n\n```\nawait expect(bookingClient.getToken(apiAdmin.username, 'wrong-password')).rejects.toThrow();\n```\n\n`toThrow()` with no message accepts any error: a network failure, a timeout, a typo in the URL.\n\nAnd here's why it passed. Restful Booker answers wrong credentials with **200** and `{\"reason\":\"Bad credentials\"}`. The client tried to read a token from that body, the schema check failed, and that error made the test pass. It passed because of a parsing error, not because the API rejected the login.\n\nThe PUT, PATCH and filter tests create bookings and never delete them. That's about 60 abandoned bookings per 20-run session, on a public service that other people use.\n\nThe test I wrote by hand had the same gap. The AI followed the pattern it was given, including the bad part.\n\nThat's the lesson I keep from this experiment: **AI multiplies your repo's conventions, good and bad.** Fix the base before you generate on top of it.\n\nThe same locators (page title, item names, cart badge) are defined in three page objects, and four checkout tests repeat the same setup. They aren't bugs, so I left them for a later refactor.\n\nSauceDemo has no documentation. Every expected value in the UI tests (the error messages, the empty cart after checkout, the text after logout) came from what the app does today.\n\nWhen I reviewed them, I got stuck on one question: if the app has a bug today, should the test adapt to it, or fail? And based on what?\n\nWithout a spec, these tests are **characterization tests**: they detect *change*, not *correctness*. That's still useful (the seeded bugs prove it), but it's worth naming. I marked all 7 UI tests \"⚠️ Oracle\": the expected behavior isn't backed by a specification.\n\nThe API has docs, so the review was clearer. PUT, PATCH and filter match the docs. The wrong-credentials response (200 with a `reason`) isn't described in the docs at all, and the test treats it as correct.\n\nFinal count for the 13 tests:\n\n| Verdict | Tests | \n|---|---|\n| ✅ Good as is | 3 | \n| ⚠️ Oracle: expected value not backed by a spec | 8 | \n| ⚠️ Weak: wouldn't catch the failure | 1 | \n| ❌ Wrong | 1 | \n\nThe AI did what we all do in a hurry: it looked at current behavior and wrote it down as expected behavior. At least it listed its assumptions. **Read that list: it's where the oracle problems are.**\n\nEach fix is a separate commit on top of the AI's output, so you can compare before and after.\n\n**Invalid payload** ([bed0c5d](https://github.com/flakelab-dev/playwright-structure/commit/bed0c5d)): assert the exact status and record the bug.\n\n```\ntest.info().annotations.push({\n  type: 'known bug',\n  description: 'Returns 500 instead of 400 for missing fields',\n});\nexpect(response.status()).toBe(500);\n```\n\nNow the test fails if the service is down (503), and it fails if the bug gets fixed (400), which is exactly when you want to update it. The annotation shows up in the HTML report, so the bug stays visible.\n\nI considered `test.fail()` and rejected it: it treats *any* failure as expected, including an outage. Same problem as before.\n\n**Wrong credentials** ([c272114](https://github.com/flakelab-dev/playwright-structure/commit/c272114)): the client now throws `Authentication failed: <reason>` when the API rejects the login, and the test asserts that exact message.\n\n**Cleanup** ([5b5ce0b](https://github.com/flakelab-dev/playwright-structure/commit/5b5ce0b)): the `bookingClient` fixture deletes every booking the test created, after the test ends. To check it, a temporary test created a booking; after the run, a GET for that booking returned 404.\n\nAsk these four questions about every test:\n\nIn this experiment, questions 1 and 2 found every real problem. Question 4 is the one the AI already handles well.\n\nOne tip from my own review: read the support code before judging a test. I flagged a missing status check in the API tests, and it was already there, in the client.\n\n```\ngit clone https://github.com/flakelab-dev/playwright-structure\ncd playwright-structure\nnpm ci\nnpx playwright install chromium\nnpm test\n```\n\nRun `npm run test:repeat` to measure it yourself, or set `UI_USER` to one of the broken users and watch the tests catch the bugs.\n\nHow do you review AI-generated tests today? I'd like to hear what you check first.\n\nI publish one measured experiment like this at a time. If you want the next one by email: [flakelab.dev/subscribe](https://flakelab.dev/subscribe/)", "url": "https://wpnews.pro/news/360-green-runs-and-one-test-that-would-pass-with-the-server-down", "canonical_source": "https://dev.to/flakelab_dev/360-green-runs-and-one-test-that-would-pass-with-the-server-down-3hco", "published_at": "2026-10-27 04:36:30+00:00", "updated_at": "2026-10-27 04:42:40.144203+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "ai-agents"], "entities": ["Playwright", "SauceDemo", "Restful Booker", "GitHub", "TypeScript", "Zod"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/360-green-runs-and-one-test-that-would-pass-with-the-server-down", "markdown": "https://wpnews.pro/news/360-green-runs-and-one-test-that-would-pass-with-the-server-down.md", "text": "https://wpnews.pro/news/360-green-runs-and-one-test-that-would-pass-with-the-server-down.txt", "jsonld": "https://wpnews.pro/news/360-green-runs-and-one-test-that-would-pass-with-the-server-down.jsonld"}}