My AI-Generated Tests Kept Passing for the Wrong Reason

A developer found that AI-generated Playwright tests passed for the wrong reason, failing to verify critical backend behavior. The tests checked UI success messages but not actual outcomes like email delivery or token expiration, leading to undetected bugs. The developer now rewrites assertions manually to ensure tests validate real business logic.

I started using AI to generate Playwright test scaffolding a few months ago. Feed it a user flow, whether it's a login form or a multi-step checkout, and you get back a working spec in under a minute that runs and mostly passes on the first try. I asked a model to write a test for a password reset flow, and it built a spec that filled in the email field, clicked submit, and checked for a success message on screen. That passed every time I ran it, but it never checked whether an email actually went out, whether the link inside it worked, or whether that same link still worked after being used once. The page reported success no matter what the backend did, so the test just checked the page's opinion of itself. Before I touched it, the spec looked like this: await page.getByLabel 'Email' .fill testUser.email ; await page.getByRole 'button', { name: 'Send reset link' } .click ; await expect page.getByText 'Check your email' .toBeVisible ; All three lines passed, which is exactly the problem: none of them checked anything that actually mattered. Once I rewrote the assertions, it looked like this: await page.getByLabel 'Email' .fill testUser.email ; await page.getByRole 'button', { name: 'Send reset link' } .click ; await expect page.getByText 'Check your email' .toBeVisible ; const resetEmail = await getLatestEmail testUser.email ; const token = extractResetToken resetEmail.body ; await page.goto /reset?token=${token} ; await page.getByLabel 'New password' .fill 'NewPassword123 ' ; await page.getByRole 'button', { name: 'Reset password' } .click ; await expect page .toHaveURL '/login' ; await page.goto /reset?token=${token} ; await expect page.getByText 'This link has expired' .toBeVisible ; The expired-token check at the end is what caught a real bug: the first version of the reset endpoint let a token be reused as many times as someone wanted, with no expiration after it was used once. None of this makes AI bad at writing tests. It's good at the boilerplate, at generating locators and shaping a spec file enough that I'm editing instead of starting from nothing, but it can't decide which outcome the business actually depends on, because that context doesn't live in the DOM. A password reset link that never expires looks identical on screen to one that works the way it should, and the same gap shows up anywhere a UI can report success independent of what the backend actually did, checkout confirmations and invite emails included. I still generate the first draft of most new specs with a model, but I rewrite the assertions myself, against what the feature is actually supposed to guarantee. Jeff Thoensen is a Context-Driven QA Engineer focused on automation, API testing, and exploratory testing. Find more at jeffthoensen.com.