{"slug": "how-to-review-ai-generated-tests-seven-checks-before-you-keep-them", "title": "How to Review AI-Generated Tests: Seven Checks Before You Keep Them", "summary": "A developer outlines seven checks for reviewing AI-generated tests before adding them to a suite, emphasizing that passing tests do not prove value unless they catch user-impacting failures. The checks include naming user risk, breaking the product to verify test failure, checking final results, varying data, reading failure messages, repeating runs, and deciding if the test would stop a bad release. The approach aims to keep only tests that protect important behavior.", "body_md": "Review AI-generated tests with seven checks. Name the user risk. Break the product. Check the final result. Change the data. Read the failure. Repeat the run. Then decide whether the test could stop a bad release.\n\nAI can write a clean test in seconds. The file may look finished. The names may sound correct. The test may even pass.\n\nNone of those facts prove value. A useful test catches a failure that matters to users. Your review must find that proof before the test joins your suite.\n\nI use seven checks for that decision.\n\nStart with the person who could get hurt. Write the risk in one sentence.\n\nFor example: \"A customer sees the wrong total and pays too much.\"\n\nAvoid risks like \"checkout may fail.\" That sentence does not name the damage. It also gives the test no clear target.\n\nAsk two questions:\n\nMoney, time, access, and trust are clear answers. \"The feature breaks\" is not.\n\nA test should fail when its protected behavior breaks. Prove that before keeping it.\n\nAI often writes a test that follows the correct steps. The test may never check the important result.\n\n``` js\ntest('customer can pay', async ({ page }) => {\n  await page.goto('/checkout')\n  await page.getByRole('button', { name: 'Pay' }).click()\n  await expect(page.getByText('Success')).toBeVisible()\n})\n```\n\nThis test checks a message. It does not check the charged amount.\n\nRemove the payment action or return the wrong total. The test must fail. A passing result means the test protects nothing useful.\n\nClicks are steps. Results are proof.\n\nThe test above clicks the right button. A stronger test checks the amount and payment record.\n\n```\nawait expect(page.getByTestId('order-total')).toHaveText('$120.00')\nawait expect(page.getByTestId('payment-status')).toHaveText('Paid')\n```\n\nAn assertion means a result check. Playwright provides [assertions that wait for results](https://playwright.dev/docs/test-assertions). The tool can wait. You still choose the right result.\n\nReview every assertion. Ask what user outcome it proves. Rewrite assertions that only confirm page activity.\n\nAI tends to generate a clean example. Real users bring missing, wrong, and extreme values.\n\nTest more than one amount. Include zero, a large value, and invalid text.\n\n``` js\nfor (const amount of ['0', '999999', 'wrong']) {\n  await page.getByLabel('Amount').fill(amount)\n  await page.getByRole('button', { name: 'Pay' }).click()\n  await expect(page.getByRole('alert')).toBeVisible()\n}\n```\n\nThe exact values depend on your product. The review question stays simple. Can one neat example hide a serious failure?\n\nRun the test against a broken result. Then read its message.\n\nAnother engineer should understand the problem without opening the whole file. Compare these messages:\n\n```\nExpected: \"$20.00\"\nReceived: \"$120.00\"\nTimeout after 30000ms\n```\n\nThe first message points to the product error. The second sends someone searching through logs.\n\nUse clear result checks and useful test names. A failure should shorten the investigation.\n\nUse the same input twice. You should get the same result.\n\nRepeated runs catch shared data and timing problems. They also expose tests that depend on another test.\n\nDo not accept a passing second run as proof. Compare both runs. Investigate any difference before keeping the test.\n\nFinish with one question: Would this test stop a bad release?\n\nName the release it could stop. For example: \"This test blocks checkout when totals are wrong.\"\n\nKeep the test when the answer is clear. Rewrite or delete it when the answer stays vague.\n\nThis step protects the review queue. Teams do not need every generated test. They need the small set that proves important behavior.\n\nImagine that AI writes a checkout test. The test adds one product and completes payment. It checks the success message.\n\nFirst, name the risk. A customer could pay the wrong total.\n\nNext, change the price calculation. The test still passes because the message appears. You found the missing proof.\n\nAdd a check for the order total. Then try an expired coupon and an empty cart. The test should explain each failed result.\n\nRun the test twice with the same data. Both runs should match.\n\nFinish with the release question. This test should stop checkout when the charged total is wrong. The answer now names one clear release failure.\n\nThat review improved one test without adding more code than needed. The team gained useful proof instead of another passing file.\n\nYou can complete this review in a few minutes:\n\nAI can handle test volume. You still decide what deserves trust. Keep tests that fail correctly and explain why the release should stop.\n\n*Anton Gulin builds AI systems and measurement tools. He tests software for a living and applies that discipline to AI. He runs independent LLM benchmarks and builds production test systems. Find him at anton.qa or on LinkedIn.*", "url": "https://wpnews.pro/news/how-to-review-ai-generated-tests-seven-checks-before-you-keep-them", "canonical_source": "https://dev.to/aiwithanton/how-to-review-ai-generated-tests-seven-checks-before-you-keep-them-3l4f", "published_at": "2026-08-15 15:00:43+00:00", "updated_at": "2026-08-15 15:12:07.879833+00:00", "lang": "en", "topics": ["developer-tools", "artificial-intelligence", "ai-products"], "entities": ["Playwright"], "alternates": {"html": "https://wpnews.pro/news/how-to-review-ai-generated-tests-seven-checks-before-you-keep-them", "markdown": "https://wpnews.pro/news/how-to-review-ai-generated-tests-seven-checks-before-you-keep-them.md", "text": "https://wpnews.pro/news/how-to-review-ai-generated-tests-seven-checks-before-you-keep-them.txt", "jsonld": "https://wpnews.pro/news/how-to-review-ai-generated-tests-seven-checks-before-you-keep-them.jsonld"}}