Most engineers expect AI-assisted automation to be the easy part. You describe a test, the model writes it, you move on.
The first week will prove you wrong.
Not because the code is bad. Because the code is almost right. And almost-right code is harder to debug than wrong code. Wrong code fails loudly. Almost-right code passes on Monday, fails on Tuesday, passes again on Wednesday, and by Thursday you are questioning whether you understand your own application.
I have watched teams adopt AI copilots into their Playwright suites and spend the first five days doing nothing but untangling false passes. If you are about to start this journey, here is what that week actually looks like.
A language model has never waited for a network response. It has never watched a flaky selector survive three CI runs and then collapse on the fourth. It writes tests from a static understanding of your page, not from the dynamic reality of your application.
You will ask it to write a test that clicks a button and waits for a confirmation toast. The model will produce something like this:
await page.click('button:has-text("Submit")');
await page.waitForSelector('.toast-success');
Looks fine. Runs fine. Then your team deploys a new build where the toast takes 400ms longer to appear because of an analytics call. The test fails. Not because the feature broke. Because the model assumed a timing that was never guaranteed.
This is the core problem. The model writes tests that match the page as it was when the model saw it. It does not write tests that match the page as it will be.
The shift is mental before it is technical. You cannot review AI-generated tests the way you review human-written tests. Human tests come with intent. AI tests come with patterns.
You need a different review lens.
First, look for every hardcoded wait. Replace it with a state-based assertion. Second, look for every selector that relies on text content that could change. Third, look for every assumption about element order on the page.
The model will write page.locator('button').first()
because it saw one button. Your page has three buttons. The test will click the wrong one.
The solution is not to stop using the model. The solution is to treat every line it writes as a first draft that needs a human edit.
I have seen three patterns repeat across teams. If you know them in advance, you can catch them in review instead of in CI.
The model assumes elements are uniquely identifiable by text or role. In practice, your page has multiple elements with the same label.
// AI-generated
await page.getByRole('button', { name: 'Save' }).click();
This works until a second "Save" button appears in a modal. Now the test clicks the wrong one. The fix is to scope the selector to a specific container:
// Human-edited
const dialog = page.getByRole('dialog', { name: 'Confirm changes' });
await dialog.getByRole('button', { name: 'Save' }).click();
The model writes actions but skips the verification that the action actually completed.
// AI-generated
await page.fill('#email', 'test@example.com');
await page.fill('#password', 'password123');
await page.click('button[type="submit"]');
No assertion that the form submitted. No check that the next page loaded. The test passes even if the submit button does nothing. You only discover the problem when a real regression slips through.
The model defaults to fixed timeouts or waits for elements that may not exist in every state.
// AI-generated
await page.waitForTimeout(2000);
This is the most common pattern I remove. It passes locally, fails in CI, and wastes hours of debugging time. Replace every waitForTimeout
with a waitForSelector
, waitForURL
, or waitForResponse
.
Here is the honest take. A good AI copilot writes tests at the level of a junior engineer who has read your documentation but never used your product. The code is syntactically correct. The logic is structurally sound. But the judgment is missing.
The junior engineer does not know that this particular button only appears after a network call. The junior engineer does not know that this toast sometimes takes three seconds. The junior engineer writes tests that work in the happy path and break everywhere else.
Your job is not to accept the code. Your job is to mentor it.
Every edit you make to an AI-generated test is teaching the model, indirectly, what matters in your application. Over time, the drafts get better. But the first week will be brutal because you are starting from zero shared context.
If you are about to integrate an AI copilot into your Playwright suite, do not generate fifty tests on day one. Generate one. Debug it. Fix it. Run it ten times. Then generate the next one.
The temptation is speed. The reality is that speed without stability is just faster failure.
Pick one critical user flow. Ask the model to write it. Then spend the time to make it production-ready. Measure how long that took. Multiply by the number of flows you need. That is your real timeline.
The model will save you time on boilerplate. It will not save you time on debugging. That part is still yours.
What was the first AI-generated test you had to completely rewrite? I would like to hear which failure mode hit you first.