A coworker recently bragged about a Playwright test that took 4 seconds to generate. It passed, sure, but it asserted nothing—it just clicked a button and checked if the page still existed. It's the ultimate "fake it till you make it" success story: a green checkmark with zero actual value. I've spent the last six months leaning on AI for my automation workflow (mostly Playwright for web and Flutter for mobile), and while some parts are legit, a lot of the hype is just noise.
What actually works in production #
Converting bug reports into failing tests
This is the only real "magic" part. When QA sends a plain-English report like "Cart total does not update when you remove the last item while a coupon is applied," I feed that and my page object file into the model. I get a usable failing test in under a minute. I still have to fix the assertions because the AI is optimistic, but it handles the imports, fixtures, and navigation. That kills about 60% of the tedious typing.
Diagnosing flaky tests
We've all dealt with that one test that fails once every 20 runs, written 14 months ago by someone who already quit. When I'm staring at nested waits and hardcoded 8000ms timeouts, I paste the test and the trace into the AI. It's right about 50% of the time regarding the race condition, but even a wrong hypothesis is faster than staring at a screen in silence.
Cleaning up messy DOM locators
If you give it a chunk of HTML, it'll usually steer you toward getByRole or getByLabel instead of the CSS selector nightmare you were probably about to write. Think of it as a linter that actually has an opinion.
Where the wheels fall off #
Zero domain awareness
AI loves the "happy path" because that's what's in the documentation. It will never ask, "What happens if the payment webhook hits us twice?" That kind of intuition comes from getting paged at 2 AM because of a duplicate webhook, not from pattern matching. I'd estimate 80% of the critical bugs I've caught came from tests that no AI would ever think to generate.
The mobile web struggle
The models are basically desktop-centric. The moment you ask about touch targets, viewport-specific behavior, or a sticky header blocking a click on a 390px screen, the quality tanks. It'll confidently hand you a desktop solution and tell you it's for mobile. Device emulation and orientation handling still require a human who has actually seen the screen.
Flutter is a disaster
If you're using Flutter, good luck. The training data is thin. When I ask for a widget test, it gives me something that looks plausible but uses an API from two versions ago. If I ask for a smoke suite, it just gives me web patterns wearing a Flutter costume. I ended up building my smoke and regression setup for Flutter almost entirely by hand because the AI suggestions were consistently wrong. Next Google and NVIDIA are teaming up with Emerald AI to make data centers more flexible →
All Replies （3） #
I'm exhausted by this. I spent three hours debugging a "passing" test that was just asserting true === true using Copilot.
Finally found a decent list. I'm curious if any of these actually ship via TCS or if it's all DHL?
Finally, a real question. I'm curious if you're logging the DOM snapshot or just relying on Healer.io for the diff?