If you've ever recorded a UI test, shipped one button change to production, and watched 40 tests explode in CI — you know exactly why "record and replay" has a bad reputation.
I've spent the past year building a browser testing tool, and I want to talk about the unglamorous engineering that decides whether a recorded test survives a redesign — or dies on first contact. Not the AI magic. The four problems underneath it.
Most recorders store exactly one locator per element — an XPath or a CSS chain — and freeze it at record time. That's the root of almost every "my tests broke" story:
//div[2]/main/section[3]/button[1]
breaks when someone adds one <div>
to the layout..v-btn.theme--dark > .v-btn__content
break on any styling refactor.data-testid
— the community's favorite answer — isn't bulletproof. Third-party components don't have it, and a cleanup sprint that renames IDs silently kills dozens of tests.The fix: don't store one answer, store a ranked list of candidates.
{
"action": "click",
"target": "Add to cart button",
"candidates": [
{ "strategy": "test-id", "value": "add-to-cart" },
{ "strategy": "role-text", "value": "button 'Add to cart'" },
{ "strategy": "text", "value": "Add to cart" },
{ "strategy": "css", "value": ".btn-primary.cart-action" },
{ "strategy": "xpath", "value": "//button[contains(., 'Add to cart')]" }
]
}
At replay time, try the strongest match first and fall through on a miss. Then — and this matters more than people think — record which candidate matched. If your test passed via the XPath fallback, the page has changed in a way that deserves human review, even though the run is green.
Ranking rules that survived contact with real apps:
data-testid
, aria-label
, name
) — most stable, but often missing.No single strategy wins. The goal is that a test degrades gracefully instead of snapping.
element.click()
from JavaScript is not a click.
It invokes the event handlers, but it skips the browser's native input pipeline: no focus management, no :active
state, no scroll-into-view, different behavior with native controls like <select>
, date pickers, and file inputs. Tests pass on synthetic events and fail for real users — or the reverse.
The sturdier path is driving input at the browser level. The Chrome DevTools Protocol dispatches events through the same pipeline a real mouse and keyboard use:
await cdp.send('Input.dispatchMouseEvent', {
type: 'mousePressed', x, y, button: 'left', clickCount: 1,
});
await cdp.send('Input.dispatchMouseEvent', {
type: 'mouseReleased', x, y, button: 'left', clickCount: 1,
});
For typing, Input.insertText
behaves much closer to a human than setting .value
and firing an input
event.
The trade-off: browser-level input is stricter — and that's the point. If a cookie banner covers your button, a CDP click fails, correctly, because a human couldn't click it either. Synthetic events would have "passed" while hiding a real bug. The price is that you must handle overlays deliberately instead of pretending they don't exist.
When the primary path can't locate an element, naive tools do one of two dumb things: fail instantly (flaky suite), or silently fall back (the test drifts away from what it was testing). Both destroy trust in the suite.
The distinction that matters is why the element wasn't found:
So the failure ladder looks like:
CDP locate (primary candidates, ranked)
→ retry within deadline # "not there yet"
→ DOM-level fallback # CDP hit-test missed
→ next candidate in the list # element changed
→ fail, with full evidence # never silently
Every fallback gets logged and shown in the run report. A green run that used three fallbacks is not the same as a green run that didn't — and your team should see the difference.
Do the math on flaky tests: one ambiguous failure costs a QA engineer 15–30 minutes of "is this a real bug or is it the test?" Twenty failures a day is a person-day of triage, most of it wasted on non-bugs.
So the most valuable feature of a testing tool isn't execution speed — it's the quality of evidence attached to a failure:
We use AI to summarize the likely cause, but the recorded steps stay the source of truth. That's deliberate: an AI that silently "fixes" tests is just moving the drift somewhere you can't see it.
Honest limits, because trust beats hype:
That's the core of what I've learned building CueCast — a no-code tool built around these ideas: multi-candidate matching, browser-level input, and evidence-first failures. The techniques above work in Playwright or Selenium too; steal them either way.
How do you handle locator brittleness? data-testid
everywhere? Playwright's getByRole
? Visual AI matching? Curious what's actually holding up at your scale.