I spent a day pointing an AI agent at a browser to publish one product across four marketplaces. Most of it worked. The parts that didn't work failed in the worst possible way: silently, with no error, no exception, and no log line.
Here are the five failure modes I hit, in the order I hit them, and the probe that turned the worst one from "this site is broken" into a two-line fix.
Everything below is from one working session. No hypotheticals.
I asked the agent to click a menu item. The tool reported success:
[computer:left_click] Clicked at (383, 734)
Nothing happened. No menu opened, no network request fired, no console error. I re-read the DOM — the element was there, visible, not disabled, not covered by an overlay:
const el = document.elementFromPoint(383, 734);
// => <span>削除</span> ← the exact element I wanted
So the coordinate was right, the element was right, and the click "succeeded". And nothing happened.
I burned close to an hour assuming the site was blocking synthetic input. It wasn't.
Instead of reasoning about it, I made the page report whether a click ever arrived. Inject a button at a known position with a counter:
const b = document.createElement('button');
b.id = '__clicktest';
b.textContent = 'CLICK TEST';
b.style.cssText =
'position:fixed;top:200px;left:200px;width:220px;height:60px;' +
'z-index:2147483647;background:#f0f;';
b.addEventListener('click', (e) => {
window.__clickOK = (window.__clickOK || 0) + 1;
window.__trusted = e.isTrusted;
});
document.body.appendChild(b);
window.__clickOK = 0;
const r = b.getBoundingClientRect();
`center=${r.x + r.width / 2},${r.y + r.height / 2}`;
// => center=310,230
Then click that exact center and read the counter:
window.__clickOK; // => 0
Zero. The click was not landing on a button that occupied 220×60 pixels at a coordinate I had just measured. That rules out "the site blocks synthetic events" — a blocked event still arrives; it just gets ignored by the handler. Mine never arrived.
So the coordinate space was wrong.
The automation tool takes coordinates in screenshot space. The page reports coordinates in CSS pixel space. If the screenshot is captured at a different scale than the viewport, every coordinate you read from the DOM is wrong by a constant factor — and the failure is invisible, because a click still happens, just somewhere else.
I measured the factor by bisecting. Device-pixel space (×2
, since devicePixelRatio
was 2) missed. Then:
window.__clickOK; // after clicking (232, 172) instead of (310, 230)
// => 1
That gives the ratio directly:
k = 232 / 310 = 0.7484
Viewport was 1702 CSS px wide; the screenshots came back ~1274 px. 1274 / 1702 ≈ 0.7485
. It matched.
From then on, every click went through one conversion:
const K = 0.7484; // measure this yourself; do not copy mine
function clickPoint(el) {
const r = el.getBoundingClientRect();
return {
x: Math.round((r.x + r.width / 2) * K),
y: Math.round((r.y + r.height / 2) * K),
};
}
Every interaction that had failed for the previous hour started working on the first try — dropdowns, confirmation dialogs, menu items.
The lesson isn't the number. k
depends on your window size, DPI, and tool. The lesson is that you should never trust a coordinate you didn't verify with a probe, because this class of bug produces no error at any layer.
Run the probe once at the start of a session. It costs three seconds.
With clicks fixed, one menu item still did nothing. Same probe logic applied — elementFromPoint
returned null
for it.
const r = item.getBoundingClientRect();
r.y; // => 981
window.innerHeight; // => 876
The dropdown extended past the bottom of the viewport. getBoundingClientRect()
happily returns coordinates for content that isn't on screen, and a click at y=981 in a 876px viewport goes nowhere.
Scroll first, then re-measure. Never cache coordinates across a scroll:
el.scrollIntoView({ block: 'center' });
await new Promise(r => setTimeout(r, 300));
const r2 = el.getBoundingClientRect(); // re-read, always
One site had an upload drop zone with no file input in the DOM:
document.querySelectorAll('input[type=file]').length; // => 0
The input is created on demand when you click the zone. So: click the zone, then look. To catch the element before the app calls .click()
on it (which opens a native dialog you can't drive), patch createElement
first:
const orig = document.createElement.bind(document);
window.__created = [];
document.createElement = function (tag, ...rest) {
const el = orig(tag, ...rest);
if (String(tag).toLowerCase() === 'input') {
setTimeout(() => {
if (el.type === 'file') window.__created.push(el);
}, 0);
}
return el;
};
Now click the zone and the input is waiting for you in window.__created
. Populate it with a DataTransfer
and dispatch change
:
const dt = new DataTransfer();
dt.items.add(file);
input.files = dt.files;
input.dispatchEvent(new Event('change', { bubbles: true }));
This worked on two of the sites I tried. On the third it didn't — see below.
Some drop zones aren't inputs at all. The obvious move is to synthesize the drop:
const dt = new DataTransfer();
dt.items.add(file);
for (const type of ['dragenter', 'dragover', 'drop']) {
const ev = new DragEvent(type, { bubbles: true, cancelable: true });
Object.defineProperty(ev, 'dataTransfer', { value: dt });
zone.dispatchEvent(ev);
}
I tried this against every plausible target — the drop zone, its ancestors, document
, window
. Nothing. The editor never registered a file.
To be sure the file wasn't the problem, I generated one in-page so there was no transfer step at all:
const c = document.createElement('canvas');
c.width = 1200; c.height = 800;
c.getContext('2d').fillRect(0, 0, 1200, 800);
const blob = await new Promise(r => c.toBlob(r, 'image/jpeg', 0.8));
const file = new File([blob], 'test.jpg', { type: 'image/jpeg' });
Same result. That's a real boundary: for that editor, drag-and-drop needs genuine OS-level input, and no amount of event synthesis substitutes for it. I stopped and did those uploads by hand.
Knowing where the wall is is worth more than another hour of clever attempts.
To avoid shipping image bytes through a JS payload, I served them locally with CORS enabled and fetched them from the page:
class H(http.server.SimpleHTTPRequestHandler):
def end_headers(self):
self.send_header('Access-Control-Allow-Origin', '*')
super().end_headers()
curl
confirmed the header. From an https://
page, the fetch hung until it timed out:
await fetch('http://127.0.0.1:8941/img1.png'); // never resolves
An https
page fetching http://127.0.0.1
is mixed content. The CORS header is irrelevant — the request never gets far enough to matter. Terminate that path early instead of debugging your server.
Five minutes of setup that would have saved me most of a day:
// 1. Coordinate probe — is my click space the page's space?
// (inject the test button above, click its center, read window.__clickOK)
// 2. Is the target actually on screen?
const onScreen = (el) => {
const r = el.getBoundingClientRect();
return r.top >= 0 && r.bottom <= window.innerHeight && r.width > 0;
};
// 3. Does this page even have the input I assume it has?
document.querySelectorAll('input[type=file]').length;
// 4. Verify before every destructive click.
// Hover first, then confirm what is under the cursor:
document.elementFromPoint(x / K, y / K)?.textContent?.trim();
Step 4 matters more than it looks. In one menu, "Delete" sat directly below "Unpublish" — 24 pixels apart. With a 25% coordinate error, "unpublish" lands on "delete". Hovering and reading back the element text before committing turns a destructive misfire into a no-op.
The productive shift wasn't a better selector strategy. It was refusing to reason about why a click "didn't work" and instead making the page report what it actually received.
Every one of these five failures is invisible from the outside: success-shaped tool output, no exception, no console error. The only reliable signal came from instrumenting the page and reading a counter.
If your automation is mysteriously doing nothing, don't start with the selector. Start with the probe.
All measurements here come from a single session driving Chrome against live marketplace sites. The scaling factor k is specific to that window and tool — measure your own.