Your Browser Automation Clicks Might Be Landing 25% Off — And Nothing Will Tell You

An engineer spent a day pointing an AI agent at a browser to publish a product across four marketplaces and encountered five silent failure modes, the worst being clicks landing off-target due to a coordinate space mismatch between screenshot and CSS pixels. By injecting a probe button with a click counter, the engineer measured a scale factor of 0.7484 and fixed all interactions. The engineer warns that such bugs produce no errors and recommends verifying coordinates with a probe at the start of every session.

I spent a day pointing an AI agent at a browser to publish one product across four marketplaces. Most of it worked. The parts that didn't work failed in the worst possible way: silently , with no error, no exception, and no log line. Here are the five failure modes I hit, in the order I hit them, and the probe that turned the worst one from "this site is broken" into a two-line fix. Everything below is from one working session. No hypotheticals. I asked the agent to click a menu item. The tool reported success: computer:left click Clicked at 383, 734 Nothing happened. No menu opened, no network request fired, no console error. I re-read the DOM — the element was there, visible, not disabled, not covered by an overlay: js const el = document.elementFromPoint 383, 734 ; // = <span 削除</span ← the exact element I wanted So the coordinate was right, the element was right, and the click "succeeded". And nothing happened. I burned close to an hour assuming the site was blocking synthetic input. It wasn't. Instead of reasoning about it, I made the page report whether a click ever arrived. Inject a button at a known position with a counter: js const b = document.createElement 'button' ; b.id = ' clicktest'; b.textContent = 'CLICK TEST'; b.style.cssText = 'position:fixed;top:200px;left:200px;width:220px;height:60px;' + 'z-index:2147483647;background: f0f;'; b.addEventListener 'click', e = { window. clickOK = window. clickOK || 0 + 1; window. trusted = e.isTrusted; } ; document.body.appendChild b ; window. clickOK = 0; const r = b.getBoundingClientRect ; center=${r.x + r.width / 2},${r.y + r.height / 2} ; // = center=310,230 Then click that exact center and read the counter: js window. clickOK; // = 0 Zero. The click was not landing on a button that occupied 220×60 pixels at a coordinate I had just measured. That rules out "the site blocks synthetic events" — a blocked event still arrives ; it just gets ignored by the handler. Mine never arrived. So the coordinate space was wrong. The automation tool takes coordinates in screenshot space . The page reports coordinates in CSS pixel space . If the screenshot is captured at a different scale than the viewport, every coordinate you read from the DOM is wrong by a constant factor — and the failure is invisible, because a click still happens, just somewhere else. I measured the factor by bisecting. Device-pixel space ×2 , since devicePixelRatio was 2 missed. Then: js window. clickOK; // after clicking 232, 172 instead of 310, 230 // = 1 That gives the ratio directly: k = 232 / 310 = 0.7484 Viewport was 1702 CSS px wide; the screenshots came back ~1274 px. 1274 / 1702 ≈ 0.7485 . It matched. From then on, every click went through one conversion: js const K = 0.7484; // measure this yourself; do not copy mine function clickPoint el { const r = el.getBoundingClientRect ; return { x: Math.round r.x + r.width / 2 K , y: Math.round r.y + r.height / 2 K , }; } Every interaction that had failed for the previous hour started working on the first try — dropdowns, confirmation dialogs, menu items. The lesson isn't the number. k depends on your window size, DPI, and tool. The lesson is that you should never trust a coordinate you didn't verify with a probe , because this class of bug produces no error at any layer. Run the probe once at the start of a session. It costs three seconds. With clicks fixed, one menu item still did nothing. Same probe logic applied — elementFromPoint returned null for it. js const r = item.getBoundingClientRect ; r.y; // = 981 window.innerHeight; // = 876 The dropdown extended past the bottom of the viewport. getBoundingClientRect happily returns coordinates for content that isn't on screen, and a click at y=981 in a 876px viewport goes nowhere. Scroll first, then re-measure. Never cache coordinates across a scroll: js el.scrollIntoView { block: 'center' } ; await new Promise r = setTimeout r, 300 ; const r2 = el.getBoundingClientRect ; // re-read, always One site had an upload drop zone with no file input in the DOM : js document.querySelectorAll 'input type=file ' .length; // = 0 The input is created on demand when you click the zone. So: click the zone, then look. To catch the element before the app calls .click on it which opens a native dialog you can't drive , patch createElement first: js const orig = document.createElement.bind document ; window. created = ; document.createElement = function tag, ...rest { const el = orig tag, ...rest ; if String tag .toLowerCase === 'input' { setTimeout = { if el.type === 'file' window. created.push el ; }, 0 ; } return el; }; Now click the zone and the input is waiting for you in window. created . Populate it with a DataTransfer and dispatch change : js const dt = new DataTransfer ; dt.items.add file ; input.files = dt.files; input.dispatchEvent new Event 'change', { bubbles: true } ; This worked on two of the sites I tried. On the third it didn't — see below. Some drop zones aren't inputs at all. The obvious move is to synthesize the drop: js const dt = new DataTransfer ; dt.items.add file ; for const type of 'dragenter', 'dragover', 'drop' { const ev = new DragEvent type, { bubbles: true, cancelable: true } ; Object.defineProperty ev, 'dataTransfer', { value: dt } ; zone.dispatchEvent ev ; } I tried this against every plausible target — the drop zone, its ancestors, document , window . Nothing. The editor never registered a file. To be sure the file wasn't the problem, I generated one in-page so there was no transfer step at all: js const c = document.createElement 'canvas' ; c.width = 1200; c.height = 800; c.getContext '2d' .fillRect 0, 0, 1200, 800 ; const blob = await new Promise r = c.toBlob r, 'image/jpeg', 0.8 ; const file = new File blob , 'test.jpg', { type: 'image/jpeg' } ; Same result. That's a real boundary: for that editor, drag-and-drop needs genuine OS-level input, and no amount of event synthesis substitutes for it. I stopped and did those uploads by hand. Knowing where the wall is is worth more than another hour of clever attempts. To avoid shipping image bytes through a JS payload, I served them locally with CORS enabled and fetched them from the page: python class H http.server.SimpleHTTPRequestHandler : def end headers self : self.send header 'Access-Control-Allow-Origin', ' ' super .end headers curl confirmed the header. From an https:// page, the fetch hung until it timed out: await fetch 'http://127.0.0.1:8941/img1.png' ; // never resolves An https page fetching http://127.0.0.1 is mixed content. The CORS header is irrelevant — the request never gets far enough to matter. Terminate that path early instead of debugging your server. Five minutes of setup that would have saved me most of a day: // 1. Coordinate probe — is my click space the page's space? // inject the test button above, click its center, read window. clickOK // 2. Is the target actually on screen? const onScreen = el = { const r = el.getBoundingClientRect ; return r.top = 0 && r.bottom <= window.innerHeight && r.width 0; }; // 3. Does this page even have the input I assume it has? document.querySelectorAll 'input type=file ' .length; // 4. Verify before every destructive click. // Hover first, then confirm what is under the cursor: document.elementFromPoint x / K, y / K ?.textContent?.trim ; Step 4 matters more than it looks. In one menu, "Delete" sat directly below "Unpublish" — 24 pixels apart. With a 25% coordinate error, "unpublish" lands on "delete". Hovering and reading back the element text before committing turns a destructive misfire into a no-op. The productive shift wasn't a better selector strategy. It was refusing to reason about why a click "didn't work" and instead making the page report what it actually received . Every one of these five failures is invisible from the outside: success-shaped tool output, no exception, no console error. The only reliable signal came from instrumenting the page and reading a counter. If your automation is mysteriously doing nothing, don't start with the selector. Start with the probe. All measurements here come from a single session driving Chrome against live marketplace sites. The scaling factor k is specific to that window and tool — measure your own.