{"slug": "your-browser-automation-clicks-might-be-landing-25-off-and-nothing-will-tell-you", "title": "Your Browser Automation Clicks Might Be Landing 25% Off — And Nothing Will Tell You", "summary": "An engineer spent a day pointing an AI agent at a browser to publish a product across four marketplaces and encountered five silent failure modes, the worst being clicks landing off-target due to a coordinate space mismatch between screenshot and CSS pixels. By injecting a probe button with a click counter, the engineer measured a scale factor of 0.7484 and fixed all interactions. The engineer warns that such bugs produce no errors and recommends verifying coordinates with a probe at the start of every session.", "body_md": "I spent a day pointing an AI agent at a browser to publish one product across four marketplaces. Most of it worked. The parts that didn't work failed in the worst possible way: **silently**, with no error, no exception, and no log line.\n\nHere are the five failure modes I hit, in the order I hit them, and the probe that turned the worst one from \"this site is broken\" into a two-line fix.\n\nEverything below is from one working session. No hypotheticals.\n\nI asked the agent to click a menu item. The tool reported success:\n\n```\n[computer:left_click] Clicked at (383, 734)\n```\n\nNothing happened. No menu opened, no network request fired, no console error. I re-read the DOM — the element was there, visible, not disabled, not covered by an overlay:\n\n``` js\nconst el = document.elementFromPoint(383, 734);\n// => <span>削除</span>   ← the exact element I wanted\n```\n\nSo the coordinate was right, the element was right, and the click \"succeeded\". And nothing happened.\n\nI burned close to an hour assuming the site was blocking synthetic input. It wasn't.\n\nInstead of reasoning about it, I made the page report whether a click ever arrived. Inject a button at a known position with a counter:\n\n``` js\nconst b = document.createElement('button');\nb.id = '__clicktest';\nb.textContent = 'CLICK TEST';\nb.style.cssText =\n  'position:fixed;top:200px;left:200px;width:220px;height:60px;' +\n  'z-index:2147483647;background:#f0f;';\nb.addEventListener('click', (e) => {\n  window.__clickOK = (window.__clickOK || 0) + 1;\n  window.__trusted = e.isTrusted;\n});\ndocument.body.appendChild(b);\nwindow.__clickOK = 0;\n\nconst r = b.getBoundingClientRect();\n`center=${r.x + r.width / 2},${r.y + r.height / 2}`;\n// => center=310,230\n```\n\nThen click that exact center and read the counter:\n\n``` js\nwindow.__clickOK;  // => 0\n```\n\nZero. The click was not landing on a button that occupied 220×60 pixels at a coordinate I had just measured. That rules out \"the site blocks synthetic events\" — a blocked event still *arrives*; it just gets ignored by the handler. Mine never arrived.\n\nSo the coordinate space was wrong.\n\nThe automation tool takes coordinates in **screenshot space**. The page reports coordinates in **CSS pixel space**. If the screenshot is captured at a different scale than the viewport, every coordinate you read from the DOM is wrong by a constant factor — and the failure is invisible, because a click still happens, just somewhere else.\n\nI measured the factor by bisecting. Device-pixel space (`×2`\n\n, since `devicePixelRatio`\n\nwas 2) missed. Then:\n\n``` js\nwindow.__clickOK;  // after clicking (232, 172) instead of (310, 230)\n// => 1\n```\n\nThat gives the ratio directly:\n\n```\nk = 232 / 310 = 0.7484\n```\n\nViewport was 1702 CSS px wide; the screenshots came back ~1274 px. `1274 / 1702 ≈ 0.7485`\n\n. It matched.\n\nFrom then on, every click went through one conversion:\n\n``` js\nconst K = 0.7484;  // measure this yourself; do not copy mine\n\nfunction clickPoint(el) {\n  const r = el.getBoundingClientRect();\n  return {\n    x: Math.round((r.x + r.width / 2) * K),\n    y: Math.round((r.y + r.height / 2) * K),\n  };\n}\n```\n\nEvery interaction that had failed for the previous hour started working on the first try — dropdowns, confirmation dialogs, menu items.\n\n**The lesson isn't the number.** `k`\n\ndepends on your window size, DPI, and tool. The lesson is that **you should never trust a coordinate you didn't verify with a probe**, because this class of bug produces no error at any layer.\n\nRun the probe once at the start of a session. It costs three seconds.\n\nWith clicks fixed, one menu item still did nothing. Same probe logic applied — `elementFromPoint`\n\nreturned `null`\n\nfor it.\n\n``` js\nconst r = item.getBoundingClientRect();\nr.y;                  // => 981\nwindow.innerHeight;   // => 876\n```\n\nThe dropdown extended past the bottom of the viewport. `getBoundingClientRect()`\n\nhappily returns coordinates for content that isn't on screen, and a click at y=981 in a 876px viewport goes nowhere.\n\nScroll first, then re-measure. Never cache coordinates across a scroll:\n\n``` js\nel.scrollIntoView({ block: 'center' });\nawait new Promise(r => setTimeout(r, 300));\nconst r2 = el.getBoundingClientRect();  // re-read, always\n```\n\nOne site had an upload drop zone with **no file input in the DOM**:\n\n``` js\ndocument.querySelectorAll('input[type=file]').length;  // => 0\n```\n\nThe input is created on demand when you click the zone. So: click the zone, *then* look. To catch the element before the app calls `.click()`\n\non it (which opens a native dialog you can't drive), patch `createElement`\n\nfirst:\n\n``` js\nconst orig = document.createElement.bind(document);\nwindow.__created = [];\ndocument.createElement = function (tag, ...rest) {\n  const el = orig(tag, ...rest);\n  if (String(tag).toLowerCase() === 'input') {\n    setTimeout(() => {\n      if (el.type === 'file') window.__created.push(el);\n    }, 0);\n  }\n  return el;\n};\n```\n\nNow click the zone and the input is waiting for you in `window.__created`\n\n. Populate it with a `DataTransfer`\n\nand dispatch `change`\n\n:\n\n``` js\nconst dt = new DataTransfer();\ndt.items.add(file);\ninput.files = dt.files;\ninput.dispatchEvent(new Event('change', { bubbles: true }));\n```\n\nThis worked on two of the sites I tried. On the third it didn't — see below.\n\nSome drop zones aren't inputs at all. The obvious move is to synthesize the drop:\n\n``` js\nconst dt = new DataTransfer();\ndt.items.add(file);\nfor (const type of ['dragenter', 'dragover', 'drop']) {\n  const ev = new DragEvent(type, { bubbles: true, cancelable: true });\n  Object.defineProperty(ev, 'dataTransfer', { value: dt });\n  zone.dispatchEvent(ev);\n}\n```\n\nI tried this against every plausible target — the drop zone, its ancestors, `document`\n\n, `window`\n\n. Nothing. The editor never registered a file.\n\nTo be sure the *file* wasn't the problem, I generated one in-page so there was no transfer step at all:\n\n``` js\nconst c = document.createElement('canvas');\nc.width = 1200; c.height = 800;\nc.getContext('2d').fillRect(0, 0, 1200, 800);\nconst blob = await new Promise(r => c.toBlob(r, 'image/jpeg', 0.8));\nconst file = new File([blob], 'test.jpg', { type: 'image/jpeg' });\n```\n\nSame result. That's a real boundary: for that editor, drag-and-drop needs genuine OS-level input, and no amount of event synthesis substitutes for it. I stopped and did those uploads by hand.\n\nKnowing *where the wall is* is worth more than another hour of clever attempts.\n\nTo avoid shipping image bytes through a JS payload, I served them locally with CORS enabled and fetched them from the page:\n\n``` python\nclass H(http.server.SimpleHTTPRequestHandler):\n    def end_headers(self):\n        self.send_header('Access-Control-Allow-Origin', '*')\n        super().end_headers()\n```\n\n`curl`\n\nconfirmed the header. From an `https://`\n\npage, the fetch hung until it timed out:\n\n```\nawait fetch('http://127.0.0.1:8941/img1.png');  // never resolves\n```\n\nAn `https`\n\npage fetching `http://127.0.0.1`\n\nis mixed content. The CORS header is irrelevant — the request never gets far enough to matter. Terminate that path early instead of debugging your server.\n\nFive minutes of setup that would have saved me most of a day:\n\n```\n// 1. Coordinate probe — is my click space the page's space?\n//    (inject the test button above, click its center, read window.__clickOK)\n\n// 2. Is the target actually on screen?\nconst onScreen = (el) => {\n  const r = el.getBoundingClientRect();\n  return r.top >= 0 && r.bottom <= window.innerHeight && r.width > 0;\n};\n\n// 3. Does this page even have the input I assume it has?\ndocument.querySelectorAll('input[type=file]').length;\n\n// 4. Verify before every destructive click.\n//    Hover first, then confirm what is under the cursor:\ndocument.elementFromPoint(x / K, y / K)?.textContent?.trim();\n```\n\nStep 4 matters more than it looks. In one menu, **\"Delete\" sat directly below \"Unpublish\"** — 24 pixels apart. With a 25% coordinate error, \"unpublish\" lands on \"delete\". Hovering and reading back the element text before committing turns a destructive misfire into a no-op.\n\nThe productive shift wasn't a better selector strategy. It was refusing to reason about why a click \"didn't work\" and instead **making the page report what it actually received**.\n\nEvery one of these five failures is invisible from the outside: success-shaped tool output, no exception, no console error. The only reliable signal came from instrumenting the page and reading a counter.\n\nIf your automation is mysteriously doing nothing, don't start with the selector. Start with the probe.\n\n*All measurements here come from a single session driving Chrome against live marketplace sites. The scaling factor k is specific to that window and tool — measure your own.*", "url": "https://wpnews.pro/news/your-browser-automation-clicks-might-be-landing-25-off-and-nothing-will-tell-you", "canonical_source": "https://dev.to/hidenari/your-browser-automation-clicks-might-be-landing-25-off-and-nothing-will-tell-you-2da", "published_at": "2026-08-16 01:52:34+00:00", "updated_at": "2026-08-16 02:11:02.276796+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/your-browser-automation-clicks-might-be-landing-25-off-and-nothing-will-tell-you", "markdown": "https://wpnews.pro/news/your-browser-automation-clicks-might-be-landing-25-off-and-nothing-will-tell-you.md", "text": "https://wpnews.pro/news/your-browser-automation-clicks-might-be-landing-25-off-and-nothing-will-tell-you.txt", "jsonld": "https://wpnews.pro/news/your-browser-automation-clicks-might-be-landing-25-off-and-nothing-will-tell-you.jsonld"}}