# Your Browser Automation Clicks Might Be Landing 25% Off — And Nothing Will Tell You

> Source: <https://dev.to/hidenari/your-browser-automation-clicks-might-be-landing-25-off-and-nothing-will-tell-you-2da>
> Published: 2026-08-16 01:52:34+00:00

I spent a day pointing an AI agent at a browser to publish one product across four marketplaces. Most of it worked. The parts that didn't work failed in the worst possible way: **silently**, with no error, no exception, and no log line.

Here are the five failure modes I hit, in the order I hit them, and the probe that turned the worst one from "this site is broken" into a two-line fix.

Everything below is from one working session. No hypotheticals.

I asked the agent to click a menu item. The tool reported success:

```
[computer:left_click] Clicked at (383, 734)
```

Nothing happened. No menu opened, no network request fired, no console error. I re-read the DOM — the element was there, visible, not disabled, not covered by an overlay:

``` js
const el = document.elementFromPoint(383, 734);
// => <span>削除</span>   ← the exact element I wanted
```

So the coordinate was right, the element was right, and the click "succeeded". And nothing happened.

I burned close to an hour assuming the site was blocking synthetic input. It wasn't.

Instead of reasoning about it, I made the page report whether a click ever arrived. Inject a button at a known position with a counter:

``` js
const b = document.createElement('button');
b.id = '__clicktest';
b.textContent = 'CLICK TEST';
b.style.cssText =
  'position:fixed;top:200px;left:200px;width:220px;height:60px;' +
  'z-index:2147483647;background:#f0f;';
b.addEventListener('click', (e) => {
  window.__clickOK = (window.__clickOK || 0) + 1;
  window.__trusted = e.isTrusted;
});
document.body.appendChild(b);
window.__clickOK = 0;

const r = b.getBoundingClientRect();
`center=${r.x + r.width / 2},${r.y + r.height / 2}`;
// => center=310,230
```

Then click that exact center and read the counter:

``` js
window.__clickOK;  // => 0
```

Zero. The click was not landing on a button that occupied 220×60 pixels at a coordinate I had just measured. That rules out "the site blocks synthetic events" — a blocked event still *arrives*; it just gets ignored by the handler. Mine never arrived.

So the coordinate space was wrong.

The automation tool takes coordinates in **screenshot space**. The page reports coordinates in **CSS pixel space**. If the screenshot is captured at a different scale than the viewport, every coordinate you read from the DOM is wrong by a constant factor — and the failure is invisible, because a click still happens, just somewhere else.

I measured the factor by bisecting. Device-pixel space (`×2`

, since `devicePixelRatio`

was 2) missed. Then:

``` js
window.__clickOK;  // after clicking (232, 172) instead of (310, 230)
// => 1
```

That gives the ratio directly:

```
k = 232 / 310 = 0.7484
```

Viewport was 1702 CSS px wide; the screenshots came back ~1274 px. `1274 / 1702 ≈ 0.7485`

. It matched.

From then on, every click went through one conversion:

``` js
const K = 0.7484;  // measure this yourself; do not copy mine

function clickPoint(el) {
  const r = el.getBoundingClientRect();
  return {
    x: Math.round((r.x + r.width / 2) * K),
    y: Math.round((r.y + r.height / 2) * K),
  };
}
```

Every interaction that had failed for the previous hour started working on the first try — dropdowns, confirmation dialogs, menu items.

**The lesson isn't the number.** `k`

depends on your window size, DPI, and tool. The lesson is that **you should never trust a coordinate you didn't verify with a probe**, because this class of bug produces no error at any layer.

Run the probe once at the start of a session. It costs three seconds.

With clicks fixed, one menu item still did nothing. Same probe logic applied — `elementFromPoint`

returned `null`

for it.

``` js
const r = item.getBoundingClientRect();
r.y;                  // => 981
window.innerHeight;   // => 876
```

The dropdown extended past the bottom of the viewport. `getBoundingClientRect()`

happily returns coordinates for content that isn't on screen, and a click at y=981 in a 876px viewport goes nowhere.

Scroll first, then re-measure. Never cache coordinates across a scroll:

``` js
el.scrollIntoView({ block: 'center' });
await new Promise(r => setTimeout(r, 300));
const r2 = el.getBoundingClientRect();  // re-read, always
```

One site had an upload drop zone with **no file input in the DOM**:

``` js
document.querySelectorAll('input[type=file]').length;  // => 0
```

The input is created on demand when you click the zone. So: click the zone, *then* look. To catch the element before the app calls `.click()`

on it (which opens a native dialog you can't drive), patch `createElement`

first:

``` js
const orig = document.createElement.bind(document);
window.__created = [];
document.createElement = function (tag, ...rest) {
  const el = orig(tag, ...rest);
  if (String(tag).toLowerCase() === 'input') {
    setTimeout(() => {
      if (el.type === 'file') window.__created.push(el);
    }, 0);
  }
  return el;
};
```

Now click the zone and the input is waiting for you in `window.__created`

. Populate it with a `DataTransfer`

and dispatch `change`

:

``` js
const dt = new DataTransfer();
dt.items.add(file);
input.files = dt.files;
input.dispatchEvent(new Event('change', { bubbles: true }));
```

This worked on two of the sites I tried. On the third it didn't — see below.

Some drop zones aren't inputs at all. The obvious move is to synthesize the drop:

``` js
const dt = new DataTransfer();
dt.items.add(file);
for (const type of ['dragenter', 'dragover', 'drop']) {
  const ev = new DragEvent(type, { bubbles: true, cancelable: true });
  Object.defineProperty(ev, 'dataTransfer', { value: dt });
  zone.dispatchEvent(ev);
}
```

I tried this against every plausible target — the drop zone, its ancestors, `document`

, `window`

. Nothing. The editor never registered a file.

To be sure the *file* wasn't the problem, I generated one in-page so there was no transfer step at all:

``` js
const c = document.createElement('canvas');
c.width = 1200; c.height = 800;
c.getContext('2d').fillRect(0, 0, 1200, 800);
const blob = await new Promise(r => c.toBlob(r, 'image/jpeg', 0.8));
const file = new File([blob], 'test.jpg', { type: 'image/jpeg' });
```

Same result. That's a real boundary: for that editor, drag-and-drop needs genuine OS-level input, and no amount of event synthesis substitutes for it. I stopped and did those uploads by hand.

Knowing *where the wall is* is worth more than another hour of clever attempts.

To avoid shipping image bytes through a JS payload, I served them locally with CORS enabled and fetched them from the page:

``` python
class H(http.server.SimpleHTTPRequestHandler):
    def end_headers(self):
        self.send_header('Access-Control-Allow-Origin', '*')
        super().end_headers()
```

`curl`

confirmed the header. From an `https://`

page, the fetch hung until it timed out:

```
await fetch('http://127.0.0.1:8941/img1.png');  // never resolves
```

An `https`

page fetching `http://127.0.0.1`

is mixed content. The CORS header is irrelevant — the request never gets far enough to matter. Terminate that path early instead of debugging your server.

Five minutes of setup that would have saved me most of a day:

```
// 1. Coordinate probe — is my click space the page's space?
//    (inject the test button above, click its center, read window.__clickOK)

// 2. Is the target actually on screen?
const onScreen = (el) => {
  const r = el.getBoundingClientRect();
  return r.top >= 0 && r.bottom <= window.innerHeight && r.width > 0;
};

// 3. Does this page even have the input I assume it has?
document.querySelectorAll('input[type=file]').length;

// 4. Verify before every destructive click.
//    Hover first, then confirm what is under the cursor:
document.elementFromPoint(x / K, y / K)?.textContent?.trim();
```

Step 4 matters more than it looks. In one menu, **"Delete" sat directly below "Unpublish"** — 24 pixels apart. With a 25% coordinate error, "unpublish" lands on "delete". Hovering and reading back the element text before committing turns a destructive misfire into a no-op.

The productive shift wasn't a better selector strategy. It was refusing to reason about why a click "didn't work" and instead **making the page report what it actually received**.

Every one of these five failures is invisible from the outside: success-shaped tool output, no exception, no console error. The only reliable signal came from instrumenting the page and reading a counter.

If your automation is mysteriously doing nothing, don't start with the selector. Start with the probe.

*All measurements here come from a single session driving Chrome against live marketplace sites. The scaling factor k is specific to that window and tool — measure your own.*