"Just automate it through the UI" sounds simple until you're the one maintaining that integration. When an API is missing, incomplete, or unsuitable for a task, an agent can often operate through the visible interface instead, the same DOM, accessibility tree, or screen pixels a person would use. That's a real capability. It's also a different engineering problem than an API integration, not an easier one.
An API is built for software-to-software communication: defined actions, structured data, predictable error responses. Strip that away and an agent has to work with whatever's actually on screen:
The core loop changes shape: observe state, pick one bounded action, perform it, verify the result, then or recover if the evidence is insufficient. "Can click" isn't the bar. Whether the system can recognize uncertainty and stop safely is the actual bar.
| Method | What it uses | Best fit | Main limitation |
|---|---|---|---|
| Browser automation | DOM, browser protocol, page state | Stable browser tools and web forms | Selectors and page states can change |
| Accessibility interaction | Roles, labels, values, control hierarchy | Accessible web/desktop/mobile UIs | Metadata may be missing or inaccurate |
| Screen and OCR interaction | Pixels, screenshots, visible text | Legacy apps, remote desktops | Visual interpretation is less deterministic |
| Keyboard/pointer/touch input | Standard user input | Cross-app and real-device tasks | Needs pairing with reliable observation |
| RPA | Rules, selectors, OCR, files | Narrow, repeatable legacy workflows | Exception handling grows over time |
| Hybrid API + UI | Approved APIs for some steps, UI for gaps | Partially integrated workflows | Requires careful state reconciliation |
The pattern worth internalizing: pick the most structured permitted method for the specific task, not the one with the broadest reach. Broad reach (screen + OCR) is also the least deterministic option, reach for it because you need to, not because it's the default.
Browser automation (W3C WebDriver, Playwright) is usually the strongest option for stable web apps, locate by semantic role and label, wait for load state, verify the confirmation state actually appeared. Still breaks on dynamic rendering, nested frames, A/B tests, and expiring sessions, explicit post-transition checks matter more than assuming a click worked.
Accessibility-tree interaction (WAI-ARIA, Accessible Name and Description Computation, plus platform frameworks like Microsoft UI Automation, Android UI Automator, Apple XCTest) gives you a semantic layer instead of raw pixels, when the app actually implements accessibility correctly. When it doesn't, you're back to guessing.
Screen and OCR interaction is the fallback for legacy apps and remote desktops with no exposed structure at all, and it's exactly as fragile as it sounds. Least deterministic, most universally applicable, use it when nothing else works, not as a default.
This is the part that's easy to skip past: no-API access doesn't override terms of service, platform rules, or the account owner's actual permissions. Before picking a method, confirm the task is something the agent (and the person deploying it) is actually authorized to do. A capability existing isn't the same as it being sanctioned.
Two hard lines, regardless of which method you're using:
Real production systems working this way need instrumentation that API-only systems mostly don't:
If you're building or evaluating a system like this, the useful question isn't "can it click the button." It's "when the button isn't where it expected, what happens next, and can a human see exactly what the agent tried." We build Aiden around this exact problem for physical devices, HDMI-based screen capture plus USB HID input, no API dependency, verification and human confirmation built into the action loop rather than bolted on after. Repo: github.com/AidenAI-IO/aiden-firmware.
Curious what verification patterns others here have found actually catch failures early, screenshot diffing, structured re-query of the accessibility tree, something else?