cd /news/ai-agents/letting-an-ai-agent-tap-an-android-p… · home › topics › ai-agents › article
[ARTICLE · art-171622] src=dev.to ↗ pub= topic=ai-agents verified=true sentiment=↑ positive

Letting an AI agent tap an Android phone with real touch events instead of guessing from screenshots

A developer has open-sourced an MIT-licensed toolset that drives Android phones with real kernel-level HID touch and keyboard events via scrcpy's UHID support, rather than synthesized `adb shell input` taps. The approach pairs UIAutomator view-hierarchy dumps for locating nodes by text, description, ID or bounds with HID input, addressing coordinate drift from scaled screenshots and apps that reject shell-injected events. The project requires a matching scrcpy-server binary supplied by the user, Python 3.11+, and uiautomator2 for UI commands.

recommendation

When an agent must operate a real Android device, locate elements via UIAutomator bounds instead of screenshot coordinates and send input through UHID virtual devices, since `adb shell input` is rejected by some apps.

more in playbook: Designing agent systems
by read3 min views2 publishedNov 18, 2026

I've been wiring AI agents up to real Android phones. Two things kept going wrong. The agent misclicked, and some apps ignored the taps entirely. Here is what I found and what I ended up building.

The usual loop is simple. Take a screenshot, show it to a model, ask where to tap, send adb shell input tap x y.

It works until it doesn't. The screenshot you send is often scaled down from the real framebuffer. The model's coordinates are then off by some factor, and the tap lands next to the button. Animations and overlays make it worse. A floating tool or a caption bubble can sit right on top of your target.

The fix is to stop guessing. Android already knows where every button is. UIAutomator can dump the view hierarchy as XML, with text, descriptions, IDs and bounds. Find the node, take the center of its bounds, tap there.

adb shell input is handy. But it injects a synthesized event from a shell. It is not a kernel input device. Some apps simply ignore it.

The README for my project puts it plainly: adb shell input is "a synthesized shell event, not a kernel HID device, and is rejected by some apps". Accessibility-based automation has a similar profile, since it acts through a service rather than through a device.

So I wanted input that looks like a real touchscreen and a real keyboard to the OS.

scrcpy ships a server with UHID support. UHID lets you create virtual HID devices on the phone. I use that to create a touchscreen and a keyboard at the physical input level. Then I combine it with UIAutomator node lookup for targeting.

The split is the useful part. UIAutomator only observes: it reads the screen. HID does the actual touching and typing. Each tool does the job it is good at.

A few details that mattered in practice:

fill focuses a field, clears it, enters the text, and reports OK or MISMATCH, so the agent can check its own work. adb shell input The tradeoff is setup. You need a scrcpy-server binary that matches your scrcpy version, and you provide it yourself. The repo does not bundle it.

Commands straight from the README:

python .\hidctl.py --serial YOUR_SERIAL --server C:\tools\scrcpy-server tap 500 900
python .\ph.py --serial YOUR_SERIAL --server C:\tools\scrcpy-server ui
python .\ph.py --serial YOUR_SERIAL --port 18733 daemon
python .\ph.py --serial YOUR_SERIAL --port 18733 tapn "Save"
python .\ph.py --serial YOUR_SERIAL --port 18733 fill 500 900 "123456"
python .\ph.py --serial YOUR_SERIAL --port 18733 stop

You need Python 3.11+, adb on your PATH, a matching scrcpy, and pip install uiautomator2 for the UI commands. Mixing client and server versions of scrcpy is unsupported.

ui prints what is on screen with bounds and centers. tapn "Save" finds a visible node by text, description or ID and taps it. There is also an optional serial allow-list, so a script can't drive the wrong device by accident.

Only use this on devices you own or are authorized to operate. Follow the rules of the apps and networks involved.

I just open-sourced this under the MIT license. It is small and it has rough edges, and the README has a long list of pitfalls I hit along the way. Feedback is welcome, especially on setups I haven't tried.

── more in #ai-agents 4 stories · sorted by recency
── more on @scrcpy 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/letting-an-ai-agent-…] indexed:0 read:3min 2026-11-18 · —