I've been wiring AI agents up to real Android phones. Two things kept going wrong. The agent misclicked, and some apps ignored the taps entirely. Here is what I found and what I ended up building.
The usual loop is simple. Take a screenshot, show it to a model, ask where to tap, send adb shell input tap x y.
It works until it doesn't. The screenshot you send is often scaled down from the real framebuffer. The model's coordinates are then off by some factor, and the tap lands next to the button. Animations and overlays make it worse. A floating tool or a caption bubble can sit right on top of your target.
The fix is to stop guessing. Android already knows where every button is. UIAutomator can dump the view hierarchy as XML, with text, descriptions, IDs and bounds. Find the node, take the center of its bounds, tap there.
adb shell input is handy. But it injects a synthesized event from a shell. It is not a kernel input device. Some apps simply ignore it.
The README for my project puts it plainly: adb shell input is "a synthesized shell event, not a kernel HID device, and is rejected by some apps". Accessibility-based automation has a similar profile, since it acts through a service rather than through a device.
So I wanted input that looks like a real touchscreen and a real keyboard to the OS.
scrcpy ships a server with UHID support. UHID lets you create virtual HID devices on the phone. I use that to create a touchscreen and a keyboard at the physical input level. Then I combine it with UIAutomator node lookup for targeting.
The split is the useful part. UIAutomator only observes: it reads the screen. HID does the actual touching and typing. Each tool does the job it is good at.
A few details that mattered in practice:
fill focuses a field, clears it, enters the text, and reports OK or MISMATCH, so the agent can check its own work. adb shell input
The tradeoff is setup. You need a scrcpy-server binary that matches your scrcpy version, and you provide it yourself. The repo does not bundle it.
Commands straight from the README:
python .\hidctl.py --serial YOUR_SERIAL --server C:\tools\scrcpy-server tap 500 900
python .\ph.py --serial YOUR_SERIAL --server C:\tools\scrcpy-server ui
python .\ph.py --serial YOUR_SERIAL --port 18733 daemon
python .\ph.py --serial YOUR_SERIAL --port 18733 tapn "Save"
python .\ph.py --serial YOUR_SERIAL --port 18733 fill 500 900 "123456"
python .\ph.py --serial YOUR_SERIAL --port 18733 stop
You need Python 3.11+, adb on your PATH, a matching scrcpy, and pip install uiautomator2 for the UI commands. Mixing client and server versions of scrcpy is unsupported.
ui prints what is on screen with bounds and centers. tapn "Save" finds a visible node by text, description or ID and taps it. There is also an optional serial allow-list, so a script can't drive the wrong device by accident.
Only use this on devices you own or are authorized to operate. Follow the rules of the apps and networks involved.
I just open-sourced this under the MIT license. It is small and it has rough edges, and the README has a long list of pitfalls I hit along the way. Feedback is welcome, especially on setups I haven't tried.