Blog · · 6 min read When an AI assistant uses a phone, you'd expect the hard part to be the screen: reading an app it has never seen and finding the right button. Between 8 October and 3 November assistants made 3,324 actions on 45 Android phones through Opposable, and that part mostly worked, with 99% of their taps and swipes doing what they asked. The most common failure was the phone not being there at all, because 27 of the 45 phones dropped off our relay while an agent was partway through using them, and that alone accounts for 108 of the 305 failed actions.
Opposable is an Android app that lets Claude, ChatGPT, Claude Code or Codex use the phone it's installed on. The assistant asks for a screenshot or a list of what's on the screen, then taps, swipes and types while the owner watches or puts the phone down. Android is designed around one person who holds the phone, unlocks it and taps it, and the failures in this post cluster where that design meets a user who isn't holding it.
#TL;DR #
91% of the 3,324 actions worked. Of the 305 that failed, 140 failed because of the phone or the OS: 108 times the phone wasn't connected, 21 times no text field had focus when the agent typed, and the rest were cancelled gestures and locked screens. The other 165 came from the agents and our own app. Android also has no way to tell an agent's taps from the owner's, which never shows up as a failure but limits what an agent can safely be allowed to do.
#When nobody is touching the phone #
Android saves battery by assuming that a phone nobody is touching doesn't need to do anything. Doze and App Standby defer background work on an idle phone, and several phone makers add battery managers of their own that close apps Android would have left running. An agent usually reaches the phone at exactly that moment: the owner asks from a laptop while the phone lies face down on the desk, or asks in the chat app and then puts the phone in a pocket. The app keeps a connection open to our relay, and every request from the assistant arrives over it, so when Android or the maker's battery manager closes it the phone stops answering.
36 of the 105 sessions in the app's activity log ran into a phone that wasn't connected, and in 16 of them the agent gave up. 7 of 9 Xiaomi phones and 14 of 20 Samsung phones dropped off at least once, against 2 of 5 Pixels and 1 of 4 Motorolas. The log can't tell a battery manager closing the app from an owner switching the phone off or losing signal, but the split by maker lines up with dontkillmyapp.com's ranking of battery managers, where Samsung and Xiaomi are both near the top.
The app now runs as a foreground service with a permanent notification, asks once to be left out of battery optimisation, and sets a 15-minute alarm that reconnects if the service was frozen, because Android still delivers alarms in Doze's maintenance windows. On Xiaomi, Samsung, Oppo, vivo, Huawei and Tecno phones it also shows a card with the maker's own setting, such as Autostart on Xiaomi or Never sleeping apps on Samsung, which only the owner can switch on. A phone an agent can reach on demand behaves like a small server, and Android has no setting for that. The battery exemption comes closest, and not every maker's battery manager respects it.
#Borrowing the accessibility service #
Android has one general way for an app to read other apps' screens and act in them, the accessibility service, which was built for people who use a screen reader or switch access instead of touching the screen, and agents borrow it because nothing else exists. Turning it on means a trip to Settings › Accessibility, and on Android 13 and later an app installed from a browser starts with that switch greyed out until the owner finds Allow restricted settings in the app's info page. 143 of the 192 installs that sent anonymous setup events turned the service on, and we can't tell how many of the other 49 stopped at the greyed-out switch and how many looked around and left.
Android 16 added AppFunctions, which lets an app offer actions to an agent directly without going through the screen. It covers only apps whose developers adopt it, so for everything else the accessibility service is still the way in.
#Whoever taps is the owner #
To Android, an accessibility service's taps look like the owner's. No flag marks an action as coming from an agent, so an app can't treat agent input differently, and the owner has no system setting that keeps an agent out of one app or asks them before it acts. Android 14 added accessibilityDataSensitive, which lets an app hide views from services that aren't real accessibility tools, and FLAG_SECURE keeps a screen out of screenshots, but both hide content and neither lets the owner approve an action.
We built approvals into the app instead: payments wait for the owner to approve them on the phone, and the app's own screens, including the switch that turns those approvals off, refuse agent taps. Over the four weeks agents tapped there 4 times and were refused each time, and the log doesn't record what they were trying to do.
The lock screen is the one place where Android does check who is there. A PIN, pattern or fingerprint stops an agent the same way it stops anyone else, so the app turns the screen on and shows the owner's normal unlock prompt, and locked phones caused 3 failed actions.
#One screen #
A desktop agent can work in one window while you use another. On a phone the agent takes the whole screen, so the owner can't use the phone until it's done, and the chat app they asked from goes to the background. The median session was 14 actions over about four minutes and the longest was 163, and agents called return_to_chat 37 times to bring the assistant back to the front when they finished.
#The part that worked #
Screenshots worked 95% of the time and taps and swipes 99%. Every action already returns the updated list of elements on the screen, each with a number the agent can tap by, and agents still asked for a screenshot 620 times and tapped by pixel coordinates 426 times against 525 taps by element. Most likely the list is enough where an app labels its buttons for accessibility and the agent falls back to the picture where it doesn't, but the log doesn't record which apps these were. A tap by coordinates always lands somewhere, so its 99% says less than the 92% for taps by element, which fail when the element the agent named isn't there.
#Smaller problems: typing and app names #
Typing worked 85% of the time. Android sends text to whichever field has focus, and focus can arrive a moment after the tap that gave it, so 21 times the agent typed before any field had it. The app now waits up to 600 ms for focus, and when nothing is focused and the screen has a single text field it focuses that one. With several fields it returns their names so the agent can tap the right one.
Opening an app failed 15% of the time, partly because the same app goes by a different name and package on each brand. The clock is com.google.android.deskclock on a Pixel and com.sec.android.app.clockpackage on a Samsung, and agents sometimes asked for one on the other. For alarms, timers and switching Wi-Fi Android has standard intents that work on every phone, and Opposable's on-device agent runs about forty common tasks like these as fixed code instead of tapping through an app.
#What would help #
Two changes in Android would help most. One is a supported way for an app to stay reachable on demand that phone makers' battery managers respect, which would remove most of the red in that chart. The other is a notion of agent input: a permission the owner grants per app, a marker on each event saying an agent sent it, and a system prompt before actions the owner has marked as sensitive. Typing and app names are problems apps and agents can work around, and the app already does.
#Limitations #
45 phones over four weeks is a small sample, and 3,048 of the 3,324 actions came through Claude, so this mostly describes Claude's habits. Because the log keeps 200 entries per phone, the earliest sessions on the five busiest phones aren't in it. Failures of our on-device model are in the chart for completeness and say nothing about Android.
#Trying it #
The app is at getopposable.com. If your phone drops off while an agent is using it, the app's home screen shows the battery setting your phone's maker needs, and I'd like to hear which phone it was at hello@getopposable.com.