{"slug": "stop-letting-your-ai-agent-test-android-apps-with-screenshots", "title": "Stop letting your AI agent test Android apps with screenshots", "summary": "A developer released android-scenario-runner, an MIT-licensed Python tool that lets AI coding agents test Android apps by driving them through JSON scenario files instead of step-by-step screenshots. The runner uses `uiautomator dump` to read the screen as text, returning a single pass/fail line and only surfacing a screenshot when a visual decision is needed, which the developer says cut image-token costs across roughly 90 scenarios run on a real device. The repo is standard library plus adb, with no on-device installation required.", "body_md": "I shipped an Android app mostly by pairing with Claude Code. Writing features went fine. Testing them on a real phone did not.\n\nWhen an AI coding agent tests an app on a device, the usual loop is: take a screenshot, look at it, decide where to tap, tap, take another screenshot. Every step sends an image to the model. A 20-step login-and-checkout flow can cost thousands of image tokens, fill the context window, and the agent still sometimes taps the wrong thing.\n\nMost of that work doesn't need eyes. \"Tap *Sign in*, wait, check that *Home* is on screen\" is a text problem.\n\nAndroid can already describe the screen as text: `uiautomator dump` returns every visible node with its text and bounds. So instead of the agent driving the phone step by step, it writes the steps once as JSON:\n\n```\n{\n  \"name\": \"Login works\",\n  \"app\": \"com.example.app\",\n  \"steps\": [\n    {\"app\": \"restart\"},\n    {\"type\": [\"Email\", \"test@example.com\"]},    {\"type\": [\"Password\", \"secret123\"]},\n    {\"tap\": \"Sign in\", \"exact\": true},\n    {\"expect\": [\"Home\"], \"wait\": 3}\n  ]\n}\n```\n\nand runs it with one command:\n\n``` bash\n$ python scenario.py scenarios/login.json\nPASS Login works (5 steps, 12 s)\n```\n\nOne line back. If a step fails, the runner prints the failing step, what *was* on screen, and saves a screenshot, which is usually enough for the model to fix either the test or the app:\n\n```\n  ✗  5 expect: expected ['Home'] (found []); screen: ['Sign in', 'Wrong password', ...]\nFAIL Login works step 5/5 (9 s)\n```\n\nThe model only opens a screenshot when a visual decision is actually needed (layout, colours).\n\nMost of the code is not the happy path; it's the stuff that broke on real phones:\n\n`uiautomator dump` fails during an animation, the old `ui.xml` is still on the device and gets pulled again, so the test \"sees\" the previous screen. The runner deletes the file on both sides before every dump.`adb input text` sends the whole string at once. Type character by character.`wlan0` has no IP.\nAdd a few lines to your `CLAUDE.md` / `AGENTS.md`:\n\n```\n## Device testing\n- Never drive the phone step by step with screenshots. Write a scenario in\n  scenarios/<name>.json and run: python scenario.py scenarios/<name>.json | tail -3\n- To see the current screen as text: python ui.py dump\n```\n\nI ran ~90 scenarios this way on my own app before extracting the runner into its own repo. It's Python standard library + adb, nothing to install on the phone, MIT licensed:\n\n[https://github.com/sinangumuskabak-sys/android-scenario-runner](https://github.com/sinangumuskabak-sys/android-scenario-runner)\n\nIt isn't a replacement for Espresso or Maestro. It's a single file you can hand to an agent so it tests cheaply. Issues and PRs welcome, especially from people on OEM skins I haven't tried.", "url": "https://wpnews.pro/news/stop-letting-your-ai-agent-test-android-apps-with-screenshots", "canonical_source": "https://dev.to/sinan_gumuskabak_45e9d8a8/stop-letting-your-ai-agent-test-android-apps-with-screenshots-1ki7", "published_at": "2026-09-24 23:24:13+00:00", "updated_at": "2026-09-24 23:58:53.482793+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "ai-tools", "mlops"], "entities": ["android-scenario-runner", "Claude Code", "Android", "adb", "uiautomator", "Espresso", "Maestro", "GitHub"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/stop-letting-your-ai-agent-test-android-apps-with-screenshots", "markdown": "https://wpnews.pro/news/stop-letting-your-ai-agent-test-android-apps-with-screenshots.md", "text": "https://wpnews.pro/news/stop-letting-your-ai-agent-test-android-apps-with-screenshots.txt", "jsonld": "https://wpnews.pro/news/stop-letting-your-ai-agent-test-android-apps-with-screenshots.jsonld"}}