How to Debug LLM-Driven Android Automation Runs

A developer building LLM-driven Android automation tools has outlined a structured debugging approach that saves detailed run traces instead of just final screenshots. The method captures UI dumps, model decisions, tool calls, and exit codes at every step, enabling engineers to identify whether failures stem from the model, the app, the automation tool, or timing issues. The approach uses numbered files and compact action tables to make each run inspectable and replayable without requiring pixel-perfect video.

LLM-driven Android automation fails in strange ways. The model may tap the wrong label. The screen may change between observation and action. A keyboard may cover the button. A permission dialog may appear. The app may still be loading. The UI dump may expose two identical "Continue" buttons. If all you saved is the final screenshot, debugging is painful. You need a run trace. For every Android agent step, save: The minimum useful trace looks like this: observe: tap Button "Continue" continue 540,860 model: tap "Continue" action: hs tap "Continue" --visible --unique result: ok wait: hs wait "Dashboard" --timeout 15s result: TIMEOUT That is much easier to debug than "the agent failed." Android agent failures usually fall into a few buckets. | Failure | What it means | |---|---| NOT FOUND | The target label or selector was not visible | AMBIGUOUS | More than one visible node matched | TIMEOUT | The expected next state never appeared | SECURE WINDOW | Android blocked screenshots for the current window | | Wrong action | The model chose a bad label or command | | Stale observation | The UI changed after the model saw it | Good tooling should preserve which bucket happened. If everything becomes "click failed", the agent cannot recover intelligently. The UI dump is the agent's view of the world. Save it before each model decision: hs ui run/0007-ui.txt For LLM agents, a compact action table is usually better than full XML: fill EditText "Email" email 540,540 fill EditText "Password" password 540,640 password tap Button "Continue" continue 540,860 When a model picks the wrong action, this file tells you whether the model had a reasonable choice. Screenshots are valuable, but you do not need a full native PNG on every step. For most agent debugging: hs see --size 768 run/0007-screen.jpg Use screenshots when: Use the text UI as the default. Use screenshots as evidence. Do not only save the final command. Save what the model actually emitted: { "step": 7, "model action": "tap \"Continue\"", "tool call": "hs", "tap", "Continue", "--visible", "--unique" , "reason": "The login form is filled and Continue is visible." } This matters because the bug may be in translation: Keep the model layer and tool layer separate. Exit codes and error codes are better than stderr scraping. Handsets has common exit codes: 0 ok 2 NOT FOUND 3 TIMEOUT 4 AMBIGUOUS In JSON mode, preserve the structured error: hs --json tap "Continue" --visible --unique Then your agent can decide: NOT FOUND : dump UI again or scroll AMBIGUOUS : ask for a narrower selector TIMEOUT : capture screenshot and logs SECURE WINDOW : continue without screenshotAndroid logs are noisy. A small tail near the failure is usually enough: hs logs --tail 200 run/0007-logcat.txt Pair logs with the UI dump and screenshot from the same step. Otherwise you end up with artifacts that are technically present but hard to correlate. Use numbered files: run/ 0001-ui.txt 0001-action.json 0001-result.json 0002-ui.txt 0002-screen.jpg 0002-action.json 0002-result.json 0002-logcat.txt This is not fancy. That is the point. Before building a dashboard, make the run inspectable with plain files. Once you have traces, replay becomes possible. The useful replay is not pixel-perfect video. It is a timeline: Step 1: observed Sign in Step 2: tapped Sign in Step 3: filled Email Step 4: filled Password Step 5: tapped Continue Step 6: timed out waiting for Dashboard For teams, this timeline becomes the product. It lets an engineer see whether the model, the tool, or the app caused the failure. Because failures can come from the model, the app, the Android UI state, the automation tool, or timing. A final screenshot does not tell you which layer failed. Not always. Save compact UI dumps for every step. Add screenshots for visual states, failures, and custom-rendered screens. The pre-action UI dump. It shows what the model saw when it chose the action. Structured traces let you build targeted recovery: scroll on NOT FOUND , narrow selectors on AMBIGUOUS , capture logs on TIMEOUT , and avoid retrying blindly. Originally published at https://handsets.dev/blog/debug-llm-android-automation-runs/ https://handsets.dev/blog/debug-llm-android-automation-runs/ .