Test Automation with agent-device (Part 1)

A developer evaluated Callstack's open-source agent-device tool, which lets an AI agent drive iOS and Android apps and inspect the screen, for testing a Flutter app after Google's Android-only Artemis fell short on iOS support. The writeup covers installing the CLI and registering it as an MCP server with Claude Code, the boot/install/open/press/fill/wait loop, and Flutter-specific pitfalls: only text-bearing widgets appear in the accessibility tree unless Semantics identifiers are added, labels vary with device language, and AdMob test banners surface as platform_view, webview and link nodes on iOS.

agent-device is an open-source tool from Callstack that lets an AI agent drive iOS and Android apps and check what is on screen. This post covers why it was picked for testing a Flutter app, how to install it for Claude Code, the basic command loop, and what to watch out for with Flutter. It is based on version 0.21.31 2026-10-24 . Flutter ships Android and iOS apps from one codebase, so the device-testing tool has to cover iOS too. Google Artemis, the first candidate, is a natural-language testing agent, but it supports Android only; iOS is still just a roadmap item. The other candidates looked like this: agent-device handles iOS and Android simulators, emulators, and real devices with one tool, and its docs state Flutter support explicitly. It ships both a CLI and an MCP server, can save a verified flow as an .ad script to replay later, and can export to Maestro YAML. It is MIT-licensed. Install the CLI globally and check the environment. npm i -g agent-device@latest agent-device doctor doctor checks connected devices, toolchains, and dev-server status. In this setup it passed with only warnings for unused platforms HarmonyOS, Vega . Register it with Claude Code as an MCP server. With -s user , the same tool is available in every project. claude mcp add -s user agent-device -- agent-device mcp The agent-device repository includes three agent skills: agent-device , ios-simulator , and android-emulator . The official instruction is npx skills add callstack/agent-device , but Claude Code's auto mode safety check blocked it. In that case, clone the repository and copy the three folders under skills/ into ~/.claude/skills/ . To undo everything, run claude mcp remove agent-device -s user and npm uninstall -g agent-device , then delete the three skill folders. The basic loop is: prepare a device, open the app, act, verify, close. Pick the device with --platform and --device . agent-device boot --platform android --device "<emulator name " --headless agent-device install /absolute/path/app-debug.apk --platform android --device "<emulator name " agent-device open <bundle-id --platform android --device "<emulator name " --foreground agent-device press @e5 --settle agent-device fill @e11 "text" --settle agent-device wait text "<expected text " agent-device screenshot /absolute/path/home.png agent-device close open reads the screen and returns a ref such as @e5 for each element, like buttons and text fields. With --settle , press and fill wait until the screen goes quiet and return only what changed, so the agent can pick its next target without rereading the whole screen. Three things commonly go wrong here: wait text or wait <selector , and keep screenshots as evidence only. install an absolute path. For a Flutter app, build the iOS simulator app with flutter build ios --simulator --debug and the Android app with flutter build apk --debug . agent-device reads the screen through the accessibility tree. In Flutter, that means only widgets with text show up by default; icon buttons and containers do not. Add Semantics identifier: or semanticLabel to those widgets so a selector can find them. Flutter's Key does not appear in the accessibility tree. Labels in the accessibility tree follow the language the device is set to, so the same app reads with different label text depending on that setting. A text-based selector changes with the device language, which makes identifier the more stable choice. On iOS, AdMob test banners show up as platform view , webview , and link nodes. Selectors in replay scripts should avoid those nodes.