{"slug": "agent-browser-equivalent-for-desktop-and-mobile-apps", "title": "Agent-browser equivalent for desktop and mobile apps", "summary": "Callstack released agent-device, a device automation CLI that gives AI agents structured access to native iOS, Android, TV, and desktop apps through token-efficient accessibility-tree snapshots and stable semantic refs such as @e2. The tool supports cross-platform sessions, scripted exploration and deterministic replay with experimental auto-healing of stale selectors via the -u flag, plus a typed TypeScript client (createAgentDeviceClient) for programmatic session and device management. Callstack also published official agent-device skills installable with 'npx skills add callstack/agent-device' to give agents canonical workflows and replay guidance.", "body_md": "# The mobile verification\n\nfor AI Agents\n\nDevice automation CLI for AI agents. Real apps on iOS, Android, TV, and desktop. Token-efficient snapshots, semantic refs, evidence captured only when needed\n\nInspired by the browser automation patterns that made web agents practical. Built for native mobile and TV apps.\n\n``` bash\n# An agent exploring a real iOS app$ agent-device open com.myapp --platform ios$ agent-device snapshot -i# @e1 [heading] \"Settings\"# @e2 [button] \"Wi-Fi\"$ agent-device press @e2$ agent-device snapshot -i\n```\n\n### Give your agent \"eyes\" on the UI\n\nAgents need a structured way to understand the current screen. snapshot exposes the accessibility tree and assigns stable refs (like @e2), keeping context smaller than raw screenshots or verbose dumps.\n\n``` bash\n# Capture only interactive elements, formatted for LLMs$ agent-device snapshot -i -c -d 4# Output:# @e1 [heading] \"Sample App\"# @e2 [button] \"Sign In\"# @e3 [input] \"Email Address\"\n```\n\n### Act with semantic precision\n\nOnce the agent understands the screen, it acts. It can use stable refs or rely on semantic selectors to find elements by text, label, or role—making flows readable and resilient.\n\n``` bash\n# Interact using stable refs...$ agent-device fill @e3 \"agent@example.com\" # ...or semantic selectors$ agent-device find \"Sign In\" click$ agent-device find role button click\n```\n\n### Cross-platform by default\n\nOne mental model across iOS, Android, and TV. The same agent workflow moves seamlessly between development simulators, physical QA devices, and TV-class targets.\n\n``` bash\n$ agent-device open com.example.app --platform android --serial emulator-5554$ agent-device open com.myapp --platform ios$ agent-device open YouTube --platform android --target tv\n```\n\n### From Exploration to E2E Replay\n\nThis is where exploration becomes infrastructure. Agents can explore a UI, save the flow to a script, and re-run it later. Experimental auto-updating (-u) can even heal stale selectors on the fly.\n\n``` bash\n# 1. Record an exploratory session$ agent-device open com.myapp --platform ios --session e2e --save-script # 2. Replay it deterministically later$ agent-device replay ~/.agent-device/sessions/e2e-run.ad # 3. Heal stale selectors automatically (Experimental)$ agent-device replay -u ~/.agent-device/sessions/e2e-run.ad\n```\n\n## Built for agents. Anywhere.\n\nWe believe all agents should have access to agent-device running real simulators and devices regardless whether they operate within your desktop or from a cloud sandbox.\n\n## Make your agent productive from the first prompt.\n\nDon't make your agent learn the tool from scratch. Install the official agent-device skills to give it the canonical workflows, targeting rules, debugging patterns, and replay guidance it needs to automate mobile apps effectively.\n\n``` bash\n# Give your agent the official toolset$ npx skills add callstack/agent-device # Add the structured exploratory QA skill$ npx skills add callstack/agent-device --skill dogfood\n```\n\n## Not just a CLI. A fully typed integration layer.\n\nUse the typed TypeScript client to integrate directly into your codebase or your custom cloud agents. Manage sessions, provision devices, and capture snapshots programmatically and deterministically.\n\n``` js\n1import { createAgentDeviceClient } from 'agent-device';2 3const client = createAgentDeviceClient({4  session: 'qa-ios',5  lockPolicy: 'reject',6});7 8// Boot the right target9const ensured = await client.simulators.ensure({10  device: 'iPhone 17 Pro',11  boot: true,12});13 14// Navigate and capture15await client.apps.open({ app: 'com.apple.Preferences', platform: 'ios' });16const snapshot = await client.capture.snapshot({ interactiveOnly: true });17 18await client.sessions.close();\n```\n\n## Built for debugging, not just the happy path.\n\nWhen things break, agents and humans both need evidence. Capture parsed logs, network dumps, and screen recordings without leaving the terminal.\n\n### Log & Network Parsing\n\n``` bash\n$ agent-device logs mark \"before submit\"$ agent-device network dump 25\n```\n\n### Visual Proof\n\n``` bash\n$ agent-device screenshot error.png$ agent-device record start fail-state.mp4\n```\n\n### Environment & Push Triggers\n\n``` bash\n$ agent-device settings airplane on$ agent-device push com.example.app '{\"aps\":{\"alert\":\"Welcome\"}}'\n```\n\n## What agents build with it\n\n### Exploratory QA\n\nNavigate apps, identify problems, collect visual evidence.\n\n### Bug Reproduction\n\nDeep link to a state, capture logs, and preserve exact steps.\n\n### Regression Loops\n\nTurn successful exploration into replayable .ad scripts.\n\n### Clean-State Testing\n\nReinstall builds, manage permissions, test onboarding.", "url": "https://wpnews.pro/news/agent-browser-equivalent-for-desktop-and-mobile-apps", "canonical_source": "https://agent-device.dev", "published_at": "2026-09-23 03:10:29+00:00", "updated_at": "2026-09-23 03:23:51.725004+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "ai-products"], "entities": ["Callstack", "agent-device", "iOS", "Android", "TypeScript", "createAgentDeviceClient"], "alternates": {"html": "https://wpnews.pro/news/agent-browser-equivalent-for-desktop-and-mobile-apps", "markdown": "https://wpnews.pro/news/agent-browser-equivalent-for-desktop-and-mobile-apps.md", "text": "https://wpnews.pro/news/agent-browser-equivalent-for-desktop-and-mobile-apps.txt", "jsonld": "https://wpnews.pro/news/agent-browser-equivalent-for-desktop-and-mobile-apps.jsonld"}}