{"slug": "show-hn-phone-harness-let-your-agent-control-your-phone", "title": "Show HN: Phone-harness – let your agent control your phone", "summary": "ShawnPana released phone-harness, an open-source tool that lets AI agents like Claude Code or Codex control a real iPhone or Android phone from a Mac, using iPhone Mirroring or adb without jailbreaking or installing an app. The tool, available on GitHub, provides OCR-based vision and HID-level input for iPhone and accessibility-tree-based control for Android, with setup requiring one-time permissions and a --doctor verification command.", "body_md": "** phone-harness** · let your agent control your phone.\n\nConnect an AI agent — Claude Code, Codex, or any LLM — directly to your real\nphone with a thin, editable harness. **iPhone** through the Mac's iPhone\nMirroring window, or **Android** over adb (USB or Wi‑Fi). No jailbreak, no\nXcode, no WebDriverAgent, no app on the phone.\n\nThe Mac is the whole transport. iPhone: `screencapture`\n\n+ Vision-framework OCR\nfor eyes, HID-level CGEvents for hands. Android: `screencap`\n\nfor eyes plus the\nphone's own accessibility tree — exact text, exact boxes — and `input`\n\nfor\nhands. Nothing between the agent and the phone. The agent writes what's\nmissing during execution in `agent-workspace/agent_helpers.py`\n\n.\n\n```\n  ● agent: wants to open Weather\n  │\n  ● ocr() → \"Weather\" at (400, 468)\n  │\n  ● tap(400, 468) → wait_stable() → ocr() confirms the forecast\n  ✓ done\n```\n\n**Your phone, driven by an agent.**\n\nPaste into Claude Code or Codex:\n\n```\nSet up phone-harness for me. Clone https://github.com/ShawnPana/phone-harness into ~/.phone-harness (its canonical home), read `install.md` first, install it so `phone-harness` is a command on my PATH, and register it as an agent skill named phone-harness using `phone-harness skill` as the body, so you reach for it automatically. Then read `SKILL.md` for normal usage, and always read `src/phone_harness/helpers.py` because that is where the functions are. Then read `onboarding.md` and walk me through it.\n```\n\nThe agent then follows [onboarding.md](/ShawnPana/phone-harness/blob/main/onboarding.md): it asks one thing —\nwhich phone is your default, iPhone or Android — and walks you through only\nwhat needs your hands. **iPhone:** pairing iPhone Mirroring once, and granting\nthe terminal **Accessibility** and **Screen Recording** in System Settings →\nPrivacy & Security (Screen Recording takes effect after the terminal restarts).\n**Android:** turning on Developer options, then either plugging in and tapping\nAllow, or Wireless debugging + one 6‑digit pairing code. Then\n`phone-harness --doctor`\n\nverifies the chain, and `phone-harness config set platform ios|android`\n\nsets the default (both can be set up).\n\nA fresh machine may prompt for more permissions the first time an action runs\n— if `--doctor`\n\npasses but taps or capture silently do nothing, watch for a\nmacOS prompt. See [install.md](/ShawnPana/phone-harness/blob/main/install.md) for details.\n\n**iPhone.** iPhone Mirroring (macOS Sequoia+) renders the phone as a Mac window\nand forwards real mouse and keyboard input as touches. That gives an agent\neverything it needs for real-device iOS automation:\n\n**See**— capture just the mirroring window, OCR it with Apple's Vision framework: every visible string with a tap-ready coordinate. The poor man's DOM.**Act**— CGEvents posted at the HID tap: taps, long-presses, drags/flicks, scroll gestures, unicode typing, and the app's own shortcuts (Cmd+1 Home, Cmd+2 App Switcher, Cmd+3 Spotlight). For icon-only controls found in a screenshot, use`tap_image_point(x, y, image_size=...)`\n\n: screenshot coordinates are image pixels, while`tap(x, y)`\n\nexpects global Mac screen points.`image_point()`\n\nperforms the conversion when a raw`tap()`\n\nis needed.**Verify**— screenshot again. No DOM means the capture is the ground truth.\n\nThings that do NOT work, learned the hard way: AppleScript `click at`\n\n(silently\nignored — the window is a video stream with no accessibility tree), unicode key\npayloads (mirroring forwards raw HID keycodes, so typing must use keycodes), a\nslow touch-drag (barely moves an iOS list — use wheel scroll for lists, a fast\nflick for pages), and input while the window isn't frontmost (swallowed).\n\n**Android.** adb reaches the phone directly, over USB or Wi‑Fi — no window, no\nfocus, nothing on the Mac has to be in front. `screencap`\n\nis the capture; the\nphone's own accessibility tree (`uiautomator`\n\n) is the text source, so `ocr()`\n\nreturns exact strings and boxes and `tap_ui(\"url_bar\")`\n\nfinds elements OCR\nnever could; `input tap/swipe/text`\n\nare the hands; Back exists. Coordinates\nare device pixels, so a screenshot is 1:1 with `tap(x, y)`\n\n. The harness finds\nthe phone itself (a plugged-in one first, else the paired Wi‑Fi phone),\nrefuses to drive a locked one, and can keep it awake for a session without\ntouching a setting. Same helpers, same agent code.\n\n```\n./phone-harness <<'PY'\nopen_app(\"Notes\")\ntap_text(\"New Note\")\ntype_text(\"hello from the harness\")\nprint([o[\"text\"] for o in ocr()][:10])\nPY\n```\n\nDay-to-day workflow lives in [SKILL.md](/ShawnPana/phone-harness/blob/main/SKILL.md), which [install.md](/ShawnPana/phone-harness/blob/main/install.md)\nregisters as an agent skill (`phone-harness skill`\n\nprints the body) so the agent\nreaches for it on its own.\n\n`SKILL.md`\n\n— day-to-day usage (the agent-facing product surface)`onboarding.md`\n\n— the first-run flow the agent walks the user through`install.md`\n\n— setup reference and troubleshooting`src/phone_harness/`\n\n— protected core:`transport.py`\n\n— the one seam: the op vocabulary every device sits behind, and`connect(\"ios\"|\"android\")`\n\n`helpers.py`\n\n— the primitives pre-imported into scripts (platform-agnostic)`ios.py`\n\n,`mirror.py`\n\n,`background.py`\n\n,`ocr.py`\n\n— the iPhone: window discovery, focus, capture, CGEvent input, Vision OCR`android.py`\n\n— the Android: adb, the accessibility tree, USB/Wi‑Fi resolution and pairing, awake sessions, the`android`\n\nCLI`config.py`\n\n— settings (`phone-harness config`\n\n) and remembered devices`admin.py`\n\n—`--doctor`\n\n;`run.py`\n\n— the CLI (`exec`\n\nstdin with helpers in scope)\n\n`agent-workspace/agent_helpers.py`\n\n— helper code the agent edits; auto-loaded into every script's namespace\n\nBoth transports are stateless per call (window bounds and captures are\nre-queried; adb has its own server), so there is no daemon — every invocation\nis self-contained. State that must persist (the default platform, paired\nAndroid phones) lives in `~/.config/phone-harness`\n\nand `~/.local/state/phone-harness`\n\n.\n\nFrom a checkout, use `./phone-harness`\n\nto run the working tree directly:\n\n```\n./phone-harness <<'PY'\nprint(screen_info())\nPY\n```\n\n- iPhone: one phone, one session; unlocking the physical phone pauses mirroring. OCR sees text, not semantics — unlabeled icons need a screenshot + a vision-capable model.\n- Android: the tree costs seconds on a slow phone and is unavailable on screens\nthat never go idle (read the screenshot instead);\n`input text`\n\nis ASCII; key chords are not a thing over adb; a PIN-locked phone needs the user. - Both: no multi-touch (no pinch), no camera/Face ID flows, DRM video renders black. Connecting the phone is always the user's job.", "url": "https://wpnews.pro/news/show-hn-phone-harness-let-your-agent-control-your-phone", "canonical_source": "https://github.com/ShawnPana/phone-harness", "published_at": "2026-08-18 15:01:08+00:00", "updated_at": "2026-08-18 15:12:28.701619+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "artificial-intelligence"], "entities": ["ShawnPana", "phone-harness", "Claude Code", "Codex", "GitHub", "iPhone Mirroring", "Android", "Mac"], "alternates": {"html": "https://wpnews.pro/news/show-hn-phone-harness-let-your-agent-control-your-phone", "markdown": "https://wpnews.pro/news/show-hn-phone-harness-let-your-agent-control-your-phone.md", "text": "https://wpnews.pro/news/show-hn-phone-harness-let-your-agent-control-your-phone.txt", "jsonld": "https://wpnews.pro/news/show-hn-phone-harness-let-your-agent-control-your-phone.jsonld"}}