# Show HN: Phone-harness – let your agent control your phone

> Source: <https://github.com/ShawnPana/phone-harness>
> Published: 2026-08-18 15:01:08+00:00

** phone-harness** · let your agent control your phone.

Connect an AI agent — Claude Code, Codex, or any LLM — directly to your real
phone with a thin, editable harness. **iPhone** through the Mac's iPhone
Mirroring window, or **Android** over adb (USB or Wi‑Fi). No jailbreak, no
Xcode, no WebDriverAgent, no app on the phone.

The Mac is the whole transport. iPhone: `screencapture`

+ Vision-framework OCR
for eyes, HID-level CGEvents for hands. Android: `screencap`

for eyes plus the
phone's own accessibility tree — exact text, exact boxes — and `input`

for
hands. Nothing between the agent and the phone. The agent writes what's
missing during execution in `agent-workspace/agent_helpers.py`

.

```
  ● agent: wants to open Weather
  │
  ● ocr() → "Weather" at (400, 468)
  │
  ● tap(400, 468) → wait_stable() → ocr() confirms the forecast
  ✓ done
```

**Your phone, driven by an agent.**

Paste into Claude Code or Codex:

```
Set up phone-harness for me. Clone https://github.com/ShawnPana/phone-harness into ~/.phone-harness (its canonical home), read `install.md` first, install it so `phone-harness` is a command on my PATH, and register it as an agent skill named phone-harness using `phone-harness skill` as the body, so you reach for it automatically. Then read `SKILL.md` for normal usage, and always read `src/phone_harness/helpers.py` because that is where the functions are. Then read `onboarding.md` and walk me through it.
```

The agent then follows [onboarding.md](/ShawnPana/phone-harness/blob/main/onboarding.md): it asks one thing —
which phone is your default, iPhone or Android — and walks you through only
what needs your hands. **iPhone:** pairing iPhone Mirroring once, and granting
the terminal **Accessibility** and **Screen Recording** in System Settings →
Privacy & Security (Screen Recording takes effect after the terminal restarts).
**Android:** turning on Developer options, then either plugging in and tapping
Allow, or Wireless debugging + one 6‑digit pairing code. Then
`phone-harness --doctor`

verifies the chain, and `phone-harness config set platform ios|android`

sets the default (both can be set up).

A fresh machine may prompt for more permissions the first time an action runs
— if `--doctor`

passes but taps or capture silently do nothing, watch for a
macOS prompt. See [install.md](/ShawnPana/phone-harness/blob/main/install.md) for details.

**iPhone.** iPhone Mirroring (macOS Sequoia+) renders the phone as a Mac window
and forwards real mouse and keyboard input as touches. That gives an agent
everything it needs for real-device iOS automation:

**See**— capture just the mirroring window, OCR it with Apple's Vision framework: every visible string with a tap-ready coordinate. The poor man's DOM.**Act**— CGEvents posted at the HID tap: taps, long-presses, drags/flicks, scroll gestures, unicode typing, and the app's own shortcuts (Cmd+1 Home, Cmd+2 App Switcher, Cmd+3 Spotlight). For icon-only controls found in a screenshot, use`tap_image_point(x, y, image_size=...)`

: screenshot coordinates are image pixels, while`tap(x, y)`

expects global Mac screen points.`image_point()`

performs the conversion when a raw`tap()`

is needed.**Verify**— screenshot again. No DOM means the capture is the ground truth.

Things that do NOT work, learned the hard way: AppleScript `click at`

(silently
ignored — the window is a video stream with no accessibility tree), unicode key
payloads (mirroring forwards raw HID keycodes, so typing must use keycodes), a
slow touch-drag (barely moves an iOS list — use wheel scroll for lists, a fast
flick for pages), and input while the window isn't frontmost (swallowed).

**Android.** adb reaches the phone directly, over USB or Wi‑Fi — no window, no
focus, nothing on the Mac has to be in front. `screencap`

is the capture; the
phone's own accessibility tree (`uiautomator`

) is the text source, so `ocr()`

returns exact strings and boxes and `tap_ui("url_bar")`

finds elements OCR
never could; `input tap/swipe/text`

are the hands; Back exists. Coordinates
are device pixels, so a screenshot is 1:1 with `tap(x, y)`

. The harness finds
the phone itself (a plugged-in one first, else the paired Wi‑Fi phone),
refuses to drive a locked one, and can keep it awake for a session without
touching a setting. Same helpers, same agent code.

```
./phone-harness <<'PY'
open_app("Notes")
tap_text("New Note")
type_text("hello from the harness")
print([o["text"] for o in ocr()][:10])
PY
```

Day-to-day workflow lives in [SKILL.md](/ShawnPana/phone-harness/blob/main/SKILL.md), which [install.md](/ShawnPana/phone-harness/blob/main/install.md)
registers as an agent skill (`phone-harness skill`

prints the body) so the agent
reaches for it on its own.

`SKILL.md`

— day-to-day usage (the agent-facing product surface)`onboarding.md`

— the first-run flow the agent walks the user through`install.md`

— setup reference and troubleshooting`src/phone_harness/`

— protected core:`transport.py`

— the one seam: the op vocabulary every device sits behind, and`connect("ios"|"android")`

`helpers.py`

— the primitives pre-imported into scripts (platform-agnostic)`ios.py`

,`mirror.py`

,`background.py`

,`ocr.py`

— the iPhone: window discovery, focus, capture, CGEvent input, Vision OCR`android.py`

— the Android: adb, the accessibility tree, USB/Wi‑Fi resolution and pairing, awake sessions, the`android`

CLI`config.py`

— settings (`phone-harness config`

) and remembered devices`admin.py`

—`--doctor`

;`run.py`

— the CLI (`exec`

stdin with helpers in scope)

`agent-workspace/agent_helpers.py`

— helper code the agent edits; auto-loaded into every script's namespace

Both transports are stateless per call (window bounds and captures are
re-queried; adb has its own server), so there is no daemon — every invocation
is self-contained. State that must persist (the default platform, paired
Android phones) lives in `~/.config/phone-harness`

and `~/.local/state/phone-harness`

.

From a checkout, use `./phone-harness`

to run the working tree directly:

```
./phone-harness <<'PY'
print(screen_info())
PY
```

- iPhone: one phone, one session; unlocking the physical phone pauses mirroring. OCR sees text, not semantics — unlabeled icons need a screenshot + a vision-capable model.
- Android: the tree costs seconds on a slow phone and is unavailable on screens
that never go idle (read the screenshot instead);
`input text`

is ASCII; key chords are not a thing over adb; a PIN-locked phone needs the user. - Both: no multi-touch (no pinch), no camera/Face ID flows, DRM video renders black. Connecting the phone is always the user's job.