** phone-harness** · let your agent control your phone.
Connect an AI agent — Claude Code, Codex, or any LLM — directly to your real phone with a thin, editable harness. iPhone through the Mac's iPhone Mirroring window, or Android over adb (USB or Wi‑Fi). No jailbreak, no Xcode, no WebDriverAgent, no app on the phone.
The Mac is the whole transport. iPhone: screencapture
- Vision-framework OCR
for eyes, HID-level CGEvents for hands. Android:
screencap
for eyes plus the
phone's own accessibility tree — exact text, exact boxes — and input
for
hands. Nothing between the agent and the phone. The agent writes what's
missing during execution in agent-workspace/agent_helpers.py
.
● agent: wants to open Weather
│
● ocr() → "Weather" at (400, 468)
│
● tap(400, 468) → wait_stable() → ocr() confirms the forecast
✓ done
Your phone, driven by an agent.
Paste into Claude Code or Codex:
Set up phone-harness for me. Clone https://github.com/ShawnPana/phone-harness into ~/.phone-harness (its canonical home), read `install.md` first, install it so `phone-harness` is a command on my PATH, and register it as an agent skill named phone-harness using `phone-harness skill` as the body, so you reach for it automatically. Then read `SKILL.md` for normal usage, and always read `src/phone_harness/helpers.py` because that is where the functions are. Then read `onboarding.md` and walk me through it.
The agent then follows onboarding.md: it asks one thing —
which phone is your default, iPhone or Android — and walks you through only
what needs your hands. iPhone: pairing iPhone Mirroring once, and granting
the terminal Accessibility and Screen Recording in System Settings →
Privacy & Security (Screen Recording takes effect after the terminal restarts).
Android: turning on Developer options, then either plugging in and tapping
Allow, or Wireless debugging + one 6‑digit pairing code. Then
phone-harness --doctor
verifies the chain, and phone-harness config set platform ios|android
sets the default (both can be set up).
A fresh machine may prompt for more permissions the first time an action runs
— if --doctor
passes but taps or capture silently do nothing, watch for a macOS prompt. See install.md for details.
iPhone. iPhone Mirroring (macOS Sequoia+) renders the phone as a Mac window and forwards real mouse and keyboard input as touches. That gives an agent everything it needs for real-device iOS automation:
See— capture just the mirroring window, OCR it with Apple's Vision framework: every visible string with a tap-ready coordinate. The poor man's DOM.Act— CGEvents posted at the HID tap: taps, long-presses, drags/flicks, scroll gestures, unicode typing, and the app's own shortcuts (Cmd+1 Home, Cmd+2 App Switcher, Cmd+3 Spotlight). For icon-only controls found in a screenshot, usetap_image_point(x, y, image_size=...)
: screenshot coordinates are image pixels, whiletap(x, y)
expects global Mac screen points.image_point()
performs the conversion when a rawtap()
is needed.Verify— screenshot again. No DOM means the capture is the ground truth.
Things that do NOT work, learned the hard way: AppleScript click at
(silently ignored — the window is a video stream with no accessibility tree), unicode key payloads (mirroring forwards raw HID keycodes, so typing must use keycodes), a slow touch-drag (barely moves an iOS list — use wheel scroll for lists, a fast flick for pages), and input while the window isn't frontmost (swallowed).
Android. adb reaches the phone directly, over USB or Wi‑Fi — no window, no
focus, nothing on the Mac has to be in front. screencap
is the capture; the
phone's own accessibility tree (uiautomator
) is the text source, so ocr()
returns exact strings and boxes and tap_ui("url_bar")
finds elements OCR
never could; input tap/swipe/text
are the hands; Back exists. Coordinates
are device pixels, so a screenshot is 1:1 with tap(x, y)
. The harness finds the phone itself (a plugged-in one first, else the paired Wi‑Fi phone), refuses to drive a locked one, and can keep it awake for a session without touching a setting. Same helpers, same agent code.
./phone-harness <<'PY'
open_app("Notes")
tap_text("New Note")
type_text("hello from the harness")
print([o["text"] for o in ocr()][:10])
PY
Day-to-day workflow lives in SKILL.md, which install.md
registers as an agent skill (phone-harness skill
prints the body) so the agent reaches for it on its own.
SKILL.md
— day-to-day usage (the agent-facing product surface)onboarding.md
— the first-run flow the agent walks the user throughinstall.md
— setup reference and troubleshootingsrc/phone_harness/
— protected core:transport.py
— the one seam: the op vocabulary every device sits behind, andconnect("ios"|"android")
helpers.py
— the primitives pre-imported into scripts (platform-agnostic)ios.py
,mirror.py
,background.py
,ocr.py
— the iPhone: window discovery, focus, capture, CGEvent input, Vision OCRandroid.py
— the Android: adb, the accessibility tree, USB/Wi‑Fi resolution and pairing, awake sessions, theandroid
CLIconfig.py
— settings (phone-harness config
) and remembered devicesadmin.py
—--doctor
;run.py
— the CLI (exec
stdin with helpers in scope)
agent-workspace/agent_helpers.py
— helper code the agent edits; auto-loaded into every script's namespace
Both transports are stateless per call (window bounds and captures are
re-queried; adb has its own server), so there is no daemon — every invocation
is self-contained. State that must persist (the default platform, paired
Android phones) lives in ~/.config/phone-harness
and ~/.local/state/phone-harness
.
From a checkout, use ./phone-harness
to run the working tree directly:
./phone-harness <<'PY'
print(screen_info())
PY
- iPhone: one phone, one session; unlocking the physical phone s mirroring. OCR sees text, not semantics — unlabeled icons need a screenshot + a vision-capable model.
- Android: the tree costs seconds on a slow phone and is unavailable on screens
that never go idle (read the screenshot instead);
input text
is ASCII; key chords are not a thing over adb; a PIN-locked phone needs the user. - Both: no multi-touch (no pinch), no camera/Face ID flows, DRM video renders black. Connecting the phone is always the user's job.