{"slug": "show-hn-open-source-app-that-lets-llms-control-any-iphone-app-no-mac-required", "title": "Show HN: Open-source app that lets LLMs control any iPhone app (no Mac required)", "summary": "Developer jkcoxson released an open-source iPhone app that lets a vision-capable LLM control any app on the same iPhone without a Mac at run time, requiring only a one-time USB pairing file generated from a Mac. The app runs on iOS 26 or later with Developer Mode enabled, mounts a developer image and uses a loopback VPN at 10.7.0.1 to send screenshots to a model via an OpenRouter key or any OpenAI-compatible endpoint, which returns tool calls to tap, swipe, type or open apps, at a few seconds per step. The build must be sideloaded because it needs developer services, and it requires a paid Apple developer team for the built-in VPN or a re-signed LocalDevVPN every 7 days with a free Apple ID.", "body_md": "**Let an AI model use your iPhone, from the iPhone itself. No Mac at run time (one-time pairing only), no extra hardware, no jailbreak.**\n\n*Left: \"Select a Lyft to the Eiffel Tower (don't book yet)\" (4× speed): the agent leaves its own app, opens Lyft,\ntypes the destination, picks it and stops at the ride options. Middle: the finished run, each step with its screenshot. Right: a new chat.*\n\n- **No computer at run time:** the agent runs on the iPhone and controls that same iPhone, no extra hardware or jailbreak.\nA Mac is only needed once, to make the pairing file\n- **Start it from anywhere:** the chat, Siri, the \"Run Agent\" Shortcut or an`iphoneuse://run?prompt=…` link.\n\n| Part | Notes | \n|---|---|\n| **iPhone** | iOS 26 or later, Developer Mode on. | \n| **Apple account** | **Paid team:** the app uses its own built-in VPN (needs the Network Extension entitlement).**Free Apple ID:** build with`BUILTIN_VPN=NO` (Xcode build setting or`xcodebuild` argument) and install[LocalDevVPN](https://apps.apple.com/app/localdevvpn/id6755608044) ; re-sign every 7 days. | \n| **Pairing file** | Made once from a Mac over USB (below). | \n| **Vision model** | An [OpenRouter](https://openrouter.ai/settings/keys) key, or any OpenAI-compatible endpoint whose model takes images and tool calls. | \n| **Mac with Xcode + rustup** | Only to build the app and make the pairing file. Not needed at run time. | \n\n- **Shortcut needed on cellular:** iOS refuses new loopback VPN connections on cellular data, so the app runs a\nCellular Toggle shortcut that turns data off while it connects, then back on (or turn it off by hand).\n- **No other VPN at the same time:** it needs a loopback VPN (the built-in one, or[LocalDevVPN](https://apps.apple.com/app/localdevvpn/id6755608044) ), and iOS allows one VPN at a time, so\nTailscale and similar stop while it is on.\n- **\"Viewed remotely from Mac\" notice on every run:** iOS shows \"Device Hub: This device was viewed remotely from\nMac\" when its developer services are used, even though the \"Mac\" here is the app itself. Nothing leaves the phone (except calls to LLMs).\n- **Permissions:** requires Developer Mode, a VPN configuration, Local Network access (to find the phone's\npairing service), and Location set to \"Always\" (keeps runs alive in the background, alongside silent audio).\nMicrophone only for voice prompts.\n- **Pairing files expire:** make a new one when the app says so.\n- **Sideloaded build only:** it needs Developer Mode and developer services, so it can't go on the App Store.\n- **A few seconds per step:** most of it is the model.\n\n```\nflowchart LR\n  subgraph App[\"iphone-use-app\"]\n    L[\"LLM calling\"]\n    D[\"Mount developer image\"]\n    O[\"Orchestrator\"]\n  end\n  L <-- \"screenshot + history /<br/>tool call\" --> R[\"OpenRouter\"]\n  D -- \"DDI mount\" --> V[\"Loopback VPN<br/>10.7.0.1\"]\n  O <-- \"screenshot / tap, swipe, type\" --> V\n```\n\nThe app mounts a developer image, then uses a loopback VPN to screenshot and control the iPhone it runs on. Each step, the screenshot goes to the model, which answers with a tool call (tap, swipe, type, open app) until it's done. A Live Activity shows progress with a Stop button, and every run is saved as a chat.\n\nThis is the one step that needs a Mac. There is no supported on-device API for the phone's developer services: the phone only opens them to a trusted host, the way it does for Xcode. The pairing file holds the keys of such a host, so the app can act as one for its own phone. Since iOS 17 these services are reached over a network tunnel (RemoteXPC/CoreDevice, the same path as Xcode's wireless debugging) rather than the USB mux, which is why the app can reach them through its loopback VPN with no cable. Plug the phone into the Mac, unlock it (tap Trust if asked), then:\n\n```\ngit clone https://github.com/jkcoxson/idevice && cd idevice\ncargo build --release -p idevice-tools --bin idevice-tools\n./target/release/idevice-tools rppairing pair my-mac pairingFile.plist\n```\n\nThe file keeps working until the phone stops trusting that host (Reset Location & Privacy, Developer Mode turned off). Make a new one if the app says the pairing file is invalid or expired.\n\n```\nscripts/build-idevice.sh        # once: builds iPhoneUse/idevice/libidevice_ffi.a (needs the aarch64-apple-ios target)\nopen iPhoneUse.xcodeproj        # set your team, run the iPhoneUse scheme on the phone\n```\n\n1. Settings (gear, top right) → **Import Pairing File** .<sup>1</sup>\n2. Allow the **Add VPN Configuration** prompt. It only routes`10.7.0.1` back to the phone.\nWith LocalDevVPN instead: turn it on, and pick LocalDevVPN in Settings → Phone Connection\n(picked for you if the built-in VPN fails).\n3. Settings → **Model** : pick a provider, paste the key, pick a model.\n4. Wait for the \"not connected\" line above the input to go away (the developer disk image downloads on first run).\n\n<sup>1</sup> Or copy it over once the app is installed:\n`xcrun devicectl device copy to --device <udid> --domain-type appDataContainer --domain-identifier blog.eliot.iphoneuse --source pairingFile.plist --destination Documents/pairingFile.plist`\n\nType a goal, for example *\"Open Settings and turn on Dark Mode\"*, and send it. Stop it from the Dynamic Island\nor the chat.\n\n```\nDEV=<udid> scripts/runscript.sh \"open com.apple.Preferences; wait 1.5; tap 0.5 0.9; shot a\" a\n```\n\nLaunches the app with `-agentScript`, waits for \"script done\" in `Documents/agent.log` and pulls the screenshots.\n`-agentPrompt \"...\"` as a launch argument starts a real run.\n\nThe simulator can't reach a device, but it shows the UI. `-demoChat YES` plays a fake run\n(placeholder screenshots, a question to answer), Debug builds only.\n\n1. `rustup target add aarch64-apple-ios-sim && scripts/build-idevice.sh sim` : the libs are\nsingle-arch, so the simulator needs its own, in`iPhoneUse/idevice/sim/` .\n2. Build Debug (the scheme defaults to Release, which drops `-demoChat` ) against that lib:`xcodebuild -project iPhoneUse.xcodeproj -scheme iPhoneUse -configuration Debug -destination id=<sim udid> LIBRARY_SEARCH_PATHS='$(PROJECT_DIR)/iPhoneUse/idevice/sim' build`\n3. Install the `.app` from DerivedData and launch it with`-demoChat YES` .\n\n**Symptom (before the fix):** a run froze on a step after about a minute in the background, and the\nLive Activity stayed on that step. The app had been killed: the phone's crash logs held an\n`iPhoneUse.cpu_resource_fatal-*.ips` report (\"exceeding limit of 80% cpu over 60 seconds\", \"Process killed\").\n\n**Cause:** the heartbeat thread (`Heartbeat.run()` in `DeviceSession.swift`) keeps CoreDevice services\nalive during a run. Its connection is closed right after the handshake; idevice's userspace TCP stack\n(jktcp) then answers every read with `BrokenPipe` in microseconds, and the loop retried with no delay,\nabout 45k times a second. iOS kills background apps above 80% CPU over 60 seconds.\n\nTwo things made it worse:\n\n- The idevice log was written to `Documents/idevice_log.txt` at Debug level, which prints every XPC\nmessage, screenshots included, as decimal byte lists: 178 MB of log and ~1 GB of memory in a minute.\n- The loop matched errors by the strings `HeartbeatTimeout` /`HeartbeatSleepyTime` , which never\nappear in idevice's messages (\"heartbeat timeout\", \"device went to sleep\").\n\n**Fix:** timeouts are matched on the error code (100, sub-code 2) and the loop waits again; any other\nerror ends the loop. The file log is at Info.\n\n**Still open:** why the heartbeat connection closes. Runs work without it (a 17-step, 2.5-minute run\nfinished fine); if CoreDevice services start dropping on long runs, reconnecting the heartbeat\n(with backoff) is the next step.\n\n**Debugging a killed run:** `idevicecrashreport -u <udid> -k <dir>` pulls the crash logs. Symbolicate\nwith `atos` against the dSYM of the exact build (the UUID in the report must match\n`dwarfdump --uuid`).\n\nFeature scope: [docs/FEATURES.md](https://github.com/EliotAndres/iphone-use-app/blob/main/docs/FEATURES.md).\n\nOnly automate devices and accounts you own or may use, and respect the terms of the apps involved. The agent acts with your hands: watch it, and keep Stop within reach.\n\n| Project | Repo | Used for | \n|---|---|---|\n| StikDebug, by Stephen and contributors | [StikDebug/StikDebug](https://github.com/StikDebug/StikDebug) | The tunnel, pairing file and developer disk image code is based on it | \n| idevice, by jkcoxson | [jkcoxson/idevice](https://github.com/jkcoxson/idevice) | The Rust library behind every device call | \n| StosVPN, by the SideStore Team and Stossy11 | [SideStore/StosVPN](https://github.com/SideStore/StosVPN) | The loopback VPN ( `TunnelProvider/` ) is based on it | \n| LocalDevVPN, by jkcoxson | [jkcoxson/LocalDevVPN](https://github.com/jkcoxson/LocalDevVPN) | The VPN app for free Apple IDs; our tunnel came from StosVPN through it | \n| esp32-iphone-agent, by Henry Yuan | [onehumanbeing/esp32-iphone-agent](https://github.com/onehumanbeing/esp32-iphone-agent) | The model provider and model picker design, and the layout of this README | \n| DeveloperDiskImage, by doronz88 | [doronz88/DeveloperDiskImage](https://github.com/doronz88/DeveloperDiskImage) | Developer disk images | \n\nAGPL-3.0. See [`LICENSE`](https://github.com/EliotAndres/iphone-use-app/blob/main/LICENSE).", "url": "https://wpnews.pro/news/show-hn-open-source-app-that-lets-llms-control-any-iphone-app-no-mac-required", "canonical_source": "https://github.com/EliotAndres/iphone-use-app", "published_at": "2026-10-19 19:55:48+00:00", "updated_at": "2026-10-19 20:17:28.142249+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "ai-products"], "entities": ["jkcoxson", "iPhone", "iOS 26", "OpenRouter", "LocalDevVPN", "Apple", "Xcode", "idevice"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/show-hn-open-source-app-that-lets-llms-control-any-iphone-app-no-mac-required", "markdown": "https://wpnews.pro/news/show-hn-open-source-app-that-lets-llms-control-any-iphone-app-no-mac-required.md", "text": "https://wpnews.pro/news/show-hn-open-source-app-that-lets-llms-control-any-iphone-app-no-mac-required.txt", "jsonld": "https://wpnews.pro/news/show-hn-open-source-app-that-lets-llms-control-any-iphone-app-no-mac-required.jsonld"}}