{"slug": "esp32-iphone-agent-let-an-ai-model-use-an-iphone-via-esp32-airplay-usb-hid", "title": "ESP32-iPhone-agent: Let an AI model use an iPhone via ESP32, AirPlay, USB HID", "summary": "Developer onehumanbeing released the ESP32-iPhone-agent, an open-source project that lets an AI model control a real iPhone using a $7 ESP32-S3 board as an AirPlay receiver and USB HID mouse and keyboard, with no computer and no jailbreak required. The board opens its own Wi-Fi hotspot and appears in the iPhone's Screen Mirroring list as \"ESP32 Agent,\" forwards the H.264 stream over BLE to an app running on the same phone, which sends screenshots to any OpenAI-compatible vision model and executes tap, swipe, draw, type and key actions via USB HID. The project requires an ESP32-S3 board with PSRAM (tested on a Seeed Studio XIAO ESP32S3, 8 MB PSRAM), an iPhone on iOS 18 or later with cellular data, and a Mac with Xcode only for initial flashing; the earlier computer-based version is preserved as tag v0.1.0 and branch legacy.", "body_md": "**Let an AI model use a real iPhone with a $7 ESP32. Just the phone and the board: no computer, no jailbreak.**\n\nEnglish · [简体中文](https://github.com/onehumanbeing/esp32-iphone-agent/blob/main/README.zh-CN.md)\n\nDemo video and updates: [X @iamhenryyuan](https://x.com/iamhenryyuan) · [YouTube demo](https://www.youtube.com/watch?v=Ly4im-P4jYg)\n\n**New: the computer is gone.** The board is now the AirPlay receiver and the mouse, and the agent runs in an iPhone\napp on the same phone. The original version (a Mac/Linux computer running UxPlay and a Python server) is kept as\ntag [`v0.1.0`](https://github.com/onehumanbeing/esp32-iphone-agent/tree/v0.1.0) and branch\n[`legacy`](https://github.com/onehumanbeing/esp32-iphone-agent/tree/legacy). See [docs/legacy.md](https://github.com/onehumanbeing/esp32-iphone-agent/blob/main/docs/legacy.md).\n\n*\"Set an alarm for 8:00 AM that repeats every Monday and Tuesday, then save it\"*: the agent leaves its own app,\nopens Clock, types the time into the wheel, ticks the days and saves. (Photo from v0.1.0; the task is the same.)\n\n*Left: the real phone. Right: what the agent sees through the board's AirPlay mirror. The second shot is the pointer calibration.*\n\n```\niPhone ──Screen Mirroring (Wi-Fi)──▶ ESP32-S3: AirPlay receiver ──BLE──▶ ESP32 Agent app (same iPhone)\n   ▲                                    own hotspot, H.264 passthrough        decode → vision model → action\n   └──────── USB HID mouse + keyboard ◀──────────── BLE commands ◀────────────────────┘\n```\n\n- **Eyes — AirPlay to the board.** The board opens its own Wi-Fi hotspot and appears in the phone's Screen Mirroring\nlist as*ESP32 Agent* . It keeps the H.264 stream (no decoding on the board) and forwards it to the app over BLE.\nThe phone keeps using cellular data for the model while it is on the board's hotspot.\n- **Hands — USB HID.** The board is a standard USB mouse and keyboard. With AssistiveTouch on, the mouse moves an\non-screen pointer, so a click is a tap and a drag is a swipe.\n- **Brain — the app.** The app decodes the stream, sends screenshots to any OpenAI-compatible vision model, and runs\na planner and an actor: tap, swipe, draw, type, press keys, go home, finish. It runs in the background while it\nworks; the Dynamic Island shows what it is doing.\n- **Closing the loop.** The pointer is relative, so the app tracks it and re-homes it in a corner; before precise taps\nit wiggles the pointer, finds it on screen and corrects the miss. After each action it waits for the screen to\nsettle and tells the model whether anything changed. A lost board, hotspot or mirroring link pauses the run and\nresumes it from the same step.\n- **Review.** Every run is kept on the phone: replay it step by step with the taps and swipes drawn on top.\n\n| Part | Notes | \n|---|---|\n| **ESP32-S3 board** | Tested with a [Seeed Studio XIAO ESP32S3](https://wiki.seeedstudio.com/xiao_esp32s3_getting_started/) (plain version, 8 MB PSRAM; not Sense, not Plus). About $7. Any ESP32-S3 with native USB on its USB-C port and PSRAM should work. | \n| **iPhone** | iOS 18 or later, with cellular data (the Wi-Fi is taken by the board's hotspot). USB-C models plug in directly; Lightning models need a Lightning to USB Camera Adapter. The phone powers the board. | \n| **Vision model** | Any OpenAI-compatible chat API that accepts images: a cloud API or your own server. | \n| **Mac with Xcode** | Only to flash the board and build the app once. Not needed at run time. | \n\nInstall [arduino-cli](https://arduino.github.io/arduino-cli/), the ESP32 core 3.3.x and NimBLE-Arduino 2.5.x, then:\n\n```\narduino-cli lib install \"NimBLE-Arduino@2.5.1\"\nhardware/flash.sh            # optional: [serial port] [--debug]\n```\n\nAlways build with `flash.sh`: it applies the BLE buffer sizes in `hardware/hid_bridge/nimble_cfg.h`.\n\n```\ncp ios/Config/Local.xcconfig.example ios/Config/Local.xcconfig   # set DEVELOPMENT_TEAM and IA_BUNDLE_ID\nopen ios/IPhoneAgent.xcodeproj                                    # run the IPhoneAgent scheme on your iPhone\n```\n\nThe project is generated with [XcodeGen](https://github.com/yonaskolb/XcodeGen) from `ios/project.yml`\n(`cd ios && xcodegen`). A free Apple ID can sign it (re-sign every 7 days). The one-tap hotspot join needs the\nHotspot Configuration capability (a paid account); without it the app shows the hotspot password to join by hand.\n\nPlug the board into the iPhone and open **ESP32 Agent**. Setup walks through four steps and checks each one itself:\n\n1. **Plug Me In** — pair the board over Bluetooth.\n2. **Let Me See the Screen** — join the board's Wi-Fi, then Control Center → Screen Mirroring →*ESP32 Agent* .\n3. **Let Me Use the Pointer** — turn on AssistiveTouch; the app calibrates the pointer by itself.\n4. **Give Me a Brain** — enter your model endpoint and key (stored in the Keychain).\n\nType a goal (for example *\"Open Settings and go to Wi-Fi\"*) and press **Start**. Stop it from the Dynamic Island,\nor come back to the app. Siri, Shortcuts, widgets and Control Center can run saved tasks.\n\n- Any OpenAI-compatible vision endpoint works (vLLM, SGLang, llama.cpp, Ollama, cloud APIs). The app lists the endpoint's models and picks one; you can override it, and set a separate planner and actor model.\n- The prompts are plain text in [`prompts/`](https://github.com/onehumanbeing/esp32-iphone-agent/blob/main/prompts) and bundled with the app: edit them, rebuild, done.\n- The project was developed with [nvidia/Qwen3.8-Flash-Next-NVFP4](https://huggingface.co/nvidia/Qwen3.8-Flash-Next-NVFP4) served on two DGX Sparks ([TensorFold](https://github.com/MiaAI-Lab/Qwen3.8-Flash-Dual-DGX-Sparks-TensorFold) ).\n\n```\nios/        the app (SwiftUI): BLE link, H.264 decode, agent, setup, replay, Live Activity, widgets; tests\nprompts/    planner / actor prompts used by the app\nhardware/   ESP32-S3 firmware (Arduino): BLE GATT, hotspot, AirPlay receiver, USB HID; flash script,\n            host tests, Mac BLE test harness (hardware/tools)\ndocs/       BLE protocol, legacy version notes\nserver/     legacy v0.1.0 computer-side server; not used by this version (see server/README.md)\n```\n\nDebugging without the phone app: `hardware/tools/board_test.py` plays the app over BLE from a Mac\n([hardware/tools/README.md](https://github.com/onehumanbeing/esp32-iphone-agent/blob/main/hardware/tools/README.md)). Board ↔ app protocol: [docs/ble-protocol.md](https://github.com/onehumanbeing/esp32-iphone-agent/blob/main/docs/ble-protocol.md).\n\n- Drop the computer: the board mirrors and clicks, the app on the phone thinks\n- Landscape apps (coordinates and calibration assume portrait)\n- Non-ASCII text input (e.g. Chinese through the iOS pinyin keyboard)\n- Faster steps: several actions per model call, pointer checks only when needed\n\n- BLE carries about 300-500 kbps, so the screen is mirrored small (480×1040, ≤5 fps); each step takes a few seconds, most of it the model.\n- iOS does not mirror some screens (passcode entry, Apple Pay, DRM video); the agent cannot see them. Mirroring stops when the phone locks.\n- While the agent runs, the phone's Wi-Fi is the board's hotspot (no internet over it); the model is reached over cellular data.\n- The app leaves itself with a private iOS API (`suspend` ); fine for a self-built app, to be replaced before the App Store.\n- Keyboard input is ASCII only. Landscape apps are not supported yet.\n\nOnly automate devices and accounts you own or are allowed to use, and respect the terms of the apps involved. The agent acts with your hands: watch it, and keep the stop button within reach.\n\nIssues and pull requests are welcome; see [CONTRIBUTING.md](https://github.com/onehumanbeing/esp32-iphone-agent/blob/main/CONTRIBUTING.md). Security reports: [SECURITY.md](https://github.com/onehumanbeing/esp32-iphone-agent/blob/main/SECURITY.md).\n\n| Project | Approach | Difference from this project | \n|---|---|---|\n| [VersaiOS](https://github.com/zhaodashuaibi/VersaiOS) | UxPlay mirroring + ESP32-S3 Bluetooth mouse + vision model | Needs a computer; one tap per instruction | \n| [ja1chen/actell](https://github.com/ja1chen/actell) | Agent app on the iPhone via Apple's CoreDevice debug channel | Needs Developer Mode and a Developer Disk Image | \n| [AidenAI-IO/aiden-firmware](https://github.com/AidenAI-IO/aiden-firmware) | HDMI capture + USB keyboard/mouse + multimodal model | Capture hardware; mainly Android | \n| [yoyicue/glassbox](https://github.com/yoyicue/glassbox) | PicoKVM HDMI capture + USB HID | macOS controller and PicoKVM | \n| [boboabw/phone-agent](https://github.com/boboabw/phone-agent) | ESP32-S3 Bluetooth HID + AssistiveTouch, agent as an iPhone app | Closest to this version; this one also brings the eyes (AirPlay on the board) | \n| [VKCOM/devicehub](https://github.com/VKCOM/devicehub) | ESP32 Bluetooth mouse for iOS device farms | Device-farm tooling, no AI agent | \n\nWebDriverAgent/XCUITest-based tools such as [mobile-mcp](https://github.com/mobile-next/mobile-mcp) and\n[Midscene](https://github.com/web-infra-dev/midscene) need a Mac and code signing.\n\n[UxPlay](https://github.com/FDH2/UxPlay) (the AirPlay receiver the firmware's is ported from) ·\n[NimBLE-Arduino](https://github.com/h2zero/NimBLE-Arduino) ·\n[Arduino ESP32 core](https://github.com/espressif/arduino-esp32) and [TinyUSB](https://github.com/hathach/tinyusb)\n\n[MIT](https://github.com/onehumanbeing/esp32-iphone-agent/blob/main/LICENSE), except `hardware/hid_bridge/src/airplay/` (ported from UxPlay), which is\n[GPL-3.0](https://github.com/onehumanbeing/esp32-iphone-agent/blob/main/hardware/hid_bridge/src/airplay/LICENSE-GPL.md); the firmware binary that links it is therefore\ndistributed under GPL-3.0.", "url": "https://wpnews.pro/news/esp32-iphone-agent-let-an-ai-model-use-an-iphone-via-esp32-airplay-usb-hid", "canonical_source": "https://github.com/onehumanbeing/esp32-iphone-agent", "published_at": "2026-10-15 12:11:25+00:00", "updated_at": "2026-10-15 12:20:45.078571+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "computer-vision", "developer-tools", "ai-products"], "entities": ["ESP32-iPhone-agent", "onehumanbeing", "ESP32-S3", "Seeed Studio XIAO ESP32S3", "iPhone", "iOS 18", "AirPlay", "OpenAI"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/esp32-iphone-agent-let-an-ai-model-use-an-iphone-via-esp32-airplay-usb-hid", "markdown": "https://wpnews.pro/news/esp32-iphone-agent-let-an-ai-model-use-an-iphone-via-esp32-airplay-usb-hid.md", "text": "https://wpnews.pro/news/esp32-iphone-agent-let-an-ai-model-use-an-iphone-via-esp32-airplay-usb-hid.txt", "jsonld": "https://wpnews.pro/news/esp32-iphone-agent-let-an-ai-model-use-an-iphone-via-esp32-airplay-usb-hid.jsonld"}}