AI Controls iPhone Via ESP32

Henry Yuan published a GitHub project, esp32-iphone-agent, that lets an AI agent control an iPhone by combining AirPlay screen mirroring through UxPlay with an ESP32-S3 acting as a USB-HID mouse and keyboard. A Gstreamer pipeline feeds the mirrored screen to the AI agent as a JPEG stream, and with iOS AssistiveTouch enabled the agent can issue touch inputs without an Apple Developer Account, Xcode, or a Mac.

Do you want to get an AI agent to do something to an iPhone? You could try and jailbreak the device and do some heavy software hacking, or… you could grab a microcontroller off the shelf and have it do the hard work. \ Henry Yuan\ explains how to achieve this with an ESP32. https://github.com/onehumanbeing/esp32-iphone-agent The first step is getting an AI agent access to the current state of the phone screen—which Henry achieves with AirPlay screen mirroring and UxPlay. A Gstreamer pipeline is then used to feed the screen as a JPEG stream to the AI agent of your choice. Then comes control, which is achieved by an ESP32-S3 acting as a USB-HID device, emulating both USB mouse and keyboard. With iOS AssistiveTouch enabled, this gives the agent control over inputs to the device. From there, it’s up to the AI agent to figure out the actions to make to achieve the desired behavior, which is obviously very use-case specific. It might sound a little convoluted, but this approach has benefits. There’s no requirement to have an Apple Developer Account, or XCode, or a Macintosh at all. You can just get your agent tinkering away on the phone without any of that fussy overhead. We’ve featured some other great iPhone hacks of late https://hackaday.com/2024/09/30/iphone-15-gets-dual-sim-through-fpc-patch/ , as well. Video after the break.