AI Controls IPhone By way of ESP32

Do you need to get an AI agent to do one thing to an iPhone? You may try to jailbreak the machine and do some heavy software program hacking, or… you can seize a microcontroller off the shelf and have it do the laborious work. [Henry Yuan] explains how to achieve this with an ESP32.

The first step is getting an AI agent entry to the present state of the telephone display screen—which [Henry] achieves with AirPlay display screen mirroring and UxPlay. A Gstreamer pipeline is then used to feed the display screen as a JPEG stream to the AI agent of your selection. Then comes management, which is achieved by an ESP32-S3 performing as a USB-HID machine, emulating each USB mouse and keyboard. With iOS AssistiveTouch enabled, this provides the agent management over inputs to the machine. From there, it’s as much as the AI agent to determine the actions to make to realize the specified conduct, which is clearly very use-case particular.

It would possibly sound a little bit convoluted, however this strategy has advantages. There’s no requirement to have an Apple Developer Account, or XCode, or a Macintosh in any respect. You can simply get your agent tinkering away on the telephone with none of that fussy overhead. We’ve featured some other great iPhone hacks of late, as effectively. Video after the break.