Large language models (LLMs) are generally thought of as machines that accept textual prompts and spit out textual content. However, if you’re creative in the way you interface with them, you can get them to do a wider range of tasks. For example, [Andrea Ricci] figured out how to get one to play DOOM.
For this project, [Andrea] began by porting the game to the SCINTIX P4. It’s a rather interesting device, being a single board designed in the Raspberry Pi CM4/CM5 form factor, but carrying an ESP32-P4 and an ESP32-C6 instead. The game runs on the P4 and is displayed on a 1024×600 MIPI DSI panel, but it’s only stepped through a few frames at a time. These frames are then passed to Claude Sonnet via a WebSockets setup. With only the same information as a human player would get, the LLM has to figure out what it’s looking at, and then respond with movement and fire commands to play the game. It’s quite interesting to watch the system play—the LLM mostly accurately describes the game world, navigates down corridors, opens doors, and shoots at enemies. There is a bit of work behind the scenes to enable it to see and understand the game world—namely, using a depth fan across the field of view so it can figure out where walls are and how not to bang into them. There’s also an ASCII automap used to allow the system to keep track of where it has already been. But fundamentally, the LLM is playing the game without any other sort of additional assistance.
We’ve seen some other great ways in which AIs have been whipped up to play various games, like Trackmania.