Your phone becomes an AI film set #
Remember those early augmented reality (AR) experiments, where a virtual object would just kind of… float in your real-world view? Simulon, a name we last heard in 2023, is back, and it’s doing a whole lot more than just floating. This new workflow combines your phone’s camera with generative video, making your living room a film set for AI characters.
Here’s how it works: you upload a character reference—say, "flamethrower girl"—and Simulon converts it into a 3D asset. Then, using your iPhone’s camera, you position and scale that character directly into your physical environment. You can place a miniature character on your desk or have a full-sized one sit in a chair.
Instead of animating frame-by-frame, you prompt an action, and the AI generates the movement. This system captures your actual environment as the baseline, maintaining precise camera tracking and real-world geometry. You won't see backgrounds "hallucinate" or objects disappear; if there's a ball on the floor, it stays there.
Crucially, this isn’t a live, real-time effect on your phone screen. While you stage the scene on your device, the final video generation and rendering happen in the cloud. Think of your phone as the director and camera operator, while powerful servers handle the heavy lifting of compositing your AI character seamlessly into your shot. The result is a high-fidelity video of your AI creations interacting with the real world.
Real locations solve AI video’s camera problem #
Simulon’s approach differs sharply from fully synthetic text-to-video models like Sora or Kling. Instead of generating an entire scene from scratch, Simulon roots its AI characters in your real-world camera footage. This grounds the camera's movement and physical setting, drastically cutting down on the background hallucinations that plague wholly generated scenes.
Imagine placing a miniature "flamethrower girl" character right on your desk, or having a full-size figure casually take a seat in a bar chair in your studio. Simulon lets you scale and position these AI characters directly within your actual environment. Your phone camera captures the real space, and the AI integrates the character, maintaining consistent perspective and lighting.
Take a tennis court scene, for example. Two AI characters can practice volleys in a neighborhood park, with the real-world court and character placement remaining consistent throughout the shot. While the actual tennis action itself can look a bit wonky, the environment and the characters’ presence within it stay remarkably stable compared to a purely AI-generated setting.
This method offers a powerful advantage: precise camera control. You dictate the camera's movement and the physical context, and Simulon ensures the AI characters adhere to that real-world framework. It’s about merging the best of augmented reality with the flexibility of generative video, giving creators more control over their virtual actors.
The fun starts when the real world pushes back #
Here’s where Simulon really starts to justify its existence: it’s not just compositing. The AI characters can react to the real world. Imagine a digital character flinching when you poke it, or a virtual cat batting at a real object, even responding to the shifting light from a physical torch. This reality-aware rendering opens up dynamic possibilities.
Traditional compositing requires scripting every interaction. Simulon, however, allows for genuine improvisation. Creators can introduce real-world elements or interventions, letting the characters and environment react spontaneously. This moves beyond static scenes to more interactive, less predictable storytelling.
Simulon demonstrated rudimentary voice interaction, with characters responding to simple commands. The roadmap promises assigning distinct voices and personalities to characters, offering a glimpse into future capabilities. For now, the reality-aware rendering and improvisational potential are the standout features, with full voice-driven personalities still an ambition.
This blending of the digital and physical is what makes Simulon compelling. You can learn more about its features and roadmap at Simulon 2.0 | The camera for your imagination. It's a system designed to encourage creative chaos, where the unexpected interactions between AI and your environment become part of the narrative.
Enjoying this? Get one like it in your inbox each morning.
one email a day · unsubscribe in two clicks · no third-party tracking
A powerful workflow, not a magic button #
Don’t mistake Simulon for a magic button. While powerful, the AR controls can be finicky, requiring practice to precisely place and scale characters. Rendering isn’t real-time, and generated details may occasionally drift or invent objects, so embrace imperfections.
Simulon shines as an accessible option for indie creators and VFX artists. It offers detailed camera control without the complexity of traditional tracking or a full 3D pipeline. The generated video outputs can also feed into other tools, like a video-to-video workflow in Kling 4.0, expanding creative possibilities.
The app is currently available for iOS, with a free entry tier offering 100 starter credits for library assets. Paid subscriptions include the Plus tier at $14.99/month and Pro at $39.99/month, providing more credits and custom subject uploads. Android availability was awaiting approval at the time of recording.
Frequently Asked Questions #
What is Simulon?
Simulon is an iPhone app that lets users stage AI-generated characters in real-world locations and create video of them there.
How does Simulon put AI characters into real spaces?
Users provide a character reference, place its generated 3D asset with their phone camera, then prompt an action for video generation.
Does Simulon generate video in real time?
No. Users stage characters on-device, but the video is generated asynchronously in the cloud.
Can Simulon characters react to people and objects?
The video demonstrates characters responding to real-world actions, lighting, and spoken prompts, though results can be inconsistent.
Is Simulon available on Android?
The video describes the app as available on iOS, with Android availability awaiting approval at the time of recording.