How to Actually Control AI Video

Matt Shumer, a developer working on AI video generation, has introduced a new technique for controlling AI video output that combines a Three.js scene with image restyling and video generation. The method involves recording a Three.js scene, restyling a frame with GPT Image, and using both as references in Seedance to achieve both motion control and desired style. Shumer reports mixed results initially but found the added step significantly improves style fidelity while preserving motion.

I’ve spent a lot of this year working on video generation. You can see a bunch of examples on my Twitter https://x.com/mattshumer . I’ve also spent quite a bit of time on 3D work, including my recent Gauntlet Loop /gauntlet-loop . That one went extremely viral a couple weeks ago and kicked off a massive wave of AI-generated games and 3D scenes. Through all the video work, one thing has been painfully clear: these models are extremely hard to control. I’ve tried to account for that in a bunch of ways. Sampling dozens of videos at a time and using Gemini and other tricks to wade through them and pick the best ones. Extracting scripts. Composing multiple models to get something done. All of that helps. I’ve still struggled to get really good scenes with the movement I want and the style I want, locked in at the same time. Eventually this will all be doable with pure video models. For now it remains a little tricky. So I’m always on the hunt for new techniques. Then I found this tweet from GMI Cloud https://x.com/gmi cloud/status/2086976751335551335 . The idea was simple and kind of obvious once you see it: code a raw Three.js scene with the exact camera movement, character motion, and timing already baked in, keep tweaking that for pennies, record it, and feed the footage into a video model they used MiniMax H3 to cook the realism. Interesting technique. I could see it working really well. So I tried it. I got mixed results. The motion control was genuinely good, but the style was not. It followed the skeleton, but the look really wasn’t that dramatically changed from the original Three.js recording. After a little experimentation, I built a new step on top of that idea that makes it a lot more useful, especially for style. Instead of going straight from the Three.js recording into the video model, I do this: - Build the Three.js scene or any rough draft with the camera, action, and timing you want. Record it. - Pull a main frame from that clip. - Pass that frame to GPT Image or honestly any strong image model and ask it to restyle the frame into the look you want, without changing the composition. - Pass both that styled still and the original Three.js video into Seedance as references, and ask for the final clip. That’s it. Adding that one step lets you get a video that preserves the motion of your Three.js scene while also bringing the style much closer to what you’re actually looking for. I have not tested this across everything under the sun. From the initial tests I’ve done you can see them on this page , it’s pretty damn good. I think we’re just scratching the surface. With a little more innovation, this approach can get extended into a lot of other useful versions of the same idea. I don’t think this is a forever technique. Long term, video models will just get more controllable on their own. But while we’re waiting for that, this is an amazing approach, and it’s useful today . If you want the prompts written for your own scene, I built a small generator for that: