If I give my daughter milk or water, I can almost guarantee she will spit it out. She’s 21 months old, and while she actually enjoys both drinks, halfway through she inevitably decides it’s time for a daily physics experiment. She wants to know: *What will happen if I open my mouth and just let the water fall out?*
While it annoys me, I remind myself that this is how humans learn about the world: by building a mental map through repetition and cause-and-effect. Over time, these experiences allow our brains to predict outcomes without needing to test them every single time. For example, I don’t need to jump off a multi-story building to know I’ll get hurt. I can infer the outcome from smaller falls or by observing what happens to other people and objects.
AI researcher Yann LeCun describes this kind of understanding as an "internal world model": a mental representation of how the world works that helps us predict what might happen before we take an action.
When most of us think of generative AI, we immediately associate it with large language models (LLMs) because that is what we have been using heavily for the past few years. LLMs learn to predict sequences of tokens, which can represent words, parts of words, punctuation, or pieces of code. This makes them great at automation tasks involving language, like coding and writing. However, LLMs have limits because the world is not made up of text.
Now, an older idea in AI is gaining new momentum: world models.
World models learn patterns in how an environment changes over time, including how it might change in response to an action. If you drop a glass, it falls. If you kick a ball, it rolls. If you open your mouth while it’s full of water, the water spills onto the floor. The goal is to learn enough about those patterns to predict what might happen next.
They can learn in different ways. Some systems learn through interaction or simulated environments. Others learn by observing large amounts of video and identifying patterns instead of performing every action.
Just like how world models learn from video, I know as an adult that falling from a high building would seriously hurt me. I do not need to personally test it. I have enough experience with gravity, my own clumsiness, and observing other people falling to predict the result.
Until recently, researchers primarily used these models for robotics and self-driving cars. An autonomous vehicle relies on a world model to evaluate real-time situations and prevent accidents: If my car changes lanes, brakes suddenly, or if a pedestrian steps into the street, what happens next? What is the safest action to take?
Recently, companies like Runway are leveraging world models beyond robotics and autonomous vehicles, applying them directly to video generation and user interfaces.
Runway introduced GWM Worlds 2 (in the video below), a model that turns video generation into a real-time interactive simulation. Instead of giving a model a prompt, waiting for a finished video, and then watching the result, you can continue steering the environment while the video is being generated. You define the world, including its environment, subjects, style, and even physical rules, and then use text or camera movement to influence what happens next.
For example, you can say, "Let's make it rain in this video," and the environment will adapt to show a rainier setting, changing how characters, buildings, plants, and other elements respond. Alternatively, a user might say, "Let's follow this character," prompting the model to generate a storyline and path for what was originally just a non-player character (NPC) in a video or game. Similarly, world models could make software interfaces dynamic. Runway recently introduced Solaris, an Interface World Model that generates an interface frame by frame as the user interacts with it. Instead of developers coding every screen and possible interaction ahead of time, clicks, drags, typing, and other actions help determine what the interface generates next.
For instance, in a traditional banking app with fixed tabs like Accounts, Transfer, Pay Bills, and Settings, a world model could allow you to ask: "Can you help me understand my spending this month and transfer $500 to my savings?" The interface would then dynamically generate relevant charts, explanations, and controls tailored specifically to that task, allowing you to skip navigating through a predetermined set of screens. But the financial tooling experience doesn’t really speak to me. So I tried to think of a situation that applied to my interests.
My best friend and I love browsing homes on Zillow that are outside our budget and imagining how we would furnish them. Today, that requires juggling multiple design apps to visualize a room or just using our imagination.
With a world model, we could prompt:
And instead of generating one static image of the redesigned room, the environment could keep responding. We could move the bed, walk around it, change the wall color, see how the room looks at a different time of day, and continue modifying the same space.
That could change the trajectory of real estate and interior design.
I think generative media is going to change a lot over the next few years, especially across tech, media, film, and gaming, because Runway is not the only company conducting these experiments.
fal recently released H3 Max Director. It keeps a video stream running while accepting new instructions about what should happen next. fal has even used it to power experimental livestreams where viewers vote on how a continuously generated show should change.
// Detect dark theme var iframe = document.getElementById('tweet-2094286082275696880-899'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2094286082275696880&theme=dark" } We’re accustomed to generating content and consuming the final result. But what happens when the thing we generate can keep changing based on our actions and requests?