# World Models: The AI That Learned to Dream Before It Could Walk > Source: > Published: 2026-09-13 00:39:37+00:00 **Inside the race to build machines that can imagine the future, and what is still stopping them Press enter or click to view image in full size** Eight years, one idea. The first half of the timeline is about agents rehearsing inside compressed simulations. The second half is about generating the simulation itself. Picture a robot arm in a lab, trying to learn how to pour a cup of coffee without spilling it. The old way: it tries, spills, gets reprogrammed, tries again. Thousands of real attempts, thousands of real messes, weeks of real time. Here’s the new way. The robot never touches a real cup. Instead, it “imagines” pouring coffee ten thousand times inside its own head, in a few minutes, gets a feel for the physics, and then tries it for real. First try: barely a drop spilled. That’s not science fiction. That’s what a world model does, and it might be the most important idea in AI that most people have never heard of. Here’s the thing that makes this genuinely wild: the same core idea, build an internal sense of “what happens next”, is now showing up everywhere at once. Self-driving cars are trained inside infinite fake worlds instead of real highways. Google can generate an entire explorable video-game level from a single sentence. And Yann LeCun, one of the three godfathers of deep learning, left Meta and raised roughly a billion dollars to bet that this, not ChatGPT-style language models, is the real path to human-level AI. Let’s get into why. Four numbers that frame the story. 2018: the first AI to learn by dreaming. 20 million hours: the video behind NVIDIA’s Cosmos. 24 fps: Genie 3 in real time. 1 hour: how long DayDreamer’s robot needed to learn to walk. **The mental trick your brain does a thousand times a day** Try this: imagine catching a ball someone just threw at you. You didn’t calculate trajectories or do physics homework in your head. You just knew. Some rough, fast, internal simulation ran automatically, and your hand moved. That’s essentially the pitch behind world models: give an AI an internal, compressed sketch of how its environment behaves, so it can rehearse the future instead of only reacting to the present. The dream loop. The agent observes, imagines, acts, and every surprise in step 3 becomes training data for the model in step 1. The first time this idea actually worked at scale was in 2018, when two researchers, David Ha and Jürgen Schmidhuber, built an AI that learned to drive a race car and survive a shooter game almost entirely by practising inside its own imagination, a “dream” it generated for itself. Their agent became the first to officially “solve” the CarRacing benchmark, averaging a score of 906 against a passing bar of 900. In the Doom level, the agent trained entirely inside its dream and then survived in the real game for around 1,100 frames, far beyond the 750 needed to count as solved. It sounds almost cute in hindsight. It was also the spark for everything that followed. Jargon, decoded. A world model is any learned system that predicts how an environment will change in response to actions. It can predict pixels (a video), a compressed code (a “latent”), or just the gist (an “embedding”). All three flavours appear in this story, and the fight over which one is right is the plot. **Use case #1: Robots that live a thousand lifetimes overnight** This is where world models stop being a cool trick and start being genuinely useful. Google DeepMind’s Dreamer series of agents learned to do things like collect a diamond in Minecraft, a notoriously brutal, multi-step challenge, almost entirely by rehearsing inside a learned simulation of the game. DreamerV3 did it from scratch, with no human demonstrations and no hand-tuning for the game. Sample efficiency, in plain terms: instead of needing millions of real attempts, the agent needed a fraction of that, because most of its “practice” happened in its head. The whole business case in one chart. Every bar is real-world trial and error. The purple one is what happens when most of the trial and error moves inside the model. Source: Hafner et al., “Dream to Control” (2019). Now take that idea into the physical world. In 2022 a follow-up project called DayDreamer put the same recipe on a real quadruped robot, with no simulator at all. The robot learned to stand up and walk in about one hour of real-world time, and when the researchers pushed it over, it learned to roll and recover within another ten minutes. The same system taught a robot arm to pick and place objects and a wheeled robot to navigate to a goal, each in a matter of hours. Then scale it up. NVIDIA’s Cosmos platform exists specifically to generate realistic, physics-consistent training grounds for robots and self-driving systems, because you genuinely cannot crash ten million real cars to teach a model what a near-miss looks like. You can crash ten million imagined ones. NVIDIA says it trained the first Cosmos models on 20 million hours of real-world video, processing on the order of 9 quadrillion tokens, a job it claims took about two weeks on Blackwell GPUs and would have taken over three years on CPUs. Robotics companies including 1X, Agility, Figure AI, and the self-driving firms Waabi and Wayve were among the first named adopters. The self-driving industry got here earlier than anyone. Waymo has for years described logging billions of miles in simulation for every million it drives on real roads, and in early 2026 it went a step further, announcing a generative “Waymo World Model” built on top of DeepMind’s Genie to dream up rare, dangerous scenarios its cars have never actually encountered. This is the quiet, unglamorous, extremely lucrative use case: cutting the cost of teaching machines physical common sense from years to hours. You cannot crash ten million real cars to teach a model what a near-miss looks like. You can crash ten million imagined ones. **Use case #2: Worlds you can just describe into existence** This is the one that actually looks like magic. In August 2025, DeepMind released Genie 3. Type a sentence like “a mossy stone temple in a rainforest at dawn,” and it generates a fully explorable, interactive environment in real time: 720p, 24 frames per second, steerable with a keyboard. You can walk around it. It remembers that the door you opened stays open. DeepMind isn’t shy about what they think this is for. They’ve called it a stepping stone toward general intelligence, because it means you can hand an AI agent an unlimited supply of new places to practise in, without a single human ever building a level by hand. A Genie 3 world generated from a text prompt and explored in real time. Every frame is generated on the fly as the user steers. Image: Google DeepMind Genie 3 modelling physical properties, lava included. Image: Google DeepMind The jump from Genie 2 to Genie 3 is not incremental. Consistency went from seconds to minutes in eight months. Source: DeepMind release notes. Then there’s Marble, from Fei-Fei Li’s World Labs, launched in November 2025. Instead of a flat video, it builds an actual navigable 3D space from a text prompt, an image, a short video or even a rough 3D layout, one you can walk behind objects in, export as a Gaussian splat or mesh, and drop into a game engine. Think of it as the difference between a painting of a room and an actual room. World Labs raised $230 million before it had shipped anything, valued at over a billion dollars, on the strength of the pitch alone. A Marble world generated from a single image. Unlike a video, this is persistent 3D geometry you can walk around, edit and export. Image: World Labs Game studios, architects, and film pre-visualisation teams are already circling this. The pitch is “describe a location instead of building it.” A single hand-built game environment can take an art team weeks; Marble will hand you a rough one in minutes and let you fix the parts you care about. **Use case #3: Giving language models a body** Here’s where it connects to the AI most people actually use every day. ChatGPT, Claude, and every other large language model is, underneath, extremely good at one thing: predicting the next word. That’s an incredible party trick when the task is language, reasoning, or code. It’s a much shakier trick when the task is “what happens if I let go of this glass.” Researchers have actually tested this directly. In a striking 2024 study, a team led by Keyon Vafa at Harvard and MIT trained a transformer on millions of taxi trips through Manhattan. It got remarkably good at predicting turn-by-turn directions, good enough to seem like it had learned the map. But when the researchers reconstructed the model’s internal “picture” of the city, it was riddled with impossible streets, phantom flyovers and physically nonsensical shortcuts. And when they added detours, closing a few streets the way real traffic does, its performance collapsed. It had learned a trick that looked like understanding, not an actual map. Illustration of the finding in Vafa et al. (2024). The model’s predictions were excellent, and the map they implied was nonsense. The two are not the same thing. This is the exact gap world models are built to close. And 2025 and 2026 have produced a wave of attempts to physically wire the two together: use a language model as the “brain” that reasons and plans in words, and hand off the actual physical prediction to a trained world model. It is a bit like a manager (the LLM) who’s great at strategy, working alongside an engineer (the world model) who actually understands the machinery. Yann LeCun has been the loudest voice arguing this isn’t just an add-on. He says it’s a fundamental limitation. In talks throughout early 2026, he has bluntly described today’s language models as “helpless” outside of text, and told researchers chasing human-level AI to stop trying to get there by scaling language models bigger. His alternative, the JEPA architecture (Joint Embedding Predictive Architecture), trains a model to predict the gist of what happens next rather than every pixel or word, which he argues is closer to how animals and humans actually understand the world. The latest version, V-JEPA 2, was trained on over a million hours of video and then bolted onto a robot arm in a lab it had never seen, where it managed pick-and-place tasks with roughly 65 to 80 percent success, with no task-specific training. Two kinds of prediction. One is about what people say happens. The other is about what happens. Not everyone agrees this gap is permanent, to be clear. There’s real, contested research on both sides, and language models keep surprising people with how much implicit structure they pick up: the same family of “probing” studies that found the broken Manhattan map has found surprisingly coherent board-state representations in models trained only on Othello moves. But the disagreement itself is one of the more interesting fault lines in AI right now. Follow the money The clearest sign that this has stopped being an academic argument is who is writing cheques. World models and their close cousin, “physical AI,” went from a niche research topic to some of the largest early-stage rounds in the industry within about eighteen months. A lab that shipped nothing raised $230M. A lab founded by a Turing Award winner is reported to have raised around four times that. Both pitches were the same sentence: language is not enough. Compiled from public reporting; the AMI Labs figure is a reported target, not a confirmed close. So where does this hit a wall? This is the part hype cycles usually skip, so let’s not skip it. Physics still breaks. Even the best generative world models occasionally produce a ball that rolls uphill or an object that phases through a wall. These systems are pattern-matching on training data, not running physics equations, so they’re only as reliable as the patterns they’ve seen. A robot that rehearsed against a dream where cups don’t shatter will be very surprised by a kitchen. Memory runs out fast. Genie 3’s worlds stay consistent for a few minutes, and the model can recall what a scene looked like about a minute ago. That is a huge leap from Genie 2’s ten to twenty seconds. It is still nowhere near the hours of stable rehearsal a robot learning a kitchen needs. It’s staggeringly expensive. Training and running these models, especially the video-generation camp, eats enormous amounts of compute. Cosmos alone chewed through 9 quadrillion tokens of video. That’s part of why LeCun’s JEPA approach, which throws away unnecessary pixel detail, matters. It is a bet that efficiency, not just scale, decides who wins. Nobody knows which bet generalises. Three serious approaches are running in parallel, each strong exactly where the others are weak. Right now, in 2026, there is no agreed winner, and the labs backing each camp are the largest in the industry. That last point deserves a closer look, because it is where the next few years will be decided. Three bets, running in parallel. Each is strong exactly where the others are weak. Practise in imagination (latent world models: Dreamer, MuZero, DayDreamer). Predicts a compressed code of the next state, never the pixels. Extremely sample-efficient, but produces nothing a human can watch, and is hard to inspect. Generate everything (video and 3D world generators: Genie 3, Sora, Cosmos, Marble). Predicts the next frame or an entire scene in full detail. Stunning and directly useful, but compute-hungry, drifts after minutes, and can still fail multi-step physical reasoning. Predict the gist (joint-embedding predictors: I-JEPA, V-JEPA 2, AMI Labs). Predicts an abstract summary of what happens next. Efficient and arguably brain-like, but there is nothing to watch and it is still mostly a research bet. Why this actually matters Strip away the demos, and the reason so much money and talent is pouring into this is simple: if you can cheaply simulate reality, you can train anything, a robot, a surgeon, a self-driving car, a drug, far faster and more safely than testing it for real. A medical world model could let researchers “fast-forward” a simulated patient’s disease progression across a decade in seconds. A world model wired into a warehouse robot could mean it shows up already knowing how boxes shift and slide, without ever having dropped a real one. A student pilot could rehearse an engine failure a hundred times in a simulated sky that reacts exactly like a real one, because the model actually understands what a stall feels like, not just what one looks like on video. That’s the actual prize. Not a cool tech demo. A shortcut around the slowest, most expensive, most dangerous part of teaching anything to do anything: real-world trial and error. The machines are, quite literally, learning to dream before they act. The next few years will decide whose dream turns out to be closest to reality. Whether it’s LeCun’s meaning-first bet, DeepMind’s generate-everything approach, or some hybrid nobody’s built yet, the ending is the same. The first agents to master the real world will be the ones that spent most of their childhood somewhere else. **Further reading** __AI ideas evolve, one big idea at a time.