AI Agents That Live Inside a Dreamed-Up World

Cisco AI engineer Shridhar Shah built a tiny multi-agent system where an agent learns to hallucinate future frames of a predator-prey game and then practices inside its own dream. Shah found that a dream model only stays accurate if it captures how players react to each other: a single-player dream fell apart after one frame, while a multiplayer dream remained 100% accurate 15 frames ahead. The work builds on the World Models approach from Ha & Schmidhuber (2018) and shows that for multi-agent dreams, modeling reactions is essential to avoid drift.

An agent watches a game, learns to hallucinate the next frame, then plays inside its own dream — but only the model that knows players react to each other stays true. TL;DR: The hottest idea in agents right now: don't feed them the real world — let them dream it. An agent watches some footage, learns to hallucinate the next frame, and practices inside its own head. I built a tiny one, and found the catch: a dream only stays true if it knows the players react to each other. Runs on a laptop. Two players in an 11-cell corridor: a predator steps toward the prey, the prey steps away. Every move is a reaction . An agent watches random games, then closes its eyes and dreams 15 frames ahead , feeding each prediction back in as the next input. I built two dreamers from the exact same footage: Training an agent inside its own learned dream goes back to Ha & Schmidhuber's World Models https://arxiv.org/abs/1803.10122 2018 ; the open frontier is making that dream % of the dream still matching reality, this many frames ahead: | frames ahead | 1 | 3 | 5 | 10 | 15 | |---|---|---|---|---|---| | single-player dream | 11 | 0 | 0 | 9 | 0 | multiplayer dream | 100 | 100 | 100 | 100 | 100 | The single-player dream falls apart almost immediately. The multiplayer dream stays locked to reality the whole way. real = dream = start for in range 15 : real = real next real what actually happens dream = model dream feed the dream its OWN last frame match += dream == real Same footage, same loop. The only difference: whether the model predicts the two players jointly or independently. The prey moves because the predator moved. A model that looks at the prey alone can't see that — so its tiny errors compound each frame until the dream is pure fiction. The multiplayer model conditions on both, so it captures the reaction and keeps re-predicting the real game. A dream you can act in is a superpower — an agent can practice a thousand risky moves for free. But a dream that forgets everyone else reacts to you isn't practice. It's a delusion. Dreamed worlds let agents rehearse infinitely, safely, at zero real-world cost. The lesson from this toy: for multi-agent dreams, model the reactions or the whole thing drifts. Get it right and agents can plan against each other entirely in imagination. git clone https://github.com/Shridhar-2205/secret-lives-of-agents cd secret-lives-of-agents/03-dreamed-world && python demo.py Shridhar Shah — Senior Software Engineer on the AI team at Cisco. GitHub https://github.com/Shridhar-2205 · LinkedIn https://www.linkedin.com/in/shridhar-shah-220b1721b/ Sources & further reading:Ha & Schmidhuber, 2018 — the "train inside a dream" idea · Hafner et al., World Models DreamerV3, 2023 · Bruce et al., Mastering Diverse Domains through World Models 2024 . Genie: Generative Interactive Environments