{"slug": "why-a-mario-jepa-predicts-well-and-plans-poorly", "title": "Why a Mario JEPA Predicts Well and Plans Poorly", "summary": "Researchers at LeWorldModel built LeMario, a Joint-Embedding Predictive Architecture (JEPA) world model trained on Super Mario Bros gameplay, which accurately predicts short-horizon dynamics and achieves near-perfect visual goal-seeking but fails to plan beyond the first major obstacle. The model's performance gap between short-term prediction and long-horizon planning highlights a fundamental limitation in reward-free world models for complex sequential tasks.", "body_md": "[AI](https://sourcefeed.dev/c/ai)Article\n\n# Why a Mario JEPA Predicts Well and Plans Poorly\n\nA from-scratch Super Mario world model clears short-horizon tests, then stalls on distant goals.\n\n[Mariana Souza](https://sourcefeed.dev/u/mariana_souza)\n\nWorld models that learn dynamics straight from pixels and actions, with no reward labels, keep looking like the clean path to agents that can plan. [LeWorldModel](https://arxiv.org/html/2603.19312v1) showed one compact Joint-Embedding Predictive Architecture (JEPA) doing reward-free planning on Push-T. LeMario rebuilds that stack from scratch and trains it on Super Mario Bros instead.\n\nThe model clears every test that usually counts as success: held-out prediction, action sensitivity, multi-step rollout quality, and short visual goal seeking. Then it fails the moment the goal sits past the first real obstacle. That gap is the useful story. Short-horizon dynamics and long-horizon progress are not the same skill, and a clean JEPA can master one while barely touching the other.\n\n## The architecture that actually ran\n\nEach training sample holds four Mario frames plus the button history between them. A vision encoder maps every frame to a 192-dimensional latent:\n\n```\nz_t = E_θ(x_t),   z_t ∈ R^192\n```\n\nFrames arrive as `[batch, 4, 3, 224, 224]`\n\n. Actions arrive as `[batch, 4, 5, 6]`\n\n(five emulator steps of Left/Right/Up/Down/A/B). An action encoder collapses each 5×6 sequence into another 192-vector.\n\nThose latents feed a causal predictor of six transformer blocks. Actions do not simply concatenate. They enter through Adaptive LayerNorm Zero (AdaLN-Zero): each action produces shift, scale, and gate values for both the attention and MLP branches. The gates start at zero so the network cannot inject random action noise early in training; it has to learn which controls matter. After the blocks, a small head emits three predicted future latents. Training minimizes MSE against the real next latents, plus a SIGReg term (weight 0.1) that keeps the representation from collapsing to a constant vector.\n\nThat is the whole machine: encoder, action-conditioned transformer predictor, prediction loss, anti-collapse regularizer. No decoder, no pixel reconstruction, no reward head.\n\n## The numbers that look like a win\n\nLeMario trained on 737,134 frames from 280 episodes spanning 32 Mario levels. Loss alone is a weak signal here. Nearby frames look similar, so “predict no change” is already a strong baseline.\n\nOn held-out episodes the comparison is clearer:\n\n| Method | One-step error | Five-step error |\n|---|---|---|\n| LeMario | 0.013773 | 0.077717 |\n| Predict no change | 0.014472 | 0.142473 |\n| Shuffled actions | 0.016555 | 0.114648 |\n\nShuffling the actions raised one-step error by 20.2%. Over five recursive steps LeMario beat pure persistence by 45.5%, while shuffled actions sat 47.5% worse. Buttons matter more the farther the rollout goes. That is real evidence the model learned short-horizon, action-conditioned Mario dynamics, not just image similarity.\n\n## Planning that works until it doesn’t\n\nOnce futures are imaginable, you can search them. LeMario’s reward-free planner could drive Mario toward nearby image goals and finish within two to five pixels of the targets. For a moment it looked like the model had learned to play.\n\nMove the goal farther into the level and the illusion ends. Mario cannot reliably clear the first major obstacle or navigate toward a single distant goal image. The world model predicts what the screen will look like if you press buttons. It does not, by itself, learn which sequences of screens constitute progress through a level.\n\nThat distinction is easy to miss when demos stay inside the comfort zone of short visual matching. Prediction quality and planning competence diverge as soon as the horizon includes a pipe, a gap, or a Goomba that actually has to be dealt with.\n\n## What this means if you build world models\n\nClassic Mario agents (Double DQN, older deep Q setups with replay and hand-shaped x-progress rewards) optimize an explicit return. They often need heavy engineering of the reward and action space, but they are forced to care about forward motion. JEPA-style world models flip the stack: learn a latent dynamics model first, plan later, keep rewards optional. LeMario is a concrete, end-to-end template for the second path on a game people already know how to instrument with [gym-super-mario-bros](https://github.com/Kautenja/gym-super-mario-bros) and related wrappers.\n\nIf you want to try the same recipe, the practical checklist looks like this:\n\n- Collect multi-level, multi-episode pixel + button trajectories (hundreds of thousands of frames, not a single stage).\n- Encode frames and short action windows into a shared latent size (here 192).\n- Condition the transformer predictor with AdaLN-Zero rather than naive concatenation so actions can modulate features without early noise.\n- Train multi-step latent prediction plus an anti-collapse regularizer; do not rely on reconstruction.\n- Evaluate with three checks that LeMario used: held-out one-step error, multi-step recursive error, and action-shuffle ablations. Then test planning on near goals\n*and*far goals that require obstacle handling.\n\nTrade-offs are real. You avoid reward hacking and label collection, which is attractive for simulation, robotics, or any environment where a dense reward is expensive or gameable. You pay with planning fragility: pure latent MPC or goal-image matching can look strong locally and still lack the inductive bias that “rightward progress past hazards” is what matters. Compared with pixel-reconstruction world models you also get a more abstract latent, which is good for planning efficiency and bad if you ever need to debug by looking at decoded frames.\n\nWho should care now? Researchers and tool builders who want a small, readable JEPA reference implementation on a familiar domain. Product teams shipping game AI or simulation agents should treat it as a diagnostic, not a drop-in controller: use it to learn dynamics, then layer hierarchical goals, learned value estimates, or hybrid reward signals before you trust long-horizon behavior. If longer-horizon JEPAs or better planners close the gap LeMario exposed, the same training recipe becomes a practical foundation rather than a demo. Until then, the honest read is “excellent short-horizon world model, incomplete agent.”\n\nThe postmortem is the product. Building the architecture from scratch, watching every conventional metric pass, then watching distant goals fail, is more informative than another polished success video. Prediction of Super Mario is tractable with a compact JEPA. Progress through Super Mario still demands something the pure dynamics model did not supply.\n\n## Sources & further reading\n\n-\n[LeMario: Training a JEPA World Model on Super Mario Bros](https://www.benjamin-bai.com/projects/lemario)— benjamin-bai.com -\n[hckr news - Hacker News sorted by time](https://hckrnews.com/)— hckrnews.com -\n[Train a Mario-playing RL Agent — PyTorch Tutorials 2.13.0+cu130 documentation](https://docs.pytorch.org/tutorials/intermediate/mario_rl_tutorial.html)— docs.pytorch.org -\n[GitHub - aleju/mario-ai: Playing Mario with Deep Reinforcement Learning · GitHub](https://github.com/aleju/mario-ai)— github.com\n\n[Mariana Souza](https://sourcefeed.dev/u/mariana_souza)· Senior Editor\n\nMariana covers the fast-moving world of machine learning and generative AI, with a particular focus on how these technologies are reshaping development workflows. When she isn't stress-testing the latest foundation models, she's usually at a local hackathon.\n\n## Discussion 0\n\nNo comments yet\n\nBe the first to weigh in.", "url": "https://wpnews.pro/news/why-a-mario-jepa-predicts-well-and-plans-poorly", "canonical_source": "https://sourcefeed.dev/a/why-a-mario-jepa-predicts-well-and-plans-poorly", "published_at": "2026-07-15 12:51:18+00:00", "updated_at": "2026-07-15 12:53:11.484418+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-research", "ai-agents"], "entities": ["LeWorldModel", "LeMario", "Super Mario Bros", "Joint-Embedding Predictive Architecture", "Mario JEPA"], "alternates": {"html": "https://wpnews.pro/news/why-a-mario-jepa-predicts-well-and-plans-poorly", "markdown": "https://wpnews.pro/news/why-a-mario-jepa-predicts-well-and-plans-poorly.md", "text": "https://wpnews.pro/news/why-a-mario-jepa-predicts-well-and-plans-poorly.txt", "jsonld": "https://wpnews.pro/news/why-a-mario-jepa-predicts-well-and-plans-poorly.jsonld"}}