LeMario Postmortem: A JEPA World Model Learned to Predict Super Mario Bros — But Couldn't Actually Play It

Benjamin Bai built a JEPA world model from scratch and trained it on 737,000 frames of Super Mario Bros. The model predicted five-step futures better than strong baselines — but when handed the controller, Mario couldn't jump over the first obstacle. A technical postmortem on why learning to predict a world isn't the same as learning to act in it.

Benjamin Bai spent weeks building a JEPA (Joint Embedding Predictive Architecture) world model from scratch — Yann LeCun's proposed architecture for learning world dynamics from pixels and actions — and trained it on Super Mario Bros. The results are a textbook case of why predicting the world and acting in it are two very different problems.

The model, called LeMario, trained on 737,134 frames from 280 episodes across 32 Mario levels. On held-out episodes, it beat the persistence baseline by 45.5% on five-step prediction, and shuffling the controller inputs made predictions 47.5% worse — clear evidence the model had learned action-conditioned dynamics. The architecture used a vision encoder compressing frames into 192-number latent representations, an action encoder for button sequences, and six transformer blocks with Adaptive LayerNorm Zero to inject actions into the predictor.

Then Bai handed the model the controller. Using the Cross-Entropy Method (CEM) to search through imagined futures and find action sequences connecting start and goal frames, the results were sobering. For a tiny goal just 32 pixels away, Mario moved from position 40 to position 44. He barely budged.

What followed was a systematic debugging odyssey that exposed three core problems. First, Bai trained a small probe on the frozen encoder and found Mario's horizontal position was almost perfectly recoverable (R-squared of 0.997) while vertical position was much weaker (R-squared of 0.188). The encoder had learned useful information — it just wasn't the right kind for control. Second, CEM did exactly what it was asked: it found actions the model believed would reach the goal, amplifying every weakness in the model's representation. Visually similar locations at different points in a scrolling level fooled both the encoder and the planner. Third, Mario broke the assumptions that made the architecture work on Push-T, the simpler robotics task from the original LeWorldModel paper. A scrolling camera replaced a fixed one. Momentum, jumping, pits, enemies, and death replaced smooth linear movement. The model had been tested on 280 episodes across 32 levels for one epoch, while Push-T used 20,000 expert episodes for ten.

The postmortem is unusually honest about failure. Bai ends with concrete takeaways: validate central assumptions earlier, establish simple baselines before trusting any metric, and design evaluations around the behaviors you care about rather than optimizing for training loss alone. The full writeup, including architectural diagrams, training curves, and video rollouts of Mario's failed attempts, is available on Bai's project page.

Sources: Benjamin Bai — LeMario Project Page, LeMario GitHub Repository, LeWorldModel Paper (arXiv), Hacker News Discussion