{"slug": "the-one-step-trap-in-ai-research", "title": "The One-Step Trap (In AI Research)", "summary": "AI researcher Rich Sutton warns against the 'one-step trap' in AI research, where relying on iterating one-step predictions for long-term outcomes leads to compounding errors and exponential computational complexity. He advocates for temporally abstract models using options and general value functions as a solution.", "body_md": "# The One-Step Trap (in AI Research)\n\n## Rich Sutton\n\n### Written up for X on July 18, 2024\n\nThe one-step trap is the common mistake of thinking that all or\nmost of an AI agent�s learned predictions can be one-step ones,\nwith all longer-term predictions generated as needed by iterating\nthe one-step predictions. The most important place where the trap\narises is when the one-step predictions constitute a model of the\nworld and of how it evolves over time. It is appealing to think\nthat one can learn just a one-step transition model and then �roll\nit out� to predict all the longer-term consequences of a way of\nbehaving. The one-step model is thought of as being analogous to\nphysics, or to a realistic simulator.\n\nThe appeal of this mistake is that it contains a grain of truth:\nif all one-step predictions can be made with perfect accuracy,\nthen they can be used to make all longer-term prediction with\nperfect accuracy. However, if the one-step predictions are not\nperfectly accurate, then all bets are off. In practice, iterating\none-step predictions usually produces poor results. The one-step\nerrors compound and accumulate into large errors in the long-term\npredictions. In addition, computing long-term predictions from\none-step ones is prohibitively computationally complex. In a\nstochastic world, or for a stochastic policy, the future is not a\nsingle trajectory, but a tree of possibilities, each of which must\nbe imagined and weighted by its probability. As a result, the\ncomputational complexity of computing a long-term prediction from\none-step predictions is exponential in the length of the\nprediction, and thus generally infeasible.\n\nThe bottom line is that one-step models of the world are hopeless,\nyet extremely appealing, and are widely used in POMDPs, Bayesian\nanalyses, control theory, and in compression theories of AI.\n\nThe solution, in my opinion, is to form temporally abstract models\nof the world using options and GVFs, as in the following\nreferences.\n\nSutton, R.S., Precup, D., Singh, S. (1999). Between MDPs and\nsemi-MDPs: A Framework for Temporal Abstraction in Reinforcement\nLearning. Artificial Intelligence 112:181-211.\n\nSutton, R. S., Modayil, J., Delp, M., Degris, T., Pilarski, P. M.,\nWhite, A., Precup, D. (2011). Horde: A scalable real-time\narchitecture for learning knowledge from unsupervised sensorimotor\ninteraction. In Proceedings of the Tenth International Conference\non Autonomous Agents and Multiagent Systems, Taipei, Taiwan.\n\nSutton, R. S., Machado, M. C., Holland, G. Z., Timbers, D. S. F.,\nTanner, B., & White, A. (2023). Reward-respecting subtasks for\nmodel-based reinforcement learning. Artificial Intelligence 324.", "url": "https://wpnews.pro/news/the-one-step-trap-in-ai-research", "canonical_source": "http://incompleteideas.net/IncIdeas/OneStepTrap.html", "published_at": "2026-07-12 18:41:15+00:00", "updated_at": "2026-07-12 19:06:06.472958+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-research"], "entities": ["Rich Sutton", "Horde", "GVFs"], "alternates": {"html": "https://wpnews.pro/news/the-one-step-trap-in-ai-research", "markdown": "https://wpnews.pro/news/the-one-step-trap-in-ai-research.md", "text": "https://wpnews.pro/news/the-one-step-trap-in-ai-research.txt", "jsonld": "https://wpnews.pro/news/the-one-step-trap-in-ai-research.jsonld"}}