Ilya Explains How LLMs Create a World Model

Ilya Sutskever, co-founder of OpenAI, explained at Nvidia's GTC in March 2023 that large language models learn a representation of the world, not just statistical correlations in text, because compressing text well requires capturing the underlying process that produced it. Daniel Miessler, a writer and AI commentator, cites this as the foundational authority for his own arguments that LLMs create world models, emphasizing the importance of the idea.

I've been arguing for many years that modern AI creates a world model. I thought I came to that conclusion on my own. I don't directly remember ever watching this clip. But now I wonder if I saw it at some point, and got the idea from Ilya without remembering where it came from. The clip is from his fireside chat with Jensen Huang at GTC in March 2023 https://blogs.nvidia.com/blog/sutskever-openai-gtc/ . Here's the core of what he says: It may look on the surface that we are just learning statistical correlations in text. But it turns out that to just learn the statistical correlations in text, to compress them really well, what the neural network learns is some representation of the process that produced the text. This text is actually a projection of the world. There is a world out there, and it has a projection on this text. And so what the neural network is learning is more and more aspects of the world, of people, of the human conditions, their hopes, dreams, and motivations, their interactions, and the situations that we are in. And the neural network learns a compressed, abstract, usable representation of that. Ilya Sutskever, GTC, March 2023 The idea is extremely intuitive to me, which is part of why I'm writing this down. I've learned to challenge and question intuition, especially my own. When something feels this obvious I want to know whose authority it actually rests on. In this case the answer is about as good as it gets. There are few people we should trust more than Ilya about ground truth related to LLMs. He was closer to the creation of these systems than almost anyone alive, and he was saying this plainly back in March of 2023. I've made the longer versions of this argument in Transformers Create Shapes of the Universe https://danielmiessler.com/blog/transformers-create-shapes-of-the-universe , World Model + Next Token Prediction = Answer Prediction https://danielmiessler.com/blog/world-model-next-token-prediction-answer-prediction , and Do Humans Really Have World Models? https://danielmiessler.com/blog/humans-dont-have-world-model , with supporting pieces in AI Predicts the Text of Answers https://danielmiessler.com/blog/ai-predicts-answers and The Chinese Room Problem https://danielmiessler.com/blog/problem-chinese-room-tokens . I've been making the short version on X as well, like asking what evidence we have that humans have world models https://x.com/DanielMiessler/status/2067391456709611846 , pointing out that LLMs are literally compressions of human knowledge https://x.com/DanielMiessler/status/2060219412507132086 , and proposing a novel murder mystery as a test of AI world models https://x.com/DanielMiessler/status/2061287859525685648 . This post is the anchor for all of it. That's the whole point. I think it's one of the most important ideas in AI, and it rests on much better authority than my intuition.