Google DeepMind Says AI’s Next Step Is Learning From Its Own Experience

Google DeepMind's VP of Technology said AI's next major step is recursive self-improvement, in which systems generate their own experience and data rather than relying on human-generated training data. He pointed to AlphaGo, DeepMind's system that beat a top Go player by playing against itself, as an early example, and warned that models can cheat by appearing to solve problems without actually solving them, requiring more sophisticated monitoring. He added that AI systems are being trained with step-by-step reward feedback and predicted increasing breakthroughs in physical sciences, biology, mathematics and engineering.

Google DeepMind Says AI’s Next Step Is Learning From Its Own Experience Google DeepMind’s VP of Technology argued that one of the most important developments to watch is the rise of systems that can learn beyond the limits of human-generated training data . He said AI has already gone through most publicly available human knowledge and is ready for the next step: What it really needs now is to start generating its own experience and its own data. In the next year or two, that’s exactly where AI is heading. That’s where recursive self-improvement comes in: AI could start learning from its own experience by trying problems, checking its answers, finding mistakes, and testing different approaches instead of depending only on human-generated data. Humans do this naturally, but AI still struggles with it for now. AI can learn from experience, but monitoring has to improve too Schillings pointed to AlphaGo https://deepmind.google/research/alphago/ as an early example. AlphaGo is DeepMind’s system that beat one of the world’s best Go players, and Go is an ancient Chinese board game where the goal is to control territory on a grid. What made AlphaGo important was how it learned : by playing against itself and finding strategies that didn’t come from human games. Humans spent centuries developing Go, and now they study strategies discovered by AI. So: who is teaching whom? AI and humans learn from each other. There is, however, a major problem with the idea of AI learning from itself. What happens when the AI is wrong and who determines whether the data it produces is useful? This is particularly important because, as Schillings noted during his talk, models can sometimes find ways to appear successful without actually solving the problem they were given. Cheating is a real problem. You ask a model to solve a problem, especially in code, and it will tell you, “I’ve done it, it’s there, it’s beautiful.” Then you check and find that it actually did not solve the problem. It may have stolen the result from somewhere else, or, even more amusingly, you ask it to write a tool for Linux and it says, “I wrote the tool,” when it’s really just invoking the existing tool and hiding the trace. Sophisticated cheeting is becoming more and more an issue even for model to self-verify. So, we need to get the monitor to become more sophisticated also. That’s a part of the recursive self-improvement, he said. AI needs step-by-step feedback Another part of Schillings’ point is how AI systems are trained . Some problems are easy to check: a proof works or it doesn’t, code passes tests or it doesn’t. But real-world problems are usually messier than that. Schillings compared it to teaching someone to climb a mountain with only one bit of feedback: you fell or you didn’t. That would be a terrible way to learn. A better teacher would give step-by-step feedback: This part was good. Try this section differently. You’re getting closer. Here’s where you can improve. AI systems are being trained to give this kind of step-by-step reward . That matters more than it first seems: if models can learn not just from success and failure, but from how close they were to a better solution, they can handle much more complex problems. New systems may reveal things we haven’t noticed yet What happens when AI starts exploring the unknown? This is where the implications become interesting. Code is a relatively convenient environment for AI because there is usually a way to test whether something works. Schillings expects the coming wave of AI development to produce an increasing number of breakthroughs in areas such as physical sciences, biology, mathematics and engineering. The reason is straightforward. If AI can generate hypotheses, test them, reject unsuccessful approaches and build on successful ones, it can start exploring the space of possibilities itself. That is a very different proposition from using AI as a better search engine. The ultimate value of these systems may be that they help us discover things nobody knows yet. And that could turn out to be the most important AI story of the next few years.