Goosebumps: A Paradigm Shift Is Occurring in Robotics

UC Berkeley researchers and the NVIDIA robotics team have developed Agentic Robotics (AR), an approach that uses multi-agent AI systems to write, test, and iteratively improve modular robot programs largely offline rather than relying on physical demonstration data. In May the team built Graph-as-Policy (GaP), a graph-based AR harness that produces lightweight ROS2-compatible executable code and, in benchmarks, significantly outperformed previous methods on success rates while continuing to improve success rate and cycle time through self-learning. The work identifies Inverse Physics—building accurate physics models for a real robot, task, and environment—as a major remaining bottleneck, distinct from Real2Sim and System Identification.

Over the past nine months, results from our group and others have changed my view of how rapidly robotics will progress. A new approach— Agentic Robotics AR —is a game changer. The shift is not simply from one robot policy architecture to another. It is a shift in where robot intelligence is developed . Instead of relying primarily on large datasets of physical robot demonstrations or on engineers manually programming and tuning each new application, Agentic Robotics uses multi-agent AI systems to write, test, diagnose, and iteratively improve structured robot programs, largely offline. In January, motivated by rapid advances in multi-agent coding, I started working with my students at UC Berkeley specifically Letian Fu, Justin Yu, Kaiyuan Chen, and Shuangyu Xie and with the NVIDIA robotics team Spencer Huang, Jim Fan, Yuke Zhu, and others on AR. AR does not rely on demonstration data 1 . For a given robot task description, AR uses the latest LLM and VLM systems to compose a library of clearly defined, modular robot "skills" which can include VLA policies into interpretable robot control systems 2 . The key idea is to treat robot policy generation as a coding problem . A manipulation system might contain separate modules for perception, grasp selection, motion planning, force control, and failure recovery. Coding agents can generate and modify these modules, test them, and improve both the individual skills and the code for a particular task. This introduces a third path between two dominant cultures in robotics. Conventional model-based engineering methods can be fast, interpretable, and highly reliable, but generally require substantial human engineering for each new task. Contemporary model-free and Vision-Language-Action approaches have the potential to adapt more broadly, but require enormous amounts of robot data and are not yet sufficiently reliable for many industrial applications. Agentic Robotics has the potential to combine important properties of both: the power and generality of frontier LLM and VLM models with the speed, structure, and reliability of executable robot programs. Graph-as-Policy GaP In May, we developed GaP, a graph-based AR harness to manage agentic context complexity and output lightweight executable ROS2-compatible code 3 . Rather than asking one large language model to generate all the code for a task, GaP represents control with a graph of modular components. Multiple coding agents can work on different nodes, evaluate their performance in simulation, revise them, and recursively improve the resulting system. GaP operates offline, producing lightweight executable code that is then exported for execution and further fine-tuning on real industrial robots. We found that GaP could significantly outperform previous methods on benchmarks in terms of success rates and could continue to improve both success rate and cycle time using self-learning. GaP suggests that coding agents can do more than generate an initial robot policy. They could participate in an iterative engineering loop: propose an implementation, evaluate it, diagnose failures, revise the relevant components, and try again. But there was a major bottleneck. Inverse Physics Self-learning directly on physical robots is possible, but it is slow. Experiments must run sequentially, failures can require manual resets, and exploration can damage hardware or the environment. Accurate simulation can be extremely valuable for robot learning. Thousands of experiments can potentially run in parallel and much faster than real time. But simulation is never perfect and it has another problem: someone has to build an accurate simulation of the physical world. Inverse Physics is needed to build an accurate physics model for a real robot, task, and evironment. It is related to but distinct from Real2Sim and System Identification. This has always required substantial engineering effort. Geometry, lighting, contact parameters, friction, compliance, mass, damping, and other properties must be specified or estimated well enough for the simulation to predict behavior that matters in the real world. In June, we began working with the open-source Newton physics simulator and with NVIDIA physics simulation experts including Erwin Coumans. Newton provides a range of physics models suitable for robot simulation. My postdoc Kaiyuan Chen started developing Robot Sim Studio RSS , an interactive front end to Newton. RSS exposes simulation parameters through a Viser-based interface with sliders and fast rendering, making it easier for a human to adjust the simulation and compare its behavior with physical observations. We viewed RSS primarily as a better tool for human engineers. Then something unexpected happened. Real2Sim2Real On 3 September, GPT-6 Astra was released. Given only a single photo of a real scene, Astra was able to rapidly create a Blender model to reproduce the photo with appropriate lighting and camera position inverse graphics . Over that weekend, several researchers also reported surprisingly strong results using Astra for robot-control tasks 4 . We wondered whether the same capabilities might extend beyond inverse graphics to inverse physics: could Astra infer the physical parameters for an accurate simulation from observations of the real world? So we gave Astra access to our new RSS tool and a short video of a real robot in our lab performing a deformable manipulation task – wiping ink off a metal bar with a sponge. In under an hour, Astra learned how to use RSS to create a Newton physics model that accurately reproduced the video including very realistic deformations of the sponge. Astra was good at inverse graphics, but we didn’t expect that it would be so good at inverse physics. We then gave Astra this physics simulation and the GaP harness. It generated a robot control graph and recursively improved robot success rate and throughput. Then we ran the resulting code on our real robot and it worked. Goosebumps This is when I got goosebumps. AR is the missing link between model-based and model-free methods; it opens a path to fast and reliable robots. But not every robot trial succeeded. In one trial, the robot pushed over the metal bar. This failure was particularly informative: the mass of the bar could not be reliably inferred from the visual evidence we had provided. But this exposes one of the most important properties of the Agentic Robotics approach. A real-world failure does not have to be the end of the engineering process. It becomes new evidence. The next step is to provide the agent with a video of that failure. The agent can update the simulation or increase the policy’s robustness to uncertainty, test revised strategies in simulation, regenerate the relevant portions of the control graph, and deploy the new code. This suggests that inverse physics could substantially reduce one of the central bottlenecks in simulation-based robot learning. Real2Sim may increasingly shift from a labor-intensive human engineering task toward an agentic learning process. AI agents can be slow. That's not a problem when the work happens offline. The robot does not need to run a frontier model in its inner control loop. Instead, the agent works offline reasoning, coding, simulating, testing, and improving—and then export lightweight code that executes rapidly on the physical system. AR creates a path toward robot systems that combine the adaptability of frontier AI with the speed and reliability of conventional control . I had discussed GaP with Jeff Mahler, my co-founder at Ambi Robotics. On Wed 17 September, Jeff announced that he had applied the GaP AR harness to solve a real industrial problem and deployed the result to robots sorting packages across the US 5 . Agentic Robotics can address what matters in industrial systems: cycle time, reliability, variability, and continuous operation. Much remains to be done. Physics models are still imperfect. We need more efficient methods for active and curriculum-based self-learning and better methods for tuning robustness and speed in both simulation and physical systems. Real-world perception and control remain uncertain, and safety-critical applications will require stronger verification and validation. Agentic Robotics is also currently slow at the level of the coding agent. That is acceptable for offline policy development, but the approach is not yet suited to every form of real-time adaptation. It is unclear whether this architecture will scale to open-ended generalist robots or humanoids 6 . The conclusion is not that generalist robotics is solved. What has changed is something more specific. For decades, one of the central constraints in robotics has been the enormous amount of human engineering required to convert a task specification into a reliable physical system. More recently, robot learning has attempted to replace some of that engineering with enormous quantities of physical data. Agentic Robotics suggests another possibility: that robotics itself can increasingly be automated. AI agents can write modular robot programs. They can test those programs in simulation. They can diagnose failures and modify selected components. Increasingly, they may also be able to construct and tune the simulations in which that learning occurs. And the resulting systems can be exported as lightweight, interpretable programs for physical robots. If these capabilities continue to improve, the bottleneck in robotics may shift from collecting enough robot data or having enough engineers hand-tune each application to: specifying objectives, constraints, interfaces, and verification criteria for increasingly capable robot-engineering agents. That would be a paradigm shift. I've always been cautious and skeptical of near-term adoption of robots. Now I'm reconsidering. The timeline for real robots has just advanced considerably. References: Plenary presentation to researchers at ICRA26. 2 June 2026 Vienna : “A Tale of Two Cultures: Can Agentic Coding Close the Gap?”. YouTube 45 mins : https://bit.ly/Agentic-Robotics-plenary-by-Ken-Goldberg https://bit.ly/Agentic-Robotics-plenary-by-Ken-Goldberg I Gave My OpenClaw Agent a Physical Body. https://www.wired.com/story/i-gave-my-openclaw-agent-physical-body-robot/ Will Knight. WiRed. 20 May 2026: “...AI-powered coding is super exciting because it has the potential to bridge the gap between conventional engineering methods, which are reliable but don't generalize, and contemporary vision-language-action models, which generalize but are not yet reliable,” says Ken Goldberg, a roboticist at UC Berkeley who is exploring the approach. World Models vs VLAs: The Rift Dividing Physical AI. https://www.theinformation.com/newsletters/ai-agenda/world-models-vs-vlas-rift-dividing-physical-ai?rc=io0e0e The Information Daily. by Rocket Drew, 25 June 2026: "...Goldberg and his collaborators recently extended their work in a new technique called 'Graph as Policy' that will be published in a forthcoming paper. In the new approach, multiple coding agents are assigned to tweak distinct 'nodes' for controlling a robot." Combining Cultures, from Code to Canvas: Interview. https://aihub.org/2026/09/01/combining-cultures-from-code-to-canvas-an-interview-with-ken-goldberg/ AI Hub. by Ella Scallan. 1 Sept 2026.: "I’ve been thinking about the gap between two communities within robotics: the traditional roboticists who have been working for 50-plus years on control methods, model-based methods, and the new wave – the younger generation that has embraced learning and, in many cases, rejects the older methods...." Footnotes 1 The 100,000 Year Robot Data Gap. https://bit.ly/Robot-Data-Gap Science Robotics Editorial. August 2025. 2 CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation. https://capgym.github.io/ Max Fu , Justin Yu , Karim El-Refai , Ethan Kou , Haoru Xue , Huang Huang, Wenli Xiao, Guanzhi Wang, Li Fei-Fei, Guanya Shi, Jiajun Wu, Shankar Sastry, Yuke Zhu, Ken Goldberg, Linxi “Jim” Fan, Nvidia, Berkeley, Stanford, CMU, UT Austin . International Conference on Machine Learning ICML . Seoul Korea. 9-12 July. 2026. 3 GaP: A Graph-as-Policy Multi-Agent Self-Learning Harness For Variational Automation VA Tasks. https://graph-robots.github.io/gap/ Kaiyuan Chen 1,∗ Shuangyu Xie 1,∗ Letian Fu 1 Justin Yu 1 William Pacini 1, Sandeep Bajamahal 1, Hudson Kim 1, Jaimyn Drake 1, Daehwa Kim 3, Haoru Xue 1, Jonathan Francis 3, Christian Juette 3, Peter Schaldenbrand 3, Muhammet Seker 3 ,Ruwan Wickramarachchi 3, Uksang Yoo 1,3, Guanzhi Wang 2, Adithyavairavan Murali 2, Balakumar Sundaralingam 2, Shankar Sastry 1, Spencer Huang 2, Yuke Zhu 2, Linxi “Jim” Fan 2, Ken Goldberg 1. 3 Bosch, 2 NVIDIA, 1 UC Berkeley. To appear: Conference on Robot Learning CoRL , Austin TX, 9-11 November, 2026. 4 “GPT-6 Astra scored 95% on a robot control task https://x.com/chooi jeq/status/2096064315115839904?s=20https://x.com/chooi jeq/status/2096064315115839904?s=20 , up from Fable 5.1's 40%, with 6.2x fewer output tokens at 2.3x lower cost.” Jay Chooi on X: 4 September 2026. 5 Agentic Robotics Solves an Industrial Robotics Problem. https://www.ambirobotics.com/blog/agentic-robotics/ 17 September 2026. 6 GPT-6 Astra as an Embodied Robot Policy. https://anonymous-report-421.github.io/public-website/?view=1 11 September 2025.