Agents have been the hottest buzzword in AI this year. New tools from UC Berkeley and Nvidia may bring them to the physical world.
On Monday, the university announced new open-source research and benchmarks for "agentic robotics," an approach that brings agents into designing, testing and refining robotics systems for industrial tasks. UC Berkeley researchers designed this with Nvidia Isaac, the company's robotics simulation and synthetic data generation platform, allowing builders to develop robotics with less data.
The release includes two main components:
- Graph-as-Policy , or GaP, is a framework that lets agents generate, connect and test skills for robots to complete tasks. For instance, the system will segment skills such as finding an object, planning a grasp and moving the gripper without collisions into individual parts. The agent then connects these skills and uses Nvidia Isaac to simulate and rehearse them.
- VABAR, or Variational Automation Benchmark for Agentic Robotics, is an evaluation designed to measure a robot's capabilities at six types of industrial tasks under different conditions. This encompasses a range of tasks, including things like tool sorting and handing, syringe packing, and cable insertion and routing. Within this, a robot can simulate "contact-rich robot-object interactions" using Nvidia's simulation technology, which is then judged on Nvidia's Isaac Lab-Arena, the open-source policy evaluation framework.
Ken Goldberg, professor and distinguished chair in engineering at UC Berkeley, told The Deep View that the idea of agentic robotics is a fundamental shift from what many in robotics are doing, relying on a language model to write code for robots on the fly, rather than the current common paradigm of using data to improve vision-language-action models or world models. While this approach can complement those methods, it also provides its own alternative, he said.
This method could allow robots to actually scale in production by solving the challenge of reliability, as "data-driven" approaches, which require tons of real-world data, have hit a "plateau at 80% reliability," Goldberg said. "That's good enough for writing research papers and for people to declare that as state-of-the-art, but the industry is not happy with that, and it's not enough … for an industrial application."
Additionally, while this is scoped towards industrial use cases for now, Jim Fan, director of AI and distinguished research scientist at NVIDIA, told The Deep View that it's a "general-purpose paradigm." That means that it can eventually expand to robotic applications in retail and, eventually, home settings, serving as a step towards a major goal in the robotics industry: generalization.
"These days, the frontier models are actually really good at generalization," said Fan. "I think agentic robotics will really help us clean up the dirty laundry that none of us want to do."
Our Deeper View #
Along with potentially representing a major step towards the long held industry challenge of general-purpose robotics, Nvidia and UC Berkeley's system also counters a common industry theme that visual AI models, such as world models and vision-language-action models, are the key to solving this challenge. These types of models are often hindered in their growth by the fact that authentic, real-world data tends to be more limited than language data, something that Goldberg has pointed out in previous research on the so-called "100,000 year data gap." However, there is an additional beneficiary to agentic robotics: frontier AI labs. If agents run by large language models are put in charge of designating token-dense actions for factory robots, that could create a massive new revenue stream for the frontier labs.