How Rhoda AI is using video data to create generalized AI models for robots

Rhoda AI, which emerged from 18 months of stealth with $450 million in Series A funding, is deploying its direct video action (DVA) model, recently extended with mobile DVA, to robots for customers in automotive, manufacturing, logistics, and e-commerce. Co-founder and CEO Jagdeep Singh told The Robot Report that the DVA model predicts the next few hundred milliseconds of video, then converts that prediction into robot actions in a closed loop, combining manipulation and mobility into one policy. Singh argued that physical AI has trailed large language models because no internet-scale dataset exists for the physical world, and that teleoperated robot demonstrations such as folding T-shirts or making coffee were not robust enough for real-world deployment.

How Rhoda AI is using video data to create generalized AI models for robots The Robot Report https://www.therobotreport.com Rhoda AI offers a direct video action model, which it recently extended with mobile DVA, to enable robots to understand the physical world. The post How Rhoda AI is using video data to create generalized AI models for robots appeared first on The Robot Report. Earlier this year, Rhoda AI emerged from 18 months of stealth with $450 million in Series A funding. The company is working with customers in automotive, manufacturing, logistics, and e-commerce to deploy its direct video action, or DVA, model for robotics https://www.machinebrief.com/category/robotics . “We’re focused on deploying in the real world, and that means you have to deal with the messiness of the real world. Every run can be different,” Jagdeep Singh, the co-founder and CEO of Rhoda AI, told The Robot Report https://www.therobotreport.com/ . “Every configuration can be different, and you need a really robust model. To build a robust model, you need a massive training https://www.machinebrief.com/glossary/training dataset.” A large training dataset is exactly what allowed large language models LLMs like ChatGPT https://www.machinebrief.com/compare/chatgpt-vs-claude to rapidly progress in recent years. Physical AI https://www.therobotreport.com/category/design-development/ai-cognition/ has trailed behind those models because the same dataset doesn’t exist for the physical world, noted Singh. “Every word ever put on the internet, from Shakespeare to Reddit, was used to train these AI models. In the physical AI world, you didn’t have an internet-scale dataset,” he said. “People tried to collect their own data by tele-operating robots, and that approach made for interesting lab demonstrations in robots that could fold T-shirts or make coffees and so on, but none of those demonstrations were robust enough to cross over into the real world. The number of autonomous manipulation capable robots that are solving real world problems today is effectively zero.” Rhoda AI is betting on using video to data to close this gap and to create truly generalized robotics. How does a DVA model work? Many physical AI developers are creating vision-language-action VLA models, which allow robots to learn from data from images or videos of its surroundings and text instructions. These models, however, still struggle to cope with the variability of the real world, according to Rhoda AI. The company https://www.rhoda.ai/ claimed that its DVA model, which it recently extended with mobile DVA, combines manipulation and mobility into one policy. “The way it works is the model will look at a history of video frames and action tokens and language tokens. From that history, or from that context, it will make a prediction in video for the next few 100 milliseconds, the next half second or so,” explained Singh. “It’ll predict the scene and what it thinks will happen in that scene. Once you have that prediction, you pass it to a second part of the model, which is an action converter. It takes the video scene and spits out robot actions.” “Once you have that action space output, you can execute those actions on a real robot. The robot then moves for that half second, and then you make a new observation, because the state of the world has changed, and you make a new prediction for the next half second,” he continued. “It’s basically making an observation, a prediction of what should happen next, a conversion to action, an execution of the action, and then you close the loop and make a new observation and a new prediction based on that new observation.” For Rhoda AI, quantity of data is key According to Singh, any AI model that’s trained on a small dataset is going to be very brittle. These models could work if the inference https://www.machinebrief.com/glossary/inference time distribution of data is very similar to the training time distribution, but as those distributions diverge, then the model starts to fail. “Hundreds of years of videos were used to train the model, so the model has seen just an incredible diversity of everything: baseball videos, people throwing darts, waves crashing on a beach, or trees blowing in the wind,” Singh said. “Not just videos that have humans in them. Every video has something to teach the model about how the world works. Essentially, the entire internet becomes like a physics classroom.” Rhoda AI uses video data for pretraining. “Once you’ve gotten that pretraining, you then apply a small amount of task-specific training, maybe 10 to 20 hours’ worth, similar to the way you would train a new worker on an assembly line,” said Singh. “They already come in with a certain set of skills, and you need to teach them the workflow of your particular company. With that little bit of post training, the model is able to do things that traditional robots cannot do.” The Palo Alto, Calif.-based company aims to make a model that can understand the world generally. For this, it doesn’t limit the data it uses to just human-centric videos. “We looked at more heavily curated videos that only had human, egocentric manipulation in them,” Singh said. “There was a question of maybe those videos were better to train on. We found that the models that perform the best were the ones actually trained on the broadest data sets, including things that were not obviously relevant to manipulation.” This generalization also enables Rhoda AI’s models to generalize across different robot embodiments. In fact, the specific robot embodiment only comes into play in the post-training phase. How does Rhoda AI offer its models? Singh said Rhoda AI is a full-stack company, and it offers its technology in a range of methods. “We always own the AI layer, that’s the core,” Singh said. “In terms of the body or hardware layer, we can run with our body or a third-party body. In terms of the skills, we can be the ones training the skills, or we can let third parties train the skills. That provides the largest set of solutions to our customers, and that’s really our mission.” Internally, Rhoda AI is focusing on developing skills in manufacturing https://www.therobotreport.com/category/markets-industries/manufacturing/ and logistics https://www.therobotreport.com/category/markets-industries/logistics-warehousing-asrs/ . “We’re going after those areas because there’s an immediate problem there,” added Singh. “Many manufacturing and logistics companies have a shortage of labor. Many industries have been offshored for decades because its too expensive to build in those economies. So this technology offers a path to repatriate https://www.therobotreport.com/tag/reshoring/ some of those industries.” Looking ahead, Rhoda AI aims to continue getting its model out into the real world. Singh said he believes robotics’ “ChatGPT moment” will be coming sooners than people expect. “The kind of approach we’re taking really has the potential to get us there because the only way to get that level of generality is to train on a dataset that has both the scale of the internet and diversity of the internet,” said Singh. “If you don’t have both those things, your models will not be able to generalize well enough.” The post How Rhoda AI is using video data to create generalized AI models for robots https://www.therobotreport.com/how-rhoda-ai-is-using-video-data-to-create-generalized-ai-models-for-robots/ appeared first on The Robot Report https://www.therobotreport.com . Get AI news in your inbox Daily digest of what matters in AI.