{"slug": "chelsea-finn-on-the-next-decade-in-robotics", "title": "Chelsea Finn on the next decade in robotics", "summary": "Physical Intelligence cofounder Chelsea Finn said at Startup School 2026 that robots can already fold laundry, make espresso, and clean kitchens, but the key challenge is making them reliable for long periods without human supervision. She highlighted that reinforcement learning has doubled robot throughput and that robotics is entering its 'GPT era,' moving from specialized models to general-purpose systems that work across tasks, robots, and environments.", "body_md": "# Chelsea Finn on the next decade in robotics\n\n### Physical Intelligence's cofounder on how robots go from impressive demos to reliable, general-purpose systems that do valuable work in the real world.\n\nRobots can already fold laundry, make espresso, clean kitchens, and assemble things. The harder problem is getting them to do those tasks reliably, for long periods of time, without a human babysitting them.\n\nAt Startup School 2026, Physical Intelligence cofounder Chelsea Finn explains what it takes to build general-purpose robots that work in the real world. She shares how reinforcement learning has doubled robot throughput, how their systems can run autonomously for hours, and why she believes robotics is entering its GPT era: moving from specialized models toward general-purpose systems that can work across tasks, robots, and environments.\n\n**Timestamps**\n\n00:00 — The State of Physical Intelligence\n\n01:23 — What It Takes to Make Robots Useful\n\n05:11 — The Reliability Problem\n\n07:43 — Reinforcement Learning for Robotics\n\n09:35 — Learning From Failures\n\n12:43 — Training Robots to Improve Themselves\n\n14:21 — Can a Robot Work for 13 Hours Straight?\n\n17:36 — Why Robots Need Memory\n\n21:22 — Building a General-Purpose Robot\n\n25:02 — From Fine-Tuning to Out-of-the-Box Models\n\n27:35 — Training on All the Data\n\n30:20 — One Model That Beats the Specialists\n\n31:21 — Compositional Generalization\n\n37:49 — The GPT Era of Robotics\n\n39:49 — Q&A\n\n**Transcript**\n\nHi everyone. Today I’m going to be talking about the state of the art of physical intelligence. In particular, two years ago, I founded a company called Physical Intelligence, and we’re really interested in how we can develop any robot or allow any robot to do any task in the real world. I spoke at this event last year, and at that event, I shared some of our progress at Physical Intelligence, where we could do really complicated tasks like unloading and folding laundry. I also talked about how, for the first time, we showed how robots can do useful tasks in environments and rooms they’ve never been in before. Since then, over the past year, we have gotten robots to do a lot of other really cool things. For example, we’ve gotten robots to be able to wash a greasy pan in the top right, peel a carrot in the video below that, make a grilled cheese sandwich in the video below that, slice zucchini, and so forth.\n\nBut what I’d really like to focus on today isn’t just cool videos of robots doing lots of different things, but what it actually takes to get robots to be useful in the real world. Specifically, how can we develop general-purpose robots that are useful in the real world? There are two aspects of this. The first is general purpose—how we can develop general-purpose models. The second is actually bringing those models into the real world so that they can have an impact and be useful to people. In the first part, I’ll talk about being useful in the real world. To actually bring a technology to the real world, I think we need to figure out—it’s helpful to look at what people have done in the past to bring AI into the real world. If we look at a timeline of major production launches that are leveraging technology like machine learning, we can see a timeline like this.\n\nI think the first early examples of machine learning being used for real in the real world were for things like product recommendations and ad ranking. Then, five years later, we started to see not just machine learning being used, but deep learning being used for the same sorts of applications. This was a really exciting advance because deep learning is an algorithm that you can really apply out of the box to scenarios that involve complex inputs and outputs, and it makes it easier to translate to other applications. But from there, I think an even more exciting moment in time that we saw in terms of machine learning and AI in production was in 2022 with the launch of ChatGPT. This was the first time we saw a general-purpose model truly being used by many different people in the real world.\n\nWithin five days, ChatGPT had reached a million users. More recently, we’ve seen things like Claude Code also be incredibly useful, hopefully to many of us in the real world, and other coding agents. If we look at how AI has been used in the real world and look at this, I think there are a few different takeaways we could make. The first is that generalist models are increasingly being used for real-world problems. We’re actually seeing general-purpose, generalist AI models that can do many different things being used in the real world, and we see that transition from the left to the right. But I also think there’s a more nuanced observation we can make from looking at these applications. In particular, if we look at all of these different applications where machine learning has actually been useful in the real world and profitable and so forth, in all of these applications, the customer is making a decision based off of the recommendation of the AI model, more or less.\n\nThis means that if the customer is ultimately making the decision, if the system makes a mistake, that’s okay because usually the person can recognize that or decide what to do even despite that mistake. So even when these sorts of systems aren’t perfect, they’re still incredibly useful to different people, and there’s less pressure on them to be completely perfect. I think that physical AI and robotics is pretty different from this. If we think about physical AI that is actually operating in the physical world, they have to be directly making decisions that affect the physical world. This means they’re going to be far more useful when they’re operating fully autonomously. As a result, this requires us to develop physical AI systems that make far fewer mistakes than the machine learning systems that have been deployed thus far.\n\nOne really exciting thing to highlight that has happened recently is that a year ago, Waymo passed the quarter of a million weekly autonomous rides, suggesting that it is really possible to develop a machine learning-based system that can operate in a trustworthy and autonomous way directly in the physical world. I think that brings a lot of hope and optimism for actually doing the same with the rest of AI in the physical world. So if we want to develop general-purpose robots in the real world, I think we need to think about how we’re going to make them autonomous for long periods of time so that they’re actually useful, rather than something where a human is basing decisions on the predictions of the model. To think about long-term autonomy, I want to ground this in a specific example and say that we wanted a robot to make espresso.\n\nAnd if we want it to be useful for us, we need it to make espresso reliably so that we don’t have to babysit the robot very frequently in order for it to help serve drinks. Even on its own, this task is really difficult. Actually operating the portafilter requires very precise and forceful control to insert it appropriately. It also needs to smoothly handle cups with liquid in them and not spill those cups. It also needs to have an accurate sense of timing, which often isn’t an issue in other areas of machine learning. Not only do we want to do this pretty challenging task, we want to do it with over 90% reliability. So how can we do this? The first step in machine learning is always to collect some dataset, train a model, and evaluate how good your model is.\n\nUnfortunately, this rarely works reliably on the very first try. In practice, it’s better to iterate on the model that you’ve developed, where you try to collect more data or improve the quality of the labels in your dataset, or make the labels more detailed, collect more data of the edge cases—the scenarios where it’s not working well—adjust the balancing of the dataset, and so forth. While this generally improves the reliability of the model, people eventually get tired, and it’s hard to get really, really high reliability with a person manually tuning this. What would be even better is if the AI system itself can iterate on the scenario in which you want it to have higher reliability, where it automatically seeks out places where it needs more data, where it needs more supervision.\n\nIf we can do this for many more iterations because it’s automatic rather than a person doing it, then this might be the way to get really, really high—like 99 plus percent—reliability from physical AI systems. This is the approach that we’ll take. This looks a lot like a reinforcement learning algorithm that’s trying to attempt the task, learn from its failures, and get better and better on its own. So then how do we develop a scalable reinforcement learning recipe for robotics? In language models, we have algorithms like PPO and GRPO. These have scaled to large language models and have enabled really complex reasoning. But there’s a bit of a challenge in applying this to robotics, which is that these algorithms have been trained with millions of attempts or sometimes even tens of millions of attempts by scaling up the compute, because each attempt is simply running the language model in a data center just by using compute.\n\nIf we were to translate this very approximately to robotics—say we had maybe not millions or tens of millions, but just one million trajectories of a one-minute robot task (this is even shorter than the espresso task that I talked about)—this would correspond to 700 robot days to get high reliability for that task. Maybe this isn’t completely out of the question, but this would be quite challenging to do. That’s because the calculus is a little bit different. We’re not just running compute to optimize for a use case. We’re actually running the robot in the real world and using the hardware and attempting the task in the real world. So we’d like to have an algorithm that can iterate much more efficiently. There are actually ways that we can make these algorithms a lot more efficient. There are a couple of large inefficiencies in these reinforcement learning algorithms for language models.\n\nThe first is that they spend a lot of time on dead-end trajectories. Maybe this is okay if you’re just spending compute on it, but this would cost a lot in the physical world. We can look at a concrete example: say that we want a robot to construct cardboard boxes and stack them on the right. In this trajectory, the robot accidentally grabbed two boxes that are flush against each other. If we let it continue, it would just continue to try to fold that box rather than separate out the two boxes. Trying to fold two boxes together isn’t useful data that will teach the model how to get better at the task. That would be wasting a lot of time on the robot attempting to go down the wrong path for solving the problem. Instead of spending a lot of time trying to do that task, we’ll have a human intervene and show the robot what to do and how to recover from that situation.\n\nWhat you can see here is a human teleoperating and intervening with the robot, showing it that to recover from this situation, it needs to essentially try to separate out the two boxes. It then puts its gripper in, sees if the robot could autonomously recover. It doesn’t autonomously recover, so the person intervenes again to help it get back on the right track so that we’re efficiently using the data on the robot.\n\nThis is the first thing that we can do: we can show the robot how to recover early or how to recover so that we’re not spending time on dead-end trajectories, or at the very least just terminate the episode early. The second thing that we can do is—PPO and GRPO and these kinds of algorithms—they make many attempts at a single prompt. Depending on the algorithm, they’re essentially trying to estimate for these different responses what is a good response and what’s a bad response. Even for an individual prompt, they’re going to roll out 10 or 50 times for that individual prompt. They’re doing this because they’re trying to estimate the value of these different attempts to then increase the likelihood of good things and decrease the likelihood of bad things.\n\nBut we can amortize this cost rather than trying to collect a lot of attempts for a single prompt. We can amortize this across different prompts and learn a much more general value estimate of what’s good and what’s bad, and use this to improve with our autonomous experience. What this looks like is we can train a general-purpose value function on lots of videos of the robot experience. This can learn things like, if it accidentally unfolds a shirt when it’s trying to fold, that’s bad, and that’s making negative progress—it’s shown in red. Or if it’s making forward progress, it recognizes that as well. The same value function can also estimate what’s good and bad for a completely different scenario, in this case for retrieving an item from a fridge. This sort of general-purpose value model that’s predicting the time to success can significantly reduce the number of attempts needed to learn how to improve from experience.\n\nWith these two improvements to a reinforcement learning system, we have a general improvement algorithm that trains a foundation model on diverse data, then collects experience from that with a human intervening as necessary to help prevent dead-end trajectories, and then trains a general-purpose estimate of what’s good and bad—the value function—and then uses that to improve the model. With this sort of improvement, we’re able to fine-tune a foundation model to higher degrees of performance. In the task of making a latte, in this case we’ll be making a latte in collaboration with a person where the robot is in charge of making the espresso and the person is in charge of steaming the milk. This is what the model looks like. The model is directly controlling the joints of the robot using the images from the robot’s cameras as input.\n\nWe can see that the model is able to do the pretty challenging task of inserting the portafilter, waiting the appropriate amount of time for the espresso to dispense, pouring the steamed milk into the cup. The last part of this task is actually the most challenging, where it needs to take a very full cup of latte and transfer that over to the coaster. Here’s the observation that the robot sees directly. You can see that the policy is super delicate and able to balance the cup appropriately and smoothly so that the latte doesn’t spill.\n\nThis gives you a sense of the difficulty of this task. Going back to this reliability question, we took this policy, and we ran it not just once, but we ran it for 13 hours straight. We wanted to evaluate: is this policy not only good at making a latte once, but can it do so reliably to the extent that it would be needed to be useful in the real world? Here’s a time-lapse of that process. We found that the robot was reliable enough to be useful for long stretches of time without making mistakes frequently. The same algorithm isn’t specific for making lattes, of course. We also applied this to other applications as well. Dandelion Chocolate Factory is a few blocks from our office, so we took a workflow that they typically have a person do, which is to construct these cardboard boxes, label them, and stack them.\n\nWe trained our robot to do exactly their real workflow and trained it with the reinforcement learning algorithm that I talked about to get a policy that is far more reliable at constructing, labeling, and stacking these boxes. We also applied this algorithm to folding clothes as well. In this case, we wanted to not just test how well the model could do one task in one environment, but to do it in many environments. These are clothing items that the robot has never seen before in a home it’s never seen before, and it’s able to do so. It acts autonomously for an extended period of time.\n\nVideos don’t always show everything, so we also quantitatively measured the reliability of these models. We care both about the reliability as well as the speed—how many boxes can it build per hour? We’re going to measure throughput, which couples both success rate and speed. We find that over the phases of training from pre-training to an SFT-like stage to an RL post-training stage, we see a drastic increase in success rate and throughput. Specifically, around a 2X throughput just from the RL stage itself, showing how we can get much greater reliability from reinforcement learning. For the espresso task, if we look specifically at the success rate, we achieved over 90% success rate on making espresso. The takeaways for this part are that we can develop a scalable recipe for high reliability of complex robotic manipulation tasks.\n\nWe saw in this case a 2X higher throughput from using experience and interventions. Most importantly, we saw how we can achieve long-term autonomy in real workflows that people actually care about in the real world. This is what it’s going to take, I think, for robots to be useful in the real world. There’s also a lot more work and a lot more opportunities. We only ran a few iterations of improvement of this algorithm, and with more iterations, we should be able to see even greater improvement or even greater reliability. Even with this improvement, the robot still makes mistakes. It’s also still slower than people. There’s a ton of room for improvement for developing even more powerful recipes.\n\nSo we’ve seen long-term autonomy for these different workflows, but there’s actually one more ingredient that I’d like to talk about for enabling robots to be autonomous and useful for long periods of time. That ingredient is memory. You might be surprised to hear that most state-of-the-art foundation models for robotics have no memory or context. They’re just operating on the current sensor observations, the current camera readings, and predicting actions based off of that. You can do short motor skills and repetitive tasks without memory. The videos that I showed before didn’t have any context either. But if you want to do a long task that involves multiple different steps in sequence, then memory is critical for tracking progress of the steps that you’ve completed so far. If it’s critical for doing these kinds of long-horizon tasks, then why don’t these models have any context or memory?\n\nThere are a couple of technical reasons for this, and I’ll talk through one of them, which is that if you naively approach memory and try to feed in context like past video to a robot foundation model, say that you just pass in 10 seconds of video. Maybe this 10 seconds of video is sampled at 50 hertz, which is a common control frequency in robotics, and you feed in all four camera streams on the robot and use around 256 tokens per image. This corresponds to passing in half a million tokens into your model, which is a lot of tokens. Trying to do that in real time into your model right now is quite challenging. Even if you sub-sample to one frame per second, you’re still going to be passing in 10,000 tokens into your model, which at least right now is prohibitively expensive for these models.\n\nAnd that’s still only 10 seconds of memory. I don’t have time to go into the technical details of exactly what we did here, but we also developed a solution for this context problem. Specifically, we developed a system that has memory at multiple timescales. The first is a short-term video memory that has about 10 seconds of video memory, but is computed much more efficiently than naively passing it into the model. Then, for longer memory, for memory that spans multiple minutes or multiple hours, we don’t necessarily need video of exactly what happened in that past history. Instead, we represent memory for those parts in text, where we summarize what happened in text space and then incorporate that much more compressed textual summary of what happened over the past 10 or 15 minutes into the model as well. With this memory at multiple different timescales, we’re able to enable robots to do tasks that can operate for 10 or 15 minutes at a time completely autonomously.\n\nWhat’s different from the previous slide or what I showed previously is that this task isn’t repetitive. This is going to be a 10 to 15-minute task that involves cleaning a kitchen. The robot isn’t just repeatedly making espresso over and over again. What it involves is wiping the counter with a sponge, then drying the counter with a paper towel, throwing away the paper towel. Next, it’s going to put away the mustard into the fridge. Then it will put the dishes away into the cabinet, wash some of the dirty dishes in the sink, and so on. By incorporating memory, it’s able to do a task that requires keeping track of all of these different steps that are done to clean the kitchen and successfully operate for 10 to 15 minutes completely autonomously.\n\nGreat. So those were a couple ingredients for long-term autonomy. Now I’d like to build on that and actually take those ingredients and put them into a general-purpose model that can do everything that I showed before, but also can do that in a single model and can do some other things as well. To think about developing such a general-purpose model, I think it’s really helpful to contextualize where robotics is at within the timeline of other developments in generalist AI. If we think about how generalist AI systems have evolved over the past 15 years, I think the first major milestone was in 2012 when we saw that a deep learning system trained from scratch was the first time that it topped an external benchmark, and all of the previous methods for that benchmark were specifically designed for that application.\n\nAll of the previous methods—specifically, this was the ImageNet benchmark—were designed specifically for image classification. This was the first time that a deep learning-based system outperformed those more specialist systems. This is a much more general algorithm that wasn’t specifically designed for image recognition. Then just a couple years later, we found that we weren’t just training algorithms from scratch, but we were able to get models like pre-trained models that are useful for fine-tuning to downstream tasks. It became the norm to take a model that was pre-trained on ImageNet and then fine-tune it on a downstream task. We saw better performance from using that pre-trained model, like BERT or an ImageNet pre-trained model. From there, I think the next big phase and the next big transition in generalist AI models wasn’t using pre-trained models, but moving from a pre-training/fine-tuning regime to a regime where we’re just using generalist models out of the box.\n\nAnd this was with models like the start of GPT-2. And of course, almost all the models that we interact with today worked out of the box without fine-tuning, at least most of the consumer models. There are actually other models that still use a lot of fine-tuning. One other milestone that I want to highlight was in 2021, where I think I saw the first signs of compositional generalization in these models. One specific instance of that was with DALL-E, and I’ll talk a little bit more about that in a later slide. This is how generalist AI has advanced over the past 15 years. Meanwhile, if we think about physical AI, even just three years ago in 2023, it was extremely common for people working on robotics to collect a bespoke dataset from scratch for an individual project and train from scratch on that dataset.\n\nThis is analogous to collecting ImageNet from scratch and training on ImageNet, or training on the dataset that you just collected from scratch. If you want to develop a general-purpose model, and you have to collect the dataset from scratch for every single project, you’re probably not going to make a lot of progress. Until just a few years ago, I think we were pretty far on the left of this timeline. Until more recently, we’ve been in the 2014 phase where we have some good pre-trained models, but we haven’t really been truly in the regime on the right. So how do we get to that regime on the right? Specifically, how do we develop a single general-purpose model that works out of the box and also shows compositional generalization?\n\nSo this is two goals. The first is an out-of-the-box model. This is analogous to going from BERT to GPT. Right now, the best robot performance—if you want to get your model to perform the best that it can on a given task—always requires fine-tuning. Some of the videos that I showed at the beginning were fine-tuned models to do things like unlocking a lock. Other work that we’ve done on measuring human-to-robot transfer also needed fine-tuning to get the best performance. All of the videos that I showed with RL post-training were also fine-tuning on an individual task to get the best performance on something like making espresso. But if you have to fine-tune a model, you aren’t getting a general-purpose model for the things that you want it to do because you have to fine-tune it for each individual thing.\n\nOur first goal is to move towards a single general-purpose model that can do all of the things that you want it to do. The second goal that I mentioned is compositional generalization. This is inspired by the DALL-E result from 2021. I think that this was a really important and exciting milestone because of the compositional generalization that it achieved. Specifically, when you have compositional generalization—when you can bridge the concept of an avocado and a chair and show that you can combine those two—it means that the model has at least some kind of conceptual understanding of what an avocado is and what a chair is, to the point that it can combine them into something that exhibits both concepts at the same time. Second, it means that you have some degree of data efficiency, where your data doesn’t need to cover all of the possible combinations of concepts represented in your data.\n\nYou don’t need pictures of avocado chairs in your dataset in order to generate something like this. Or you don’t need combinations of other things that you might ask the model to do when it’s deployed. Even back in 2021, it wasn’t perfect, but these signs of compositional generalization were really exciting for demonstrating these two attributes of the model. So we have these two goals that we’d like to achieve: an out-of-the-box model and compositional generalization. The tried and tested recipe for developing this kind of model is to first take a sufficiently large and diverse dataset, and second, train a model with sufficient capacity. So what we’re going to do is try to use all of the data that we have available. This includes really diverse robot demonstration data, including really low-quality demonstration data.\n\nIt’s also going to include policy rollout data—attempts from the robot at doing the task. All of the training data that was used for reinforcement learning for the previous tasks will be included in the training recipe. We’re also going to include videos of humans, and we’re also going to include data from the web—all of the data that we have. To train a model with sufficient capacity, of course we’ll train a model that’s large enough, but to fit data that’s so heterogeneous, we also find it particularly important to prompt the model with all of the context that it needs in order to predict actions. We found that this idea was really the key unlock to using this kind of data and this degree of heterogeneity. Specifically, what this looks like is we’re going to train a foundation model that takes as input the memory that I mentioned before, an instruction of what to do, and it’s also going to take as input a sub-task instruction of what the next immediate thing it should do is.\n\nIt’ll also take as input metadata that indicates the quality of the data, the length of the episodes, and so forth. This metadata gives it a lot more information about how it should predict the next action. Optionally, we’ll also train the model with a sub-goal image as a prompt to the model, essentially saying, a few seconds from now, you should try to reach something that looks like this image. With this detailed prompting, we find that the model can really make use of much more heterogeneous data. I’ll show some comparisons later that really show how important it is. To actually deploy this model, we need to provide things like the sub-task construction and sub-goal images. With that, we can train a high-level policy that predicts the sub-task construction—what to do next, what is the next sub-task for the task of cleaning the kitchen.\n\nWe’ll additionally train a world model to generate images for what the robot should do next as sub-goal image conditioning. With this, we’ll train a single model with those attributes on all of the diverse data that we had available. Here are some examples of what that single model can do. All of these videos are from a single model, specifically a model that we called the π0.7 model. On the left, you can see it doing things like folding a collared shirt. On the top right, it’s doing a really precise assembly step where it needs to insert a screw and drill that screw into a robot arm. On the bottom right, the robot is replacing a trash bag in a trash can.\n\nWe had two goals at the start of this. The first was to move towards an out-of-the-box model. Even those videos showed that out of the box, the model’s able to do quite a bit, but really the key question here is how does this pre-trained model compare to the specialists that were trained specifically for coffee making, specifically for box building that I talked about previously? If we measure the throughput and the success rate of this single π0.7 model versus the fine-tuned π0.6 model, we see that across the board, the single pre-trained π0.7 model matches or outperforms the fine-tuned specialists that were developed with reinforcement learning post-training for those downstream tasks. We see that it’s able to match the performance of specialists. It also holds for SFT specialists, not just RL post-trained models as well, suggesting that we do indeed have a single model that can do a lot of different tasks with a really high degree of performance out of the box.\n\nThat was the first goal of out-of-the-box models. The second goal is compositional generalization. There are a few different ways to measure this, and there are many different ways you might try to combine concepts in robotics. The first test that we wanted to do was to see if a robot could interact with an appliance that’s quite rare, like an air fryer. This is an example. We wanted to see if it could open an air fryer, put a sweet potato in the air fryer, and close the air fryer. We picked this because we thought that the data set didn’t have any air fryers in it. We didn’t intentionally collect any training data with air fryers.\n\nAfter we did some analysis on the data set, we found that our data set was so diverse that it did actually have three episodes with air fryers in it. We expect that they likely weren’t having an impact and that even if you didn’t include those exact three episodes, it likely would still work. What we found generally is that the robot was able to interact with an appliance that was hardly represented at all in the training dataset and combine the skill of interacting with it—opening it, closing it, and so forth—with this object that it hasn’t seen before. After instructing it, as Lucy did, we can train a high-level policy to do this task fully autonomously. You can see the robot doing that in this video. That’s the first form of compositional generalization. The second compositional generalization test that we wanted to look at is whether we could compositionally generalize between tasks and robot platforms.\n\nWe wanted to take a robot platform called the Byarm UR5E robot. It’s a very large industrial robot platform. We wanted to see if it could fold clothes despite the fact that we didn’t collect any folding data on this robot platform. Specifically, we had data of folding clothes, like folding a shirt, on the robot platform that’s pictured here on the left. Then we wanted to see, out of the box, without collecting any folding data on this very different robot platform on the right, could the robot successfully do the task? What we see in this video is that it did compositionally generalize in this manner. The first time we saw the robot do this, we were floored because there was no training data for this task. The robot here is quite different from the other robot, not just in size, but also in the lengths of the linkages of the robot, in the configuration of the joints of the robot, and so forth.\n\nThis is a 1X speed video, so it’s not the fastest thing. Obviously, if you haven’t seen any training data on something, you might not get it right the first time. It’s literally the robot’s first time folding a shirt. It might take a few attempts, but eventually it will get to the folded shirt. You can also see the generated sub-goal images on the top left. Those are the model’s attempts to generate images that will make progress on the folding task. Those are passed as input to the model. We see the folded shirt here. I think it’s going to make a couple small corrections at the end to try to make it a little bit smoother.\n\nCool. So the takeaway here is that both in terms of language-object interactions and in terms of task-robot interactions, we see strong signs of compositional generalization in this model. Quantitatively, we also see that as we get to these more advanced models like the PIO7 model, the performance of folding towels and folding shirts on this platform that it hasn’t seen before increases dramatically. It even approaches the performance of human tele-op, despite the fact that we didn’t have any robot-specific training data for folding clothes. For the last experiment that we did here, I think this is perhaps the most interesting experiment: we wanted to test how important are the two ingredients that I mentioned. How important is diverse data, and how important is this capacity or detailed prompting for the kinds of results that I showed?\n\nIf we remove the most diverse data from the model training, shown in the grayish color, we find that the performance on held-out tasks decreases dramatically. Whereas if we just take out a random 20% of the data that’s less diverse than the most diverse subset, the performance only decreases a little bit. So this suggests that having really diverse data plays an important role in enabling it to generalize to new tasks. We also tried to ablate the fact that we are prompting the model with metadata. For this experiment, we looked at with and without prompting with metadata. With prompting is shown in yellow and without prompting is shown in the gray color. With prompting, it helps significantly. But the most interesting thing is if you look at when you add—so this plot is showing as you add more and more data, and specifically as you add more and more low-quality data, what is the performance?\n\nWithout metadata prompting, when you add lower-quality data from 80% data to 100% data, the performance actually decreases, which is perhaps not too surprising because you’re adding low-quality data to your data mixture. Whereas with the metadata prompting, the performance actually increases when you add that low-quality data, suggesting that it’s able to get a lot more juice out of even low-quality data when you include this kind of prompting.\n\nCool. So the takeaways here are that we found that we’re able to train a single model to control the robots that matches or exceeds the performance of specialized post-trained models, kind of like going from a BERT-like pre-trained model to a model that really works out of the box like GPT. We also saw strong signs of compositional generalization in a DALL-E-like way—for example, in compositionally generalizing skills applied to appliances and skills applied to new robots in ways that weren’t seen in the training data. All the videos and experiments that I showed were just evaluating the model out of the box without any post-training. The paper and the technical report online have a lot more experiments and a lot more details. So we talked about long-term autonomy. We then showed how we can develop that in a single general-purpose model.\n\nWhere are we at now? The first thing that I’ll mention is if we go back to the timeline of generalist AI, I think that we now firmly have physical intelligence in the right side of this timeline. We’re more in a GPT- and DALL-E-like era for robotics and physical intelligence, which is really exciting. I think that we’ve gotten there in just a few years. Lastly, we also have these models that are actually deployed in real-world circumstances. The two videos on the top are actually two YC companies, Ultra and Weave, that have taken fine-pi models and post-trained them in deployment to do tasks like folding laundry and packaging in a warehouse. The video on the bottom left is the video that I showed previously. This kind of model works for a really diverse set of robot embodiments.\n\nThe ones on the top and the left are a more standard bimanual platform, but it also can be adapted to things like drones, quadcopters, surgical robots, and on the bottom right for things like tractors. This is truly showing how physical intelligence can make an impact, not just in demos and research and so forth, but actually in real-world deployment. I think that as we go, we’ll start to see more and more robots actually deployed for real in the physical world with all the advances that we’ve been seeing over the past few years. Awesome. The last thing that I’ll mention shamelessly is that we are hiring at Physical Intelligence. So if you’re excited about some of the stuff that I talked about, we encourage you to take a look at some of the open roles and apply.\n\nAnd yeah, definitely have time for questions and happy to get all your thoughts. Thanks.\n\n### Q&A\n\n**How far away are we from a ChatGPT moment for robotics, and what will that look like?**\n\nI’ll start with the second part, which is that I’m not sure it will really look like the ChatGPT moment that we saw in language models. With something like ChatGPT, we saw it pass a million users in five days. I think that the distribution channel for physical models is going to be slower, unfortunately, because you actually need a physical robot there. For something like Waymo, the rollout has actually been incredible to see, but it still takes time to actually deploy things on physical devices. So I don’t know if we’ll have a single moment that has the distribution that ChatGPT had. At the same time, in terms of the capabilities of these models, I think that we are really starting to get to the point where these models are actually useful in the real world.\n\nAnd I think that getting to the capabilities of ChatGPT is very much on the horizon in the next few years.\n\n**When should a small team switch from scaling per-site models to a generalist policy? What does that transition look like, and what signals tell you it’s time?**\n\nThis is a good question. At the very least, I think that just starting with the generalist policy and then fine-tuning it, even right off the bat, can be really effective. Fortunately, there are really powerful generalist policies that are open source. The π0 and π0.5 models are open source, for example. We’ve seen a lot of people get a lot of use out of those models already. We’re also working with a lot of partners like the tractor company, Ultra, and Weave to take our most recent models and get even more juice out of them—more powerful models for their own applications.\n\nSo even right off the bat, I think you can use them. The only scenario in which I wouldn’t use them is if you’re in a really constrained environment. I’ve talked to some folks working on surgical robots that are in an operating room in the basement with no internet connection and a really bad GPU. Sometimes it’s just really hard to use a larger model, but you still can do local inference on a workstation with these models. So I think that, right away, just taking π0.5 or your favorite model and fine-tuning it is the way to go. I think we’ll see lots of these small companies, and there’s so much work to do in terms of actually getting these robots to work with this technology in the real world.\n\n**Given how fast robotics is moving in industry, what are the real advantages and drawbacks of doing a PhD today — especially for someone who wants to go into industry afterwards?**\n\nI was not planning to do a PhD. I was always planning to go straight to industry. My parents are engineers and worked in industry. I thought that the way to have impact was to go to a company and so forth. My dad even told me that he wouldn’t hire someone with a PhD, so I thought maybe I shouldn’t get a PhD if I wouldn’t be able to get a job. But he’s in a different field as well, in civil engineering.\n\nAt the same time, I think that a PhD is an incredible opportunity, and I love my PhD. Obviously, it depends a lot on the advisor, depends a lot on what you would be doing, and so forth. But I think a PhD is an incredible opportunity to first learn a lot about how to handle uncertainty, how to pick good problems to work on. In research, no one even gives you the problem to work on. You have to pick the problem. And you don’t know for the problem that you picked if it is achievable to make progress on that problem in a six-month time span, two-year time span, ten-year time span. So you learn about how to deal with that uncertainty.\n\nThat’s really useful. It’s also an opportunity to do amazing research and, in many cases, have a lot of freedom to work on the research that you find most exciting. I think that today it’s still an amazing opportunity to do work, to learn about uncertainty. Learning about uncertainty is really useful in startup environments, in being at the frontier of AI, because we don’t know now. No one knows what the best route is to make these models more and more powerful. At the same time, there are also a lot of incredible opportunities in industry. In terms of what goes into developing everything that I showed, it’s not just the research. There is a whole software stack that needs to run on the robot, needs to run reliably. There’s everything on the hardware side.\n\nThere’s also the machine learning infrastructure, the data infrastructure, and all that. You don’t need a PhD necessarily to do a lot of that engineering work. On the research side, there are often opportunities to get involved as well. A lot of research is engineering these days. So I think it kind of depends. It’s a very personal decision and depends on what you want.\n\nEven today, I think I probably, retrospectively, would want to do a PhD just to learn about how to handle uncertainty and to do research, because I really love being at the frontier and thinking about challenging problems. But there are also a lot of really amazing opportunities in both paths.\n\n**Large language models learned from the internet, but robots don’t have an internet-scale dataset of physical experience. What’s the robotics equivalent, and how do we get it?**\n\nIn robotics—well, maybe in language models to start off, the data on the web is language data. Not all of it’s high quality, but some of it is really informative and useful. It is data that reflects a lot of what you want a model to do. You want it to be able to predict text, to complete text, answer questions, and so forth.\n\nAnd there’s a lot of questions being answered on the internet and a lot of text that’s being completed on the internet. In general, with machine learning, you want train to match test. You want the thing that you’re going to be training your model on to be reflective of the thing that you’re going to be asking it to do later on. I think the equivalent in robotics is data of robots operating in real-world circumstances. The way that we approach it at Physical Intelligence is to collect data, to collect robot experience of robots doing all sorts of tasks. You can collect this with teleoperation to get initial data of robots doing useful things. But in the long run, I think it will also contain a lot of autonomous experience of robots deployed attempting things. Just like how we see in language models, how now a lot of time is spent generating data, generating synthetic data by actually running the model and having it think through things.\n\nI think a lot of the data in the future in robotics is going to be the robot attempting to do lots of tasks in lots of real-world circumstances. That’s kind of what it looks like. I also think that there are other possible sources of information that are really useful for model training, like videos of people doing things like YouTube, web data, and captioned images that tell you this is a kitchen that has a fridge on the right of the sink and so forth. All of that data can be really useful for developing a kind of frontier multimodal model that can control robots to do things, reason through how to do a long task, and also control the robot to do those tasks. I think that there’s no substitute for the robot experience itself. If you watch a human do something—like if I watch Roger Federer play tennis—it doesn’t mean I can play tennis as well as him, unfortunately.\n\nLikewise, robots can’t watch a person doing something and then figure out how to do it themselves directly. They really need their experience on their own platform to learn effectively. I think that we will need large data sets. That doesn’t mean the human video isn’t useful. It’s useful to watch Roger Federer play tennis, but the actual experience on robot platforms will be a critical component of developing an analogous data set for robotics.\n\n**Is it possible that general-purpose robotics models get democratized via open source the way large language models did? Or will the cost of embodied data and hardware keep the best models concentrated in a few well-resourced labs?**\n\nI think this is a good question.\n\nI do think the cost of embodied data and hardware could very much make this look different because it’s harder to get data even to distill a model, for example, just readily on the internet. I also think that we’ve seen pretty large data sets get open-sourced as well and pretty powerful models get open-sourced. It’s really hard to say exactly what will happen. I don’t know. The one thing that I will say is that with language models, even aside from democratization and really getting models that perform at the state of the art, even then companies that are focusing a lot on closed-source models are also doing a lot of open sourcing. There are Gemma, for example, and the GPT open source and so forth. I think these companies like to support open source because it helps build the ecosystem around the things that they’re building.\n\nSo I guess I’m optimistic that there will be a strong open source community regardless, but I don’t know if it will play out exactly the way that language models played out.\n\n**Does the model output raw motor commands directly, or a target hand position that a controller solves for? What makes that the right level to learn at?**\n\nAll the models that I showed were outputting target joint positions. So, what is the angle of this joint? What is the angle of this joint, and so forth, that you want to hit? Then there’s a controller, like a PD controller, that is trying to hit that target position for those joints. The model is also trained to predict target gripper positions: where in 3D space should my gripper be? You could also use that as well and back out the joint positions. Another thing you could do is go directly to motor torques or to voltages or efforts.\n\nThere are pros and cons of different options. We have found controlling joints and controlling in the 3D space of the gripper to both work well.\n\nThere’s pros and cons. One thing that would be nice about going directly to the voltages is that you could get a more stiff kind of output or a less stiff output. Whereas the controller—if you have a fixed controller—then you’re not letting your model control that aspect. So there are different pros and cons. What we’re working on seems to work. It doesn’t seem to be a bottleneck. I often like to focus on the things that seem to be bottlenecks versus things that don’t seem to be bottlenecks.\n\n**Do robots need something like imagination — the ability to picture what should happen next — before they can become truly useful?**\n\nThe pilot seven model that I showed has something like this, where it can imagine what a future image should look like and then try to accomplish that.\n\nWe found that leads to improvement. And we saw on the short folding example, we saw a quantitative bump from using that sort of imagination compared to not using it. At the same time, I think the model performs surprisingly well without that as well. We were thinking about writing an entire paper, an entire technical report just about that capability in that model. But the model without that was so good that we felt like we needed to have that play a bigger part of the story because that seemed like it was really delivering in terms of getting really strong results. So it seems like one design choice.\n\nI think it’s hard to say if it’s going to be a critical component or not. The good news with these kinds of models is that if you develop a good data set, you can run experiments and continue to test things with the data set that you have quite effectively. I also think that being able to predict the future seems like a very relevant objective compared to predicting future actions. That should help in terms of learning from all the data that you have available to you. So yeah, hard to say if it’ll necessarily be a critical component or not. It seems like, empirically so far, it helps, although perhaps not as much as you might expect. And even without that imagination, the robot can do pretty incredible things. Okay. Next is right now it seems that robots are doing amazing tasks, but in a very slow manner.\n\nWhat is needed to improve the speed? I’m really excited about improving the speed. We did see speed improvements from reinforcement learning. We also have another release called the RL token, where we showed even faster speed and actually faster speed than human teleop. I think one of the bottlenecks is that when you teleoperate robots to do things, which is the easiest way to teach a robot to do something, people are slow at teleoperating the robot. We have a couple projects in the pipeline that I think have really promising results in terms of getting fast policies. So I think more to come there. I think it’s either you need to figure out how to make the data faster or you need to figure out how to be faster than the data. We’ve seen evidence of being able to be a little bit faster than the data.\n\nAnd in terms of the next steps, it’s either to go even further than that or make the data faster.\n\nCool. What’s the most surprising task you’ve seen a robot complete recently? What do you want to see it do next? So the most surprising thing was not really a task, but when we were working on π0.7, I personally trained one of the policies for some of the initial tests for assembling this pinwheel. When I was working on training it to construct the pinwheel, one thing that really surprised me was in all the data, we carefully controlled the strategy for how to assemble the pinwheel, where you take the precut piece of paper and take a little pin and insert the pin into a hole in the paper. In all of the data, we picked up the pin with the right hand and picked up the paper with the left hand and inserted it. The robot started doing that.\n\nThen it made a mistake, and the paper ended up on the right side, and the pin ended up on the left side. What the robot did is it picked up the paper, and it picked up the pin with its left gripper. It put the pin with its left gripper and inserted it into the paper with its right. It had never seen data of inserting the pin with its left gripper. That wasn’t in the post-training data at all. It wasn’t even in pre-training either. The robot essentially had learned this sort of equivariance between its left hand and its right hand so that it could transfer behaviors from one hand to another, despite the fact that that was never in the data. That was a really cool moment. I don’t know if other people were as excited about it as I was when I shared it with some people, but it shows this emergent capability in these models that I hadn’t seen before.\n\nIn terms of what I’d love to see, I don’t know. I love seeing robots do anything. I think there’s still a long way to push in terms of reliability for robots being able to do tasks for really long periods of time. I don’t necessarily think that much about individual tasks, but more so about capabilities and how to get the next capability from these models. So yeah, anything. A robot doing anything always gets me excited, even if it’s something that hasn’t been done before. One thing that we’ve been doing recently is having robots use knives to slice vegetables. I think there’s a lot that you can do there once you can use knives safely, which is one thing that we’ve done recently.\n\n**How can someone break into robotics from a software engineering background?**\n\nGreat. So I think that first, there’s a lot of software engineering in robotics. I think you could try joining a robotics company as a software engineer. Another thing I would mention—and I’ve actually seen someone take this path—is someone who now works at Physical Intelligence. Her name is Jenny. She worked in algorithmic trading for a while, then she worked at Harvey doing legal work. She was really excited about robots, so she bought a cheap robot and in her bedroom played around with it, tried fine-tuning an open-source model, and tried to get it to do something. She shared what she had done and sent me a cold email saying, “Hey, I’m interested in working in your lab.” Her profile was promising, and she actually got out there and tried it. She was really excited about that.\n\nNow she works at Physical Intelligence. I think just getting your feet wet, trying things out, and learning from that experience—and then using that experience to share with people, have it on your resume, and so forth—is a great way to do so. Fortunately, there’s a lot of open-source material out there that can allow you to get started on those kinds of things. Great. That was the last question. Thanks everyone for listening.", "url": "https://wpnews.pro/news/chelsea-finn-on-the-next-decade-in-robotics", "canonical_source": "https://www.ycrootaccess.com/p/chelsea-finn-on-the-next-decade-in", "published_at": "2026-08-12 15:42:06+00:00", "updated_at": "2026-08-12 15:49:54.412626+00:00", "lang": "en", "topics": ["robotics", "artificial-intelligence", "machine-learning"], "entities": ["Chelsea Finn", "Physical Intelligence", "Startup School 2026"], "alternates": {"html": "https://wpnews.pro/news/chelsea-finn-on-the-next-decade-in-robotics", "markdown": "https://wpnews.pro/news/chelsea-finn-on-the-next-decade-in-robotics.md", "text": "https://wpnews.pro/news/chelsea-finn-on-the-next-decade-in-robotics.txt", "jsonld": "https://wpnews.pro/news/chelsea-finn-on-the-next-decade-in-robotics.jsonld"}}