Memory, SimToolReal, and World Action Models | YC Paper Club

A robotics edition of YC Paper Club: embodied memory, self-taught reasoning, dexterous sim-to-real, and the fastest way to run a world model.

This week’s Paper Club is all about robotics. Every year for the last decade, someone has promised that the era of robotics is just around the corner. But we’re still waiting.

So we gathered a bunch of the top researchers working in AI and robotics to present the latest findings on where we are and what comes next. We open with a discussion of the biggest roadblocks still in the way: the sim-to-real gap, action representation, the sensorimotor problem, and embodiment drift.

Then we cover giving robot policies memory, teaching models what’s worth reasoning about, dexterous tool use learned entirely in simulation, why the next great robotics companies will start with teleoperation, and how to run world action models without two GB200s per robot.

Timestamps

0:00 – Francois Chaubard: Ten years of “next year, robotics is solved”

7:59 – [Marcel Torne](https://marceltorne.github.io/): MEM - [Multi-Scale Embodied Memory for Vision Language Action Models](https://arxiv.org/abs/2603.03596)

20:21 – [Milan Ganai](https://milanganai.github.io/): [Self-Supervised Bootstrapping of Action-Predictive Embodied Reasoning](https://arxiv.org/abs/2602.08167)

33:42 – [Tyler Ga Wei Lum](https://tylerlum.github.io/): [SimToolReal - An Object-Centric Policy for Zero-Shot Dexterous Tool Manipulation](https://arxiv.org/abs/2602.16863)

51:21 – [Niko West](https://www.linkedin.com/in/nikolauswest/): Why the next great robotics companies will start with teleoperation

1:08:30 – [Bill Jiao](https://www.linkedin.com/in/zhandong-jiao/) & [Guanming Wang](https://www.linkedin.com/in/guanming-wang/): World action models and what comes after VLAs

Transcript

Francois: Welcome to YC Paper Club. How are you guys doing today? How do you guys like this one? We’re going to change it every time now. Today it’s actually YC Robotics Club. This is the 10th year of “next year robotics will be solved” that I’ve encountered in my career. I remember when AlphaGo came out 10 years ago, everyone said next year, clearly we have the algorithm. All we need to do is scale it up. And then MuJoCo came, and then we had literally in 3,000 iterations where we can train a robot to walk. This is quite amazing. Clearly next year robotics is going to be solved. This time it’s different. We figured out UMI data collection. And so ALOHA was the big breakout. Everyone thought next year we’re going to have robotics. Now, look at this. We can water our plants.

We can fix our bike up. We can do my Keurig here, even shave me. We have the algorithm. All we need to do is scale it up. And then 2026 was promised. They promised me that this would be the year of the robots. I was very convinced. I read the diffusion policy paper. I even played around with it a little bit myself. I’m like, this is surely the year of robotics. We have multi-step reasoning. This is totally going to happen. We have VLAs, they’re amazing. You just need tele-ops data and then you’re good to go. And I would say honestly, we’re halfway through 2026 and you can pre-order 1X Neo. I can’t buy a Pi or Figure robot just yet. We’ve had some success in work cells, but Rosie the robot is still not here. And we have six months left. It’s only July.

So maybe it will be, maybe I’ll be wrong, but it’s definitely the year of the demos. I know that for certain. The reason why is because—raise your hand if you’ve ever done tele-ops data collection. Is it easy or hard? It’s remarkably hard, especially if you have this little gripper thing, that’s all you have. You have the wrist cameras and all that stuff. It’s extremely difficult. It’s very, very finicky. And if we’re relying on this type of data and we need to scale that up like crazy, then we’re kind of doomed. I think about it in these four—there’s probably more, but at least these four—walls that we have to scale and get over, and we really haven’t yet. The first one is physical real world modeling. If you take these video models and you deploy them to, let’s say, drive a car and you’re doing the world models paper by Jürgen Schmidhuber or the Danijar Hafner Dreamer V1, V2, V3, V4, and you do this game and you play Doom in the simulated game.

In the real world, it really doesn’t respect physics. Those models don’t respect physics all that well. If you’re driving a car in that simulated world and you drive into a grocery store or Whole Foods, it magically just turns into a highway and then you don’t crash. It doesn’t really respect the real world physics. That’s all called the sim-to-real gap. We really haven’t figured out how to solve the sim-to-real gap. Deformable objects are even worse—just determining the transition function from S_t to S_{t+1}. If you condition further on the action, then it really doesn’t work. You need a lot of data to support that action conditioning. If you’re trying to estimate the dynamics function, T of S_{t+1} conditioned upon both, you have to figure out some coarse representation.

Nico’s actually here—we worked in 2013 on feature pyramid networks for our little robotic policy, bot war simulation way back when. But still, that was to solve this giant Q matrix where we’d have some linear thing with the pyramids. Representation for your action space is actually a completely unsolved thing if you want to learn quickly. This is the most important one that I don’t think is talked about enough: the sensory-motor issue. We have these nerve endings that can do so much. We can detect the normal force. We can detect the tangent force. We can detect moisture, temperature, vibration. We can estimate the coefficient of friction. It’s everywhere, all over our bodies. These robots don’t have that. They have one little coin FT on each fingertip and that’s all you get in the best case.

Maybe you have a wrist camera here and that’s kind of the state of the art. If you talk to neuroscientists about this, it’s actually incredible how good we are at building world models without vision. If you’ve ever tried to find your charger in your backpack and you put your hand in your backpack and you’re feeling around, you can tell exactly what’s in your backpack just by feeling around. You don’t even need eyes. There’s no way we have robots that can do that now because we don’t have an epidermis. I think that is a really important thing. If you’ve ever tried to tie your hockey skates when it’s really, really cold out, you start to see the policy fail. You’re like, I can’t even untie my skates because my hands are too cold.

And the last one that really only robotics people will understand—people that have deployed real robotics for long periods of time—is this embodiment drift. In this state, if I take this action, this is how much force is actually going to be applied. The actuators get dust in them. They get corroded. If they’re near the ocean, they may have some corrosion, some rust, and they just don’t work as well. This is especially true in self-driving cars; these are very real issues. When I push the gas on my Toyota Prius, it’s variable how much I’m going to get, and that shifts over time. The amount of power you get out of a battery over time also shifts. So then you have to retrain your entire VLA because it’s not mapping anymore. The teleops data is almost stale and you have to recollect it.

These are very real challenges that we don’t have answers to—until tonight. We have some great speakers, some great talks tonight. We have Marcel, who is a PhD in Chelsea Finn’s lab, and he’s going to talk about some of the cool work that he did at Physical Intelligence. We have Milan Ganai, who’s a PhD student with Marco Pavone and Clark Barrett, currently working at Waymo. And then we have Tyler, who is doing his PhD with Jeannette Bohg and Karen Liu. And Nico, who’s one of my close friends from 2012 when I was doing my EE master’s, founded this cool company called Rerun. He’s going to talk about some of the practicalities of data and how you actually deploy these robotics. And then we have Bill and Guanming, who are founders of this really cool YC company called General Instinct. They’re going to be talking about world action models, real-time world action models, and previous research experience at DeepMind.

It’s a great lineup. Thank you very much. Please welcome some of our speakers here. Marcel, you want to come on.

Marcel: I’m Marcel. I’m a PhD at Stanford. Today I’m very excited to present to you some of the work that I did during my internship at Physical Intelligence. We call this system MAM, which is multi-scale embodied memory. I want to start with some of the policies that we trained when I was doing my internship there at Physical Intelligence. We’re trying to solve these robot Olympic tasks. As you can see, they are actually quite dexterous tasks. You can see the robots—this is all fully autonomous—you can see the robots unlocking locks, folding clothes that were inside out, making a peanut butter sandwich. It’s actually very impressive to me. They are super dexterous and all. But if you see, actually, the longest of these tasks is about two minutes long. Ideally, when I think about what I would want my robots to do, I would want them, for example, to be able to manufacture something, to be able to clean a full bedroom, which is going to involve making the bed, folding clothes—a super long task.

Or for example, cooking a full meal, including cleaning the kitchen and everything. When we think about what we need in order to solve these very long horizon tasks in robotics today, I can think of a few things, such as being able to keep track of task progress, being able to keep track of time, and also having very reliable dexterity—dexterity that can actually adapt in context in case the robot finds a new scenario or makes a mistake. It should be able to adapt to it. And probably a few more things. But my claim is that in order to obtain all of these insights, we actually need to add memory into our policies. Memory, for me, is a very necessary thing for solving these long horizon tasks. However, if we look at most policies like π0, Five, Groot, all of these policies actually don’t have any memory, which means that basically at every time step, the robot will obtain a new set of observations and has absolutely no context of what happened before.

Here I can show a couple of examples of what happens when you train these policies without any memory. On the left, you’re going to see a robot that is washing the dishes and has no context of how long it has been washing the dishes, so it just keeps washing forever. On the right, you have one where the robot is cooking a grilled cheese, but again, it has no context of how long it has been there and it becomes fully burned. Then you might ask, why are we not adding memory into the policies? One reason is that it is hard. I’m going to take a small detour to another paper that we wrote last year where we basically observed that there are two main problems. One is effectiveness: when we add memory into these policies, they actually perform a little bit worse because of distribution shifts and lack of data. The other is an efficiency problem, which basically means that when we increase the context for the robot policies, this actually becomes much more resource intensive.

Then you’re going to get much longer training time and it’s also going to take much longer to run inference. Having said this, what do we propose and what is our solution? We propose compression. We basically take this model where we’re going to decompose our robot policies into two parts. One is a high level policy that tells the low level policy what the next step should be. Then we have a VLA, a low level policy that is actually going to execute the robot actions. We’re going to decompose the memory into two different types. One is going to be short context, which is some dense frames that are needed for the actual dexterous manipulation and is going to be fed into the low level policy.

And then we’re going to have a long context memory. There’s going to be basically a compressed language representation of the last few minutes, and it’s going to go into the high level policy. I don’t want to go too much into details, but I’ll give an overview of how we add this short-term visual memory. Our idea here was to design a new encoder that is based on the VIT, but instead of just taking a single frame, we’re also going to add some attention temporal layers. We’re going to drop all of the tokens except the current image, which should have all of the information necessary there because of this temporal attention. Then you basically get a lot of compression. Some of the reasons why this architecture is actually quite good are because, first, you have an easy VIT initialization since only the temporal attention is new.

We actually get to compress this image sequence again—we reduce the number of tokens, and because of this, you get fast inference. Now I want to show you what tasks we can actually solve with this short-term dense memory. Here we have our main flagship task that we did when I was at Physical Intelligence, and it was making these grill sheets. It’s going to be able to do all of the dexterous parts, but also be able to wait for as long as needed in order for the grill sheets to not be burned. But we also can solve some other types of tasks, such as here, that maybe seem like they wouldn’t need memory, such as un groceries from a grocery bag. But here you actually only see the items inside with a wrist camera sometimes.

So you actually need to remember where the items are and how many items there are. Another example is cleaning a window, where again, you need memory in order to not stay there forever. Now I talked about how to add memory into the low-level policy. Next I will talk about how we add the longer-term memory into the high-level policy. This is the technique that we propose, which basically consists of the high-level policy predicting a recurrent memory—like a memory scratchpad, basically. It’s going to keep track of everything that has happened. Then whatever its prediction was, it’s going to be fed again into the high-level policy. In this way, it can keep explaining what has happened before and what it can remember also what happened. This is actually a much more compressed representation than images, since text uses way fewer tokens than images.

So this is great for training. It is also quite physically accurate because the policy is going to be able to modify its memory with whatever has happened. And it is also less prone to distribution shifts. Let me show you here, what can we actually solve with this long-term memory? Here we show a task that takes up to tens of minutes and it actually is going to prepare all of the items for preparing a recipe. Just for the sake of time, I’ll skip over it. Something nice is that you can actually see the memory string that is predicted in the green box. We also compare with a bunch of different baselines such as no memory, different types of memory. We also get to beat here the state of the art. Something that I’m especially excited about with regards to memory is that actually when we add memory into VLAs, we’re going to be able to get this property of in-context adaptation.

It’s actually something that is really lacking right now in VLAs and I think it can be super promising in the future. Let me show you what this corresponds to. Here we have some policies, no memory. They have no memory at all. You’re going to see that they keep making the same mistake over and over and they are incapable of reacting to the mistake they did. It’s just going to be stuck in a loop forever and never going to be able to adapt. Here it’s trying to open the fridge or picking this chopstick forever. But actually now when we add memory, we’re going to see that the robot policies are going to make a mistake the first time, right? Maybe they have some bad priors or something, but they are going to actually be able to see this mistake and react to it.

For the chopstick, it made the mistake, but then it’s going to go lower to pick it up. For opening the fridge similarly, it’s going to switch sides to correct its mistake. I think this is something that is very lacking now from the robot policies that I’m super excited about for the future with VLAs with memory. Having said this, this was a huge effort at Physical Intelligence with a bunch of collaborators I’m super thankful for. I just wanted to point out Karl, who was the co-first author here, but then Homer, Sergey, Chelsea, and Danny for all their help. Please let me know if you have any questions. Audience: Thank you for the presentation. I was curious about the long-term memory that you were talking about. If it’s only represented in the textual space, how did you figure out the right information to actually give to the policy?

Marcel: Yeah, that’s an awesome question. So right here we train our high-level policies with SFT. We had to annotate all of our data, but that’s a very good point because basically we had to think beforehand what information we think is important here. We had to tell our annotators, okay, you need to keep track of all of these things because this is what is important. But I think that some of the follow-up works that I’m thinking about, and I think everyone should think about, is could you, for example, do reinforcement learning on this memory space in order to be able to know what is the right information to keep track of. But at least the first proof of concept was with SFT.

Audience: Exactly. I think that was going to be my suggestion too. Figuring out the right message to store, the memory, the right things to recall.

Audience: A follow-up question to that was how are you actually right now storing that information? It’s represented in textual space, but how is it actually used in inference time when you actually use the policy to take actions?

Marcel: So it’s a fairly short text. This can all be kept in RAM and then you just feed it as normal text tokens to the VLA.

Audience: So one question. When you extract your task into a high-level text description, how do you make sure that the text itself is generalized enough to handle different variants? When you make an omelet, there are different versions of omelet, right? And also how does this memory affect the number of episodes you need to train for a novel task?

Marcel: Great point. So here I don’t show the examples, but we actually give some quite detailed descriptions of the task. For example, for preparing the ingredients or for making a recipe and everything, you tell exactly where the items are, like what are the exact items for making a pizza, for example, and all these. So that is very detailed. Something good about this is that the high-level policy is a VLM trained with internet data. So it actually doesn’t need that much data in order to generalize well. And the good thing is that the VLA is completely separate and doesn’t need to—we don’t need to train it per task to be able to do things in the kitchen and all this. But if you make the task more complicated, the VLA is still only receiving a small text description of what it should do next.

So at least we don’t have to collect that much robot data with the complexity.

Audience: Really appreciate the talk. So your embedding for the memory is textual descriptions, but it also seems like you could have solved the problem with just adding, for example, for doing a grilled cheese, just adding time. And so it seems that training policy only based on textual descriptions limits the representation space for what that memory can describe and potentially constrains it. Have you thought about actually expanding the memory representation or looking at other approaches for that?

Marcel: I try to think about how humans even keep track of all their memory. And it’s definitely not in text space. But I think something that is quite hard and I feel we haven’t really managed to do it very well. Ideally we would have a latent embedding that would just keep track of all the memory. But just with text, you can actually put a very strong bias there and be able to supervise it, be able to debug it and everything, which just makes it right now the most practical way to do it. But yeah, I’m very excited to try to explore this further and can do some latent, for example, work there.

Francois: Right. Thank you, Marcel.

Marcel: Thank you.

Milan: I’m a PhD student at Stanford, and I’ve done research at AWS and Waymo. And talking about how can we move toward robots that teach themselves how to reason? You’re probably familiar with vision-language-action models, but just a quick primer. VLAs are a powerful class of generalized policies. You’ve probably seen them in demos for manipulation like RT-2 and Pi or even self-driving. Maybe you’ve been on a Waymo or Alpamayo, which is from NVIDIA. So how exactly are VLAs trained? You take a vision-language model. These are multimodal models trained with an internet scale of visual and textual priors. And then this can be like Gemini or Qwen and you continue training them on relatively scarce robotics data sets. This can be tele-op for manipulation or maybe self-driving. Someone has driven a car and recorded that. And then you end up with a VLA, which takes as input an image or some perception feature, maybe language instruction prompts, and learns to generate actions which can be executed.

So, steering commands or end effector position. Now there’s this recent trend of leveraging and doing embodied reasoning for better action prediction. The idea is similar to chain of thought in the LLM domain, where you go from question to answer by explicitly providing logical steps. Why are we interested in embodied reasoning in the form of chain of thought? The idea is that there’s not a lot of data in robotics—it’s data scarce. Any form of signal that you can use to augment your dataset is very valuable. You can start injecting different types of annotations, and that can help you improve action generation and related tasks. You get a richer training signal with reasoning, but also, because this is in text, as a human, you can go and read it. If you’re trying to decipher the reason why a robot made a particular decision, you can actually read through the chain of thought trace.

To this end, I’ll be talking about our recent work published at the Robotic Science and Systems Conference this year called Self-Supervised Bootstrapping of Action Predictive Embodied Reasoning. We’re really interested in this question of what should we be reasoning about. Specifically, what should particular embodiments and form factors reason about? What makes good embodied reasoning is a hard question, but I’ll decompose it into two problems. One is the grounding problem. We don’t have an oracle source of reasoning data saying, “Just train on this reasoning data and you’ll get good reasoning.” It’s not clear what we should be reasoning about. If there existed some traces on the internet, just like there were pre-training documents for text or captions for images, that doesn’t exist for robotics. Second is, where’s the oracle source of the model? You can’t just open a human’s brain and figure out how they made a decision from an image to the movements in their fingertips.

That’s why I like to show this image of a chicken and egg problem, which is: where’s the source of the model and the oracle source of the data? The second issue is associated with verbosity. There are different types of reasoning. You can be planning, for example: “Go to the pepper, pick up the pepper,” and so on. Maybe you would be reasoning in the form of perceptual traces. This can look like visible object lists—a list of bounding boxes of all the objects in your scene—or even gripper position, such as the position of the end effector or the position of your car on the road. But the question is, should we be planning at every step, or is that too verbose? Latency is a big problem in robotics. It’s not clear if we should be planning or analyzing every single object in the scene, or if that is distracting and could mislead us in action prediction.

And similarly, is gripper position reasoning action predictive or is it misleading? All of these questions can be summarized into this one research question: how does textual reasoning for specific form factors and embodiments look? That’s where our approach comes in. It’s called R&B Encore, which is short for refine and bootstrap embodiment-specific chain-of-thought reasoning. This is a self-improving pre-training cycle for embodied reasoning VLAs. The insight is that, because we are claiming that reasoning is this black box for robotics, we treat this as an unobserved latent variable for the observed context and action. By doing so, we can leverage a theoretical framework called variational inference.

How does this look? There are two high-level components. One is a reasoning proposer. You can think of this as an annotator model, which proposes, for a given context and demonstration, various types of proposed reasoning. It’s asking the question: is visible object and move reasoning a useful type of reasoning? Or should you be reasoning about plans and gripper position? Or maybe visible object and sub-task reasoning? What is the sort of reasoning, annotations, or good data that you should be producing before your actions?

The second component is a reasoning validator. The idea is that this is a scoring metric based on the theoretical ground of variational inference. There’s all this theory that we’ve proved in the paper, but to concisely summarize it, there are three main parts to the score. One is concision, ensuring that your reasoning trace is short and not too verbose. The second is non-triviality, which encourages generalization in the reasoning behavior. The third, and most important, is action predictiveness. This ensures whether that reasoning trace or annotation is grounded in that embodiment.

At the end of the day, you score all these reasoning traces, re-sample, and end up with a new dataset of synthetic but action-aligned and embodiment-aligned reasoning. You can analyze this reasoning data to understand which types of reasoning are good and which are not. More importantly, you can retrain your embodied reasoning VLA and have a better and more robust policy.

We tried our pre-training cycle on a bunch of embodiments. We looked into manipulation, so we pre-trained embodied reasoning manipulation VLAs. We found that move and gripper positioning type of reasoning is very useful, whereas perceptual reasoning is not very useful and can be pruned out. We improved success rate because we have this action-aligned form of reasoning. We also looked more into the question of why perceptual reasoning is not useful.

We find that there are a lot of distracting objects in many of these scenes. By pruning out using our pre-training cycle, we can actually look into what is a good way to improve your annotation data quality. Questions about improving and task saliency of your traces and the data that you use to collect and annotate are important. This not only improves success rate, but also the object criticality rate or task saliency rate. We also test this entire pre-training cycle for hardware manipulation relays, and we’re able to have improved out-of-distribution performance, particularly on novel target objects or even cluttered scenes. We also pre-train legged locomotion navigation models. We find that reasoning about structural affordances and movements is way more important than reasoning about terrains and counterfactuals. We also use our cycle for refining human annotations for self-driving.

There is a lot of garbage data that could be out there. The idea is that with our approach, you can remove a lot of the human annotations that are not very useful. We observe that meta action and visible objects and perceptual reasoning are useful, whereas our approach can prune out hallucinated experiences, thereby lowering the L2 path errors and performance metrics like collision rates. Takeaways: selective reasoning is way more important than exhaustive reasoning. Even if that reasoning is valid, it’s not necessarily useful. Second, self-supervised bootstrapping works. This is really important to address the chicken and egg problem of the Oracle source of model and reasoning data. Third, we show the approach generalizes across embodiments like manipulation, navigation, driving, as well as VLA sizes from one billion to 30 billion parameter models.

We are basically addressing this question: how should an embodied agent reason? We argue that embodied reasoning is not a fixed template that should be applied uniformly, but rather a resource that should be discovered, deployed, and budgeted carefully. We are very excited about this problem, about how we can deploy embodied intelligence. We’re very excited about problems like data quality, how we can recover from failures, how we adapt to novel scenes and specialize in them. If you’re interested in some of these questions, feel free to reach out. We’re excited for collaborations. We would like to help you on problems of reducing the friction of deploying embodied intelligence for everyday tasks. If you’re interested in reaching out to us, you can scan the QR code on the right and find more details about R&B Encore, including the website, models, code, and papers on the left.

Francois: How do you think about the conversation we had where a dog is figuring out all of this, but not in token space? How do you think about this without using tokens?

Milan: Okay. So I think there are all sorts of latent reasoning capabilities where the model can navigate in its continuous space of understanding, unhindered by the textual format. This requires good architecture and theory for developing this kind of continuous reasoning. There are ways of, if I’m understanding your question correctly, non-textual reasoning. I think some of these approaches can really help with that. The idea here is that we want to leverage particular priors that exist, especially with LLMs and BLMs where there’s huge amounts of data out there. We want to use those priors for data-scarce regimes like robotics.

Audience: I wanted to understand what’s the thesis behind using reasoning for autonomous vehicles because there’s already the streaming latency problem, which is even more exacerbated in the case of AV. In that case, is your argument just that it would help build better self-supervised data or is there something more there that can be leveraged for better

Milan: Performance?

Audience: Yeah,

Milan: Absolutely. Latency is a big question, which we actually address in this paper. Our main claim here is that reasoning can help introduce priors into the model. You can think of this almost like co-training data, where instead of just training on robotics demonstrations, you can co-train and use textual reasoning as good annotations to help it. We actually show that we have this approach called action forcing, where you can drop the reasoning during inference time. So you don’t have the inference latency problem, but at the same time, you can still leverage or extract the enhancements from the textual annotations and reasoning. You can get the best of both worlds.

Audience: And a follow up to that is, do you see drift in terms of state space? So you reason about something, the state has moved now. So the reasoning is no longer relevant or can damage future actions that the model might take. Do you see situations like that as well?

Milan: The reasoning happens assuming it’s happening in real time. It’s based on the current image and the current instruction. So the drift isn’t necessarily a problem that’s going to fundamentally appear here.

Audience: Thank you so much. I just had a minor point. When you were talking about pruning things that weren’t too useful for the model, you made a point on how counterfactuals weren’t too useful in terms of better reasoning, if I’m right. Could you maybe just explain a little bit more as to exactly where you were going with that?

Milan: Yeah. So the claims that are being made about this are focused on the particular benchmark that we were working on. It isn’t to say particular types of reasoning shouldn’t ever be done and should be removed, but it is to say that there are certain types of reasoning that don’t need to happen at every single moment. To give you an example, if there’s nothing novel or interesting happening in the scene, a lot of self-driving is just driving in a straight line and there’s nothing around you. So there’s no interesting counterfactuals that you might explore. But even plan reasoning, or as I had mentioned earlier, plan reasoning appears to be quite useful. It seems that it might be useful to plan ahead, but you don’t want to be planning at every single step because it’s quite redundant.

Francois: Thank you, Milan.

Milan: Thank you.

Tyler: Hi everyone. My name’s Tyler Lum and I’m excited to present our work SimToolReal. Let me start with what SimToolReal can do. Every clip here is at 1X speed. This is a single policy that is working zero shot, meaning it never saw any of these tools or tasks during training. We do not need to retrain for a brush or hammer or a new target behavior. Many tasks like the screwdriver spinning are very dextrous. They would really require a multi-fingered hand. They just wouldn’t be possible with a parallel jaw gripper. The policy is very fast and reactive. It runs at 60 hertz and simultaneously controls both the 22 degree of freedom hand and seven degree of freedom arm. I also want to highlight that the level of dexterity is very difficult to demonstrate through teleoperation here. With that preview, I want to give a little bit of context.

Teleoperation for dextrous hands has increasingly been used to collect demonstrations for imitation learning, but highly dextrous actions remain difficult to demonstrate reliably and at scale. In the video on the right, even this simple in-hand rotation task requires slow, deliberate control from the human because of the embodiment mismatch and limited force feedback, which make precise contact regulation very difficult. Rather than learning the policy from teleoperation, we train it entirely in simulation using sim-to-real reinforcement learning. Sim-to-real reinforcement learning uses GPU-accelerated simulation to run tens of thousands of robots in parallel and generate experience about a thousand times faster than real time. This allows us to scale data collection with compute rather than human effort and collect decades of interaction data in only a few days. The result is extreme dexterity because it is not just imitating demonstrations, but it’s optimizing for reward-maximizing behaviors.

These policies are then deployed in the real world, demonstrating impressive dexterous behaviors that would be very difficult to teleoperate. These videos are from Dextreme, which is one of the first works to demonstrate the effectiveness of this approach for dexterous manipulation. This task isn’t particularly useful looking, but it’s definitely an impressive demonstration of dexterity. Most prior works in this space learn a separate policy for each particular skill—one for grasping, one for reorientation, another for object spinning, and another for tool use. Each new behavior typically requires a round of new reward design, new task-specific engineering, and another round of training. We instead ask, can we train a single policy just once and have it perform all of the different tasks and skills? What we really want is a single policy that controls both the hand and arm through the full sequence.

It first grasps the brush, reorients it within the hand, and finally uses it to sweep the objects. We want this all from a single policy so we don’t need to do any manual switching between separate policies. Our key insight is that we can unify dexterous tool manipulation as goal reaching. The policy doesn’t need a task label such as sweeping or hammering. It only needs to move the object from its current pose to the desired pose. Thus, we train a goal-conditioned policy that can move arbitrary objects through a sequence of desired goal poses, visualized here in green. We find that this is a very general objective for a wide range of manipulation tasks. At train time, we procedurally generate primitive objects in simulation, sample random goals, and train a goal-conditioned policy with massively parallel RL in simulation to manipulate random objects to these random goals.

Of course, there are many details to get right here: system identification, domain randomization, and algorithm details to get exploration right. We’re going to skip all of that for now, but feel free to ask questions in the Q&A. At inference time, we need to specify the sequence of goal poses, which can come from any source. In this work, we choose to condition the policy on a human video demonstration. We use FoundationPose and SAM to extract the sequence of goal poses to track. The RL policy tracks these goals one by one in a 60 Hertz control loop. I want to highlight here that the human video is not providing robot actions and it’s not used to train or fine-tune the policy. It only specifies the desired object trajectory for the frozen policy to track. Concretely, the policy takes in proprioception, the current object pose, a bounding box of where it should be grasped, and the current goal pose.

It runs them through an LSTM policy and then outputs joint position targets for the full hand and arm. This policy, again, is not limited to just sweeping with brushes. This single policy works zero shot across novel tools and tasks never seen during training. What’s really nice about this is that a new task simply becomes a new sequence of goal poses rather than a new training run. Conceptually, the trajectory acts like a task prompt that we can provide at inference time for a frozen policy, allowing us to perform a new task on a new object in minutes instead of hours or days. We evaluate the same frozen policy across 12 unseen tools and target behaviors and achieve substantial task progress across every tool family, particularly those with long handles. Performance is weaker on heavier tools, which are easier to drop, and also for smaller objects.

But this is really because the pose tracker has a lot of problems when it gets very occluded. Next, we want to measure train and test correlation. Our goal is to see how well our training objective supports downstream test success. On the left, we show training objects consisting of primitive objects and random goals. On the right, we show our test objects with human demonstrated trajectories. We evaluate checkpoints throughout training. As the policy improves on the generic goal reaching task with primitive objects shown on the left, performance on unseen tools and human demonstrated trajectories on the right sharply rises. This validates that SimToolReal’s training objective with random objects and random goals is effective for generalizing to real world tools and tasks. Next, we compare SimToolReal against two really common baselines. Our method successfully grasps and reorients the tool into a functional pose to complete the task.

The fixed grasp baseline can grasp the object, but it must rotate it using only the arm, resulting in a table collision. This really highlights the importance of in-hand reorientation ability, as most prior approaches assume that just acquiring and maintaining a fixed grasp is sufficient, but it really isn’t for many cases. Lastly, kinematic retargeting: this is where we try to imitate the human video demonstration by transferring the human fingertip motion to the robot, but it doesn’t reason about contact forces, so it fails to even grasp the object. Next, we analyze the failure modes of SimToolReal and find that pose tracking failures dominate. Next, object dropped after being grasped. Lastly, failed grasp, but it really tries to chase it down. The policy demonstrates really strong recovery behaviors. Here, when the robot drops a hammer, it immediately re-grasps it and completes the task.

Lastly, please check out our website. All of our code assets and policy weights are open sourced. We even have an interactive demo that runs right in your browser and works on your phone. It’s not running on a separate server—it’s running on your phone. It will drain your battery, so don’t leave it running for a long time, but it’s pretty fun. That was a quick overview of SimToolReal. I want to spend the last couple of minutes talking about a follow-up work called Play2Perfect. Many real tasks like precise assembly require sustained contact and millimeter-level precision, but learning the skills for precise assembly from scratch is really difficult. We argue that before we can learn the hard problem of precise assembly, we must first learn the easier problem of playing with objects in free space. This motivates Play2Perfect, a framework that leverages the familiar pre-training fine-tuning paradigm.

We first learn a shared dexterous prior through task-agnostic play, which is very similar to the SimToolReal task-agnostic training. We then fine-tune that prior on a sparse reward contact-rich assembly task and deploy the policy zero-shot in the real world. This enables diverse contact-rich assembly behaviors, including tight insertion and multi-part assembly.

In conclusion, SimToolReal enables broad reactive dexterity across novel tools and tasks, while Play2Perfect extends this towards precise contact-rich assembly. Thank you.

Audience: I noticed that in the demos, the recovery was insanely impressive to say the least. Is that something you all specifically aimed for, for the robot to have good recovery, or was that just an accidental byproduct of the policy?

Tyler: That’s a really good question. We didn’t specifically train for that, but one thing that we importantly added—one of the details about domain randomization we added—was that in simulation, we added these random forces on the object that randomly pushed at it. Sometimes it would knock it out of the hand, so it gets experience having to pick it back up again. If we didn’t add that, I think it may not be as good at recovering because it might never drop it in simulation.

Audience: How similar did y’all ensure the primitive tools you used were to the normal brushes and spatulas you use?

Tyler: Great question. It’s called SimToolReal because we’re trying to focus on tool use. Many tools have some sort of graspable region that can be somewhat approximated with a bounding box. That’s our interface here that we’re using. We’re not giving it detailed object geometry; we’re just telling it roughly what the bounding box size is of the graspable region. In simulation, one detail here is that we’re using all primitive objects, just cylinders and cuboids. We could extend it to more. Then you have to generate a whole other dataset and remove stuff that’s not stable. But the key advantage is that the simulation runs two to three times faster when you use simple objects like that. That’s what we use here—things that can be reasonably approximated as a cuboid. Something like a spatula or even a sphere would probably be fine.

But if you’re trying to pick up something like a bowl or maybe scissors, it probably could pick it up in some weird way, but not in the way you’d want it to.

Audience: I’m super interested in this kind of novel approach rather than learning from human demonstrations. Basically, you learn a bunch of sub-goals and then use RL to reach those sub-goals. But one problem for this, I think, is probably how would you be able to generalize to a lot of real-world objects other than tools? Because for tools, you can easily define those primitives. Have you thought about those? For example, articulated objects—how would you be able to do those?

Tyler: Simulation has its pros and cons. One answer would be we can maybe just simulate all of them—simulate scissors and simulate all kinds of other articulated tools. But there are just so many things that cannot be simulated well, like water or even a zipper or cracking open a can of bubbly or something. That kind of stuff is not currently in the realm of simulation. What we do there is a really good question. Can we transfer these priors into the real world and keep fine-tuning it? I think that’s most likely the way I would do it, but that’s a really good question—how to integrate this kind of great dextrous behavior, but let it keep improving with real-world experience.

Audience: I think my question is more on—even if you assume you can simulate all the objects in the world, but with this paradigm, are you able to generalize? So

Tyler: Just still

Audience: RL?

Tyler: That’s a really good question. I think if you take a really far step back, our policy is being given the current pose and the goal pose. And it’s kind of like an inverse dynamics model in pose space. But if the object can’t be specified as pose, maybe if it’s something like a towel, maybe you can specify it with key points. Or maybe the most general version of it is you have a video goal and you’re almost an inverse dynamics model of the current state and the final goal state. It’s probably possible to train something like that, but exactly how to get those details right—generating the goal at inference time is a really hard problem. But I think your question’s a really good one. How to make that more general.

Audience: Yeah. We’re trying to basically reproduce this. Oh, cool. Cool.

Audience: Yeah. Hey, I’m trying to interact with that one right now. And I think I’ve been trying to poke around and find failure modes. I found that the only failure mode that consistently shows up is when the arm continuously twists, trying to find different objectives. It fails usually when it twists the arm itself too many times. I was wondering if you have a reset dynamic maybe between actions where, let’s say my arm is here, and in order to reach this state, I could turn it backwards instead of trying to twist it even further.

Tyler: Yeah. Good question. I’m honestly not sure exactly what the right solution is there. In theory, if RL’s really good, it should know to spin it all the way around again. And it gets really contorted. I don’t know the exact solution to that. I guess in most real world applications, you don’t go crazy contorting your arm, but I think it’s a good question.

Audience: Cool. Thank you.

Audience: Okay. First of all, this is amazing. Congratulations. My question is, from the team’s perspective, how do you guys iterate on such a thing? For example, if you have one policy, you start seeing regressions in other tasks. How do you guys handle that? Do you have evals or what does iterating even look like in this space?

Tyler: Good question. It’s a bunch of things. How do you eval these kinds of policies? You can eval in the real world, but it’s very expensive and takes a lot of time. And a lot of times you don’t actually get statistically significant numbers. You almost get a vibe check unless you run it truly a hundred trials. It’s hard to tell if you’re at 90% or 87% success rate. The other thing, one thing we really try to do is have a lot of automated ways to evaluate our policy across all those novel tools and tasks in simulation. So it gives us a sense—we train on the random objects and random goals and see how well it works on our real world objects and real world trajectories, but we put them in simulation. So we get some metric, a feel for if it’s doing way worse or way better.

But it’s a good question because honestly it’s really hard to tell. Sometimes it can do better in SIM, but it’s doing some magical tossing and catching behavior that may work, but our pow tracker would probably fail. You need a little bit of that human insight still. It’s a really, really hard one.

Francois: I have a quick question. So you use this prehistoric thing called an LSTM. Can you tell us about that? What is that?

Tyler: Yeah. It’s a long short-term memory. I can go into the math, but basically RL packages are—how do I say it—RL is a very finicky thing where you almost don’t want to change too much of the code because once it works well once, you don’t want to rewrite it from scratch because any one value you change could break the whole thing. It’s really good to start from a good starting place and then iterate from there. That code base already had LSTMs baked in. We were like, let’s just try it out and it worked better. There’s probably other ways to integrate something.

Francois: Better. LSTMs work better than transformers. Well,

Tyler: There’s no transformer in that one.

Francois: Yeah. But do you think that transformers would have worked better?

Tyler: I’ve talked to people about this. I think that broadly, transformers are probably better data sponges if you have unlimited data. But here I feel like we’re getting a lot of data, but we’re constantly updating the policy. I feel like it’s actually not in that regime where we need an enormous, huge data set. Some people have actually tested this and not shown any improvement. I haven’t had the motivation. The coding agents are out now, so probably no excuse for me to not try it.

Francois: The diffusion policy paper, a lot of people don’t know this, but if you go through their table, the conv actually outperforms the transformer on half the policies.

Tyler: Yeah. Unless you tune the transformer better, it’s actually much more sensitive. I think exactly that.

Francois: And then the last question, I didn’t really understand how you create random goals. What does that mean actually in the code? What are you actually

Tyler: Doing? Oh yeah. It’s the simplest thing you can imagine. We sample a position, we sample a rotation, and then we put that as their first goal. Every subsequent goal is some delta pose of up to 10 centimeters away and up to 90 degrees difference.

Francois: I see. So that’s really good for training the policy to get from A to B, but it’s not good for generating the goal. So to generate the goal, you need human labels.

Tyler: Exactly. Or you could do something else. You could actually imagine there’s some high level planner looking at the scene, understanding the full context, and then generating the goals for you. I think that’s probably a pretty interesting direction.

Audience: You were mentioning some of the failures were due to wrong estimation of the pose.

Tyler: Oh yeah.

Audience: What fraction of those would be attributable to that? And second question is, are you always using the third person views or did you also perform any experiments with

Tyler: First

Audience: Person views? Oh,

Tyler: Good question. Yeah. About more than half of our failures, I think roughly 60% of our failures were purely from the pose tracking. It was really one of the big bottlenecks of the system. It’s honestly not my favorite part, but it really happened the most on these smaller objects. So that marker, that one is really easy to be occluded. You barely see it at some points. I think that’s an example of one that would be really hard. This kind of longer leg or this bigger brush has a lot more features, so those would have less pose tracking issues. That’s roughly how that looks.

Audience: Were you using the third person view or videos

Tyler: Experiment with first

Audience: Personnel? It

Tyler: Was a third person view roughly where this camera is. And that’s pretty much all we tried. We just found a good angle where it didn’t seem to occlude, and then we just used the pose tracking from there. I think it could be better. Maybe you could even use a pose tracker from the camera. You probably can distill this to some image-based policy, but there are some details about getting the goal conditioning right.

Francois: Awesome. Thank you, Darren.

Tyler: Thank you.

Nico: Yeah, I’m Nico. I’m one of the co-founders and the CEO of Rerun. In a prior life, I used to do machine learning, computer vision for physical world applications, for shipping products like that for about a decade before this company. At Rerun, we are building this unified data layer for physical AI. Basically tools and infrastructure to help you work with physical data from collection all the way to training. This open source SDK is pretty popular. I know a bunch of people in here use it for working with physical data. It’s for logging, visualizing, generic querying, and data for training. Basically all the tools you need to transform and analyze data. We have an info product called Rerun Hub, which is a data catalog and large-scale data backend for doing all those same things, but for lots of data in the cloud.

As part of building this, we get to talk to and work with amazing companies doing robotics, from frontier labs all the way to two-person YC startups. There are many ways to do robotics companies and robotics projects. There’s one pattern that we’re seeing a lot of that’s really, really working right now that I’m super excited about, and I want to see way more of. This talk is actually mainly me trying to tell you all to start companies like this. Basically, that new category is what you might call just robotics application companies. I think some people call these the neo integrators. Their pattern is really taking ownership of a full business problem end to end. Right now, we see this working a lot in data center management and construction, warehouses, manufacturing, things like that. Just being really, really excellent at operations, deploying, support, things like this.

Building as absolutely little custom hardware as possible. Then, generally, often starting with teleop, making sure the business works with pure teleop, fine-tuning models. Not feeling the need to start out with foundation models and solve very general problems. My personal belief is that this kind of category of company is going to be the new SaaS. The way SaaS companies came and just took over software over the last, I don’t know, 10 years—until I guess SaaS is dead now—but this is not dead. This is the same thing that’s going to happen for work in the physical world. These kinds of companies are going to do a lot of the transformation of the world’s economy. I think it’s a very ripe time to get into it. To talk about that, I thought I’d just walk through a little bit.

If I wasn’t doing Rerun, how would I do it? This is the pattern that I see. How do you get started? Pretty simple pattern. Start with a single customer problem that someone will pay you for. Solve it with teleoperation first and just off-the-shelf hardware, just scrappy, get started. The basics you need for learning. With that in place, you just iterate more and more at scale, and then you’ll do the rest of your time with your company. But that’s the fun part. To me, I would clearly pick this very important problem. Everybody loves paper planes. Folding is super annoying. So I would make a robot for automating paper plane factories. First thing you do, right? Sell and deploy super fast. Ideally with teleop and off-the-shelf hardware, as I said. And the reason for this is basically the physical world is brutal. Everything that you do is going to break. You will not have thought of all the different failure modes up front. It is not possible to think of them all in the lab. It’s really important that you understand the end-to-end real business requirements super fast because you can’t fix all the theoretical things. If you can solve something with tele-op, generally you can train a model to do it. The converse is not always the case, as we heard about in a lot of examples. If you can, that’s great.

Just some examples of what we might learn doing this: maybe we learn that you need to produce a thousand perfect planes per day to be viable as a business. Maybe it’s okay to fail as long as we can sort out bad planes, so we need to be an efficient discriminator. Paper’s cheap. Customers care most, it turns out, about the speed to onboard new plane designs. If you add the right little paper tray, maybe you reduce failures by 50% because most of the failures were actually picking up the paper from a pile.

It turns out it takes a human 20 hours of practice to get good enough to meet a customer’s demand requirements. That has huge impacts on how you’re going to run operations. Maybe you need to hire all the teleoperators because they need training, for instance. You may also learn that it’s 10 times more valuable if your robot can also go pick up the paper and pack the boxes for shipping at the end. Then you have an idea of your V2 product. You’ll definitely learn that your arms are going to break. The cheap research arms that you bought are going to break after some use, and you need to change your supplier.

That’s part one. Part two is setting up the basics for learning. Hello World in this space is basically fine-tuning, let’s say, a Pi model, open model of some kind, just for the simplest possible case on a few hours of demonstration—tele-op demonstration—and just making sure that it somehow works a little bit.

You’re up on the treadmill. It’s really important to do this early as well because training on the data early will change how you collect and how you run operations. That’s super key. When you have that in place, you need to make sure that you can evaluate and understand performance, and then obviously collect the data that you can train well on.

Number one, evaluating performance: the first thing you need to do is have a replica of the customer’s environment in your own office. I’ve been in a lot of robotics companies’ offices. Among the companies who actually ship working products, I haven’t seen a single office that doesn’t have a replica of customer environments. You just need somewhere to test, and you need to test a lot. Number two is finding a repeatable way to evaluate success.

And this is where you are going to do—this is the backbone of all the learning that you’re going to do. Here you can really encode things that the generic model companies will not do. You’re going to encode what is important to this business, and that you learn on the ground with your customers. There’s a lot of stickiness in that. In this case, maybe we care a lot that the edges on the planes are sharp. We care that it matches the design, it’s a metric, maybe the weight distribution is right. I don’t know. It’s super important to do this yourself manually until you really understand it, and it’s kind of stabilizing. Definitely automate it somehow, train a model, outsource it, but do it manually first. Then you need to be tracking metadata of all the rollouts, failure classifications, that kind of thing.

Second thing is collecting data that is effective to train on. It’s almost tautological, but good data is data that makes the model better. What that means in practice for you is that you need to be training and evaluating and debugging your data constantly. If I talk to researchers at big robotics companies with huge budgets and so on, they’ll often tell me that one of the most common things they’ll do when they’re debugging their policy or analyzing their data is they actually find out that the right thing to do is to send a different instruction to their data collectors to collect data differently or to stop doing some mistake. Doing this early is important. You don’t want to be collecting all your data upfront and then train later. Huge mistake. There’s lots of literature and expertise on what kind of data you want.

You want the right kind of variability, no bugs. That’s a deep one. But the most important thing isn’t the specific ways of doing it. It’s that you are testing and iterating really fast and getting your hands on real problems. To do all this, you need the right data and formats, tools to work with your data through all this. You need to be able to record and store and inspect your data and obviously train on it. Just a couple smaller examples: one could be this—your customer said that they care a lot about quickly onboarding new designs. That means you have to have some strategy to be a little bit more sample efficient. Very commonly, it’s more practical at these companies; what they’ll do then is split this task into more composable sub-tasks.

So then you need to design the taxonomy, figure out how you want to annotate this efficiently and repeatably. And you’re now in a situation where your annotation is too complex to be doing live. Maybe in a simpler case, you could actually have the operator just speak or have a little foot pedal or something like that to do annotation, but now you can’t do that. That changes your operations. Second, on the data tooling, you get into why not just use Postgres or whatever data infrastructure was built for the prior generation to do LLMs or feature stores for prior ML and so on. The answer is basically that physical data—all the data that you’re going to be working with in robotics—is just very, very different than web data. It’s multimodal. It’s multi-rate. It’s episodic. It has this weird semantics of 3D and deep nested structures.

That means if you try to put that kind of data in normal data systems, like table-based databases, it’s incredibly hard to query and very inefficient to store and process. This is really at the heart of a lot of the complexity of working with physical data. There’s a lot to say about that, but the end effect is that most teams, if you don’t set the right storage layer at the bottom, end up building a lot of workarounds, and that’s a huge amount of friction. But if you have these very basic things in place, you’re ready to hill climb. So you’re deploying and learning from real, valuable robotic service that’s doing something worthwhile. You know how to evaluate performance. You can improve to collect data and train on it well, and you can debug across the stack.

Super important. You don’t know where the problems are in robotics—real robotics. It’s a death by a thousand cuts kind of industry. You just have to find all the problems. After that, it’s just iterate and scale. This is what you’re doing the whole company. Super, super fun. Includes improving intelligence, scaling up crazy amounts of data perhaps if you need it, iterating on algorithms and more advanced use of data, maybe adding in Francois’ favorite with tactile or depth or sound. But you’re also really going to have to get excellent at sales and assembly of your robot, shipping it really fast, having a good unboxing experience, operation, support, everything in between. Importantly, these other areas that are not just modeling are a lot of the source of your moat. This is the kind of stuff that the pure model companies will not do.

Just on that, the thing that we see, even the absolute teams that we’ve seen really succeed—and there are some companies taking this approach that raised reasonably little, small amounts of capital that were already making a lot of money, doing very well, and growing super fast—is basically iterating super fast. On the frontier lab side, that tends to mean they’re investing huge amounts of compute for every researcher. Everybody knows they spend a lot on GPUs for model experiments, but also quite a big spend on CPU for really turning down latency on searching and exploring data. But these robotics application startups instead really focus on very pragmatic and simple, flexible systems to have very minimized moving parts. It’s super important to have fast turnaround with new data, full stack debugging, and making sure that you can understand all moving parts.

Yeah. This is my pitch to all of you. At least someone in here should go start a robotics application company. The market or markets are enormous. The base models will keep getting better. There is actually enough friction in the physical world to build real business moats. So it means you can stick around, which is great. And you can do this with a relatively small amount of capital. You will not need to raise a billion dollar seed. But you still need great AI, you need great engineering to win. That means all of you here and, I guess, people listening, you have a leg up and it will still be super fun. I’m going to do a quick plug or reiterate what Rerun does. If you’re building a company like this, definitely check out Rerun or talk to me.

As I said, we have an open source SDK. It’s meant for you to iterate super fast with robotics data. It has all the pieces you need. It works really well with agents if you want to make it work exactly like you like it. And a production catalog and storage engine to make it fast and easy to use when, at some point, you need to start scaling and you have a lot of this data for production or for training or whatever it is. All right. That’s me and you can find us here.

I think where we see a lot of early success tends to be in things that you can tell off basically. Data centers are quite a significant category, but a lot of warehouse robotics—there are so many pieces that go into just moving things around in the world. A lot of them are quite repeatable. Labor is fairly cheap, but it’s also hard to manage. And there’s too little labor out there. We’ve found with the companies that we work with and are generally known that can build a reliable robot, they basically are 100% supply constrained. They have a very easy time filling their demand. So that would be one area. Small scale manufacturing, like tabletop manufacturing of different kinds—we see a lot of action there. Food as well.

Francois: Aniko, thank you. Why haven’t there been a bunch of these robot application companies yet that have been at billions of revenue?

Nico: I think robotics is this death by a thousand cuts kind of thing. In any area like this, it matters that you can try out an application fairly cheaply. And that’s actually quite new. There are a lot more arms on the market now. The base models are way better now than two years ago. So there’s been this lack of the basics that you need to do this fairly cheaply. Everybody’s had to go out and raise huge rounds and go for much more general things to start with.

Audience: I think one dilemma one might face while building this kind of company is how to estimate the scale of data you would need to solve a particular problem before the model is deployable. How do you go about that? How do you estimate the scale of data you would need and whether it’s the right problem to solve or switch the problem so that we need less data to iterate faster?

Nico: Yeah. I think that’s really one of the core ideas between if you can teleop first. It’s not obvious that many businesses work without full autonomy. Even a lot of the robo taxi businesses don’t need full autonomy. A lot of these companies see autonomy as a scaling factor. So you teleop; generally if you can teleop, you’ll be able to learn the least important parts at some point. It just becomes a question—you learn that by training models and trying to plot your own scaling curves and so on. I don’t know that I know anything upfront, but that’s kind of the idea of the strategy. You don’t guess and avoid because it’s equally likely that the task you thought you needed to solve isn’t really the important task anyway.

Audience: Thanks.

Nico: All right. Thank you,

Francois: Nico.

Bill: We’re from General Instinct. My name’s Bill and then Guanming’s going to present later. I come from a technical background working on VLMs at the beginning. Worked at Siemens on their foundation model to train to predict time series. And then Guanming worked mostly on robotics RL. What our company does is we build infrastructure for you to run physical AI models really fast. For LLMs, you have vLLM and SGLang. For physical AI models like world action models and VLAs, you would have us, General Instinct. This is a meme from Jim Fan’s talk that VLAs are dead and then we’re going to have world action models from now on. Most of us in the room know what VLAs are already. I’m not going to try to explain it. Basically, you have a VLM that’s trying to predict an action through an action head.

What a world action model is, however, is you have a central diffusion transformer that’s trying to predict what the future looks like and future kinematics at the same time. So you have current observation from a robot’s camera in the form of video streams, and then you’re trying to imagine future frames as a condition to try to predict future kinematics. One example is NVIDIA’s DreamZero. You have basically the robot trying to predict future actions, and then you have the flow matching that allows the robot to act on those action chunks and future frames. And DreamZero did really well. So these are some benchmarks that you have on DreamZero compared to some of the state-of-the-art models—some models from Pi, some models also from NVIDIA. But one problem that we noticed is that although world action models perform really well, because you’re still trying to use a diffusion model to try to predict frames, it’s really heavy.

So even after all these optimizations that you can do on it, it still takes two GB200s to run the same model. And then each one costs around 70K. So economically for robotics as an industry, this is not scalable. Another thing, VLAs are not dead because they’re small.

Guanming: So since we know world action models like DreamZero are super slow, we will talk about how we can optimize it. So this is the architecture. On the left is a training pipeline. On the right is the inference pipeline. So for the training pipeline, you basically treat current observation as the condition for the flow matching, and you add noises to the future latents. And then you will train the model to learn the future latents and then predict the future velocity field. And then send it to the OD, then you can drift back to the future latents with clean states. That’s the same thing for the inference as well. For the inference, you’re doing this ultra aggressively for maybe 50 steps, or some people do 100 steps to ensure the accuracy of the models. And a problem for this will be on the left, you have the video prediction.

On the right, you have the IDM, which is the inverse dynamics model. So basically for each chunk production to produce one chunk for 16 frames, you need to run the DiT, which is a diffusion transformer, 32 times because of the CFG. The CFG is you need to run the condition for the flow matching and also run another unconditioned flow matching. Then you can take the derivative of the gradient that you can do for the gradient design for the flow matching. If we go back to the architecture like this, people were talking about, oh, why not just not run the diffusion models? So we don’t need to predict the future frames. That works. And there’s a research paper called ImageWAM. Basically, they’re not predicting the future video chunks. Instead, they are predicting the future end state, which is the future end state of the single frame.

For instance, I’m predicting the future video for maybe 16 frames. Rather than predicting the whole video, we can just predict at T plus N, which is the end state of the frame. And there’s another research paper called Fast-WAM. Fast-WAM is more extreme in some sense. They think all the world representation is already learned in the hidden states of the DiT. So you don’t even need a decoder to decode all the videos. You can just use the hidden state as a condition to train your action head. By doing this, you don’t even need a decoder in the training and also in the inference pipeline. If we take an analogy of those two different world action models, for the generative world action model, where you need to decode and then predict the future frames, you’re pretty much like a VR of Google Maps. But for the latent world action model, it’s like you look at the navigation of your Google Maps and think about where the model or the policy is heading to and what kind of action you’re going to produce in the future. In conclusion, all the problem comes down to the question about how to keep the rich world representation. Some people like LeCun, they think about, because they were doing JEPA, they think about maybe we can have two different encoders. And then one encoder is encoding current observation, and then the other one is encoding a future observation. And by doing loss on the current observation latent and the future observation latent, then you can teach the model to learn how to predict the future. And that’s one way of doing this. And they’re doing this using MSE loss. And other people, they treat the future as a distribution of possibilities.

It might be you take this possibility of this action, you might take another possibility of taking another action. So you treat the massive distribution and then you estimate those kinds of action distributions using flow matching. We talk about those different optimization angles we might have. And for us, since we are doing the infra thing, we did all those optimizations on our infra. We did distillation on the VAE part, which is a VAE encoder-decoder. We also did distillation on the DiT part. So the DiT became smaller. We also divided them. Because previously they were using the same DiT, we divided them into two different DiTs. Rather than decode all the future frames, we can just use the cross-attention from the video transformer to the action transformers. So the action transformer learns the head and state, which is the world representation from the video transformer.

And you don’t have to decode the future frames anymore. We also did distillation on the auto-regressive flow matching sampling. Previously, it might take 50 steps or 100 steps to do the flow matching decoding, but we made it down to one or two steps, which is immediately a 50 times speedup. We also did some changes on the modality side because we know future representation or world representation can be learned through pixel level or latent level. Is it possible we can find a more suitable modality to represent or retain the world representation? One way of doing this might be mask, and the other way might be flow. We tested both of them, and you can see the heat map is the visualization of the model. It guides the world model about which action you’re going to take in the future, 0.5 seconds.

By using our infra, the world action model can run 500 milliseconds per chunk, which is 16 actions on Jess and SOAR. We also wrote a full blog about how we did this on our website. This is the QR code if you need to learn more about it. Also, this is the LinkedIn of the founders of us. That’s it. Thanks, guys.

Audience: Going back to that slide that you had with the training and inference of the world action models. In the

Audience: Training

Audience: Stage, you’re using flow matching and teacher forcing. That makes sense. But in inference, you’re going to start from a fully noisy space and then you’re going to come to the future time step. How do you ensure that at inference time, the model actually collapses to the right thing and it just doesn’t degrade to noise?

Guanming: Oh, that’s a very good question. For the training part, you begin with the clean future latency and then you add noises to the clean future latency gradually. Eventually, the future clean latency will become pure noise at the end for the training part. By doing this, the inference learns how to reverse it back. So when you give it pure noise, the inference learns how to do this gradually and then reverse back to the future clean latency. That’s basically how it works.

Audience: So like curriculum learning or starting

Guanming: With very

Audience: Little noise initially. And as you train for longer, you have more noise.

Guanming: And that’s why it’s super slow because people are doing this maybe for 50 steps. We think it’s too slow, so we just found a way to distill it to two steps or three steps so it’ll be way quicker and without performance drop.

Audience: Yeah, that’s super interesting.

Audience: So when it comes to the world models, is the performance improvement due to the action heads looking at more details when we are forcing them to predict the whole frame? And second one is when there are multiple agents that are involved in the scene, does the model actually develop some kind of a theory of mind and predict other agents’ actions in order to be able to predict the world?

Guanming: Oh, that’s a very good question. I want to go back to the architecture of VLA and world action models. I got a lot of questions about what is the difference between VLA and the world action model? Why does the world model need to predict the future rather than just predict the action itself? A way to answer this is for VLA especially, they’re just based on the current observation to predict the current action. They don’t have the explicit learning of the future kinematics. The only reason we need to introduce videos, especially for future videos, for the world action model, is we want to teach the model to learn the future kinematics. For example, I’m holding a bottle of water and then I drop the bottle of water. From the pixel level, if we give the future videos to the model, the model learns how the kinematics will change at the pixel level.

We know this is a kind of supervision, teaching the model to learn the future dynamics. We believe the future dynamics help with the action generation as well because it’s physics. By doing this, we teach a model how to learn the correlation between the physics from the pixel level to the action you produce.

Francois: To test that hypothesis, could you train a transformer stack body to do next frame prediction and then more and more and then rip off the head and just do the action? Would it do the same thing?

Guanming: As in, like, we divide them into two different transformers?

Francois: Different stages.

Guanming: Yeah. Of course, there’s a more efficient way to do so. It’s called mixture of transformers. If you check ImageWAM, they actually do the same thing. They did the cross attention from the image encoder backbone to the action expert. Just like ImageWAM and also Fast-WAM, they realize the world representation—you don’t have to explicitly decode the frames. You can just keep them in a hidden state and then do a cross attention from the last layer of the backbone to the action head. So, short answer for this is of course you can do so. We realize that this is the more efficient way to maintain the world representation, meanwhile producing the best action based on the world representation.

Francois: And then is there no test time planning that’s done with WAM where you will invoke, let’s say, 10 samples and I’ll get 10 different end states and 10 different actions, and then I’ll pick the best end state of the ones that were sampled and then emit that action? Is that not done?

Guanming: Yeah. I think it’s pretty much how flow matching works, right? Because for flow matching, you treat action as a possibility of distributions. And then you always sample the best trajectory. And we use teacher forcing to teach the model to sample the distribution of the actions. So I think for flow matching, they’re already doing the same thing.

Francois: And then last question from me, why choose the business model of being a vLLM equivalent for WAMs versus just actually do like Nico says, become a robotics application company and actually go end to end?

Guanming: I think Bill can answer this question.

Bill: Yeah. I think the reason we went with this route is because we really believe in having this understanding of the world for your models. But I think right now everyone’s focused on maintaining the research so that it can be as generalizable as possible. But eventually every model like that needs to go on the edge and needs to go in real time. So I think right now, not a lot of people are focused on building that infrastructure that allows those models to perform really well on your robots first. We want to be the first company to do that.

Audience: Hi, I have two questions. The first one is more of a clarifying question. So for a WAM, is the prediction auto-regressive in previous frames or is it just one previous T minus one and then to T without any other T minus two, minus three, and so on?

Guanming: I think what you’re talking about is the chunk size, because you can change the parameter as well. So for the training, you can do 16 frames, which people, they all do 16 frames, which is you predict in the future 16 frames. And people, they also do 32 or even higher frames. But if you increase the chunk size, which is more frames you produce, then it will be harder for the model to learn the future states because it’s longer.

Audience: Understood. Okay. And my second question is, are you familiar with any work that uses encodings for the differences between frames? So like temporal difference encoding, I think like a recent work by Yann LeCun, as well as a tech blog by, I think, Induction Labs where they train an imagination model that predicts latent encodings for the differences between adjacent frames in a video.

Guanming: Yes. And if you check here, we mentioned asymmetrical de-noising because we found a way that you can actually measure the energy of the KV cache. So rather than producing all the future frames, why not just produce those, maintain the highest details of the action you’re doing right now? And that’s basically what we added to the infra as well. So we found a way to measure the different energy of the KV cache, and then based on the energy we decide if the model is going to predict different resolution of the frames or just purely doing this on a latent space.

Audience: Thank you.

source & further reading

ycrootaccess.com — original article Garry Tan: Own Your Intelligence Eight ML Papers, Explained by the Researchers Behind Them Starcloud: Solving AI’s Energy Problem in Orbit