Robotics • 51.6M • Updated • 5
#
From Operations to a Robot Arm
[Community Article](https://huggingface.co/blog/community)
I come from operations research, not robotics. Two months ago I had never trained a model that controls a motor. This is the story of teaching myself to put a Kirby toy in a bowl with a low-cost robot arm, across four model versions, one catastrophic failure, and a climb from 60% to 90%, plus everything I got wrong on the way.
#
TL;DR
- I am an operations researcher who spent two months teaching himself to train robot policies. The task: get a low-cost SO-ARM101 arm to pick up a Kirby toy and drop it in a bowl.
- Four versions. v1 reached 60% success, v2 65%, v3 broke completely, v4 landed at 90%.
- Three things the project beat into me: data coverage and quality matter more than raw episode count; slow inference can disguise itself as a bad model; and you cannot fine-tune on correction data alone, you have to combine it with the original data (DAgger ), or the model forgets the task.
- My clearest mistake: I trained all of it without a validation set. Next project, validation loss from day one.
- The thing that surprised me most: most of the hard parts of robot learning are operations problems in disguise.
#
From operations to robots
For about seven years, my work lived between algorithms and people. I am an operations researcher by training: optimization, queuing theory, the math of why a system is slow and how to make it faster. I spent those years inside fast-scaling startups and one very large company, building the models and also building and leading the teams that had to run on them. It is a useful combination. The algorithms teach you that the real world is always messier than the model. Running the teams that depend on those models teaches you something else: the gap between a clean demo and a system that holds up every single day is where the real work lives. Hold that thought, because it turns out to be the whole story of robot learning too.
Then I took a sabbatical, most of it across Asia. I went for the usual reasons. I came back thinking about robots.
Here is what got me. In Shanghai and Beijing, in Tokyo and Osaka, robots were not a keynote slide. They were just around. A robot vacuumed the hotel lobby at night. Another rode the elevator and brought a late order to my door. In a café, a robot arm pulled the espresso. None of it was presented as the future. It was just Tuesday.
The last few years, I had known AI mostly as a thing you type at: a chat box, a dashboard, a screen. This was different. This was AI with a body, doing a physical task, in a room with people walking around. And the more I looked, the more it lined up with the numbers (all from the IFR World Robotics reports):
- China now installs more industrial robots than the rest of the world combined, roughly 54% of the global total in 2024, a record 295,000 units in a single year.
- South Korea has the world's highest robot density, over 1,000 robots per 10,000 manufacturing workers, more than double almost every other country.
- Japan no longer tops the density chart, but it still builds more of the world's industrial robots than anyone else, the FANUCs and Yaskawas, and the precision gearboxes nearly every arm on Earth quietly depends on.
I was not looking at novelties. I was standing in the part of the world where the physical layer of AI is furthest along.
I am an operations person. When something interesting is happening at scale, my instinct is not to read about it. It is to go find the floor and see how it actually works. So I came home and did the only thing that made sense to me.
I bought a robot arm.
A service robot working a café in China. Nobody looked up.
Room service, autonomous. This one rode the elevator up by itself and waited at the door.
#
The rig
The arm is an SO-ARM101, the open-source design that LeRobot is built around. You get two of them: a leader you move by hand, and a follower that copies it. That is how you record demonstrations. You puppet the leader, the follower mirrors you, and the system logs what the cameras saw and what every joint did. Twelve servos in total, six per arm. Two cheap cameras, one looking at the workspace from the side and one over the top.
Assembly took a weekend. Calibrating the servos, flashing IDs, getting the leader and follower to agree on where "zero" is. The first time the follower mirrored my hand in real time I sat there grinning like it was magic. It is not magic. It is a feedback loop and a serial bus. But it feels like magic the first time.
This was the first thing that surprised me, and it is the same point those café robots were quietly making: the hardware is the easy part now. A weekend of screws gets you an arm that can do almost anything a simple gripper can do, as long as a human is driving it. Getting it to do those things on its own, with nobody holding the leader, is the hard part. Everything that follows is about that.
Two early lessons that cost me time. First, the cameras. I started at 1080p because higher resolution sounded better, and immediately hit a USB bandwidth wall that dropped frames. I came down to 640 by 480 at 30 frames per second, which is the LeRobot field standard, and everything got stable. Second, placement. Where you put the cameras decides what the model can and cannot see. If the gripper hides the object at the moment of contact, the model is blind exactly when it matters most. I moved cameras around for an evening before I had angles that did not occlude the grasp.
Then the real work started.
The rig, assembled. A leader you move by hand and a follower that copies it.
Earlier the same weekend. Printed parts, a pile of servos, a lot of small bags.
#
v1: the first model that worked, and what it taught me about data
The task I picked is deliberately simple. Pick up a small Kirby toy and put it in a bowl. Easy to judge: either Kirby ends up in the bowl or he does not.
I recorded 60 demonstrations by teleoperation, moving the leader through the pick-and-place by hand 60 times, varying where Kirby started. Then I trained an Action Chunking Transformer, the ACT policy, on a Colab A100. About two and a half hours, 100,000 steps, and a bill of roughly 1.60 euros. The training loss looked beautiful. Final L1 around 0.038, which is deep in the zone where the loss curve says the model has learned the motion well.
Then I ran it on the real arm, and reality showed up.
The first five rollouts went four out of five, which felt great. Then I ran a proper evaluation, 20 trials with Kirby placed all over the workspace, and the honest number was 60% strict success. Twelve clean successes, two partial, six failures. A model whose loss said "polished" was missing one pickup in three.
The failures were not random. I had been logging every Kirby placement and every rollout outcome onto a photo of the workspace, a small dashboard I built so I could see the spatial pattern. When I looked at it, the story was obvious. The successes clustered where I had recorded a lot of demonstrations. The failures clustered in the corners and edges where I had recorded almost none. The model was not bad at the task. It was bad at the places I had barely shown it.
That is the first lesson, and it is the one every practitioner writeup will tell you too. For a task like this, the coverage and quality of your data matter more than the raw count, and the training loss will not warn you about any of it. The loss measures how well the model fits the demonstrations it saw. It says nothing about the places you forgot to demonstrate. The only thing that told me the truth was running the policy on the real arm and watching where it fell apart.
Every placement and every rollout, mapped onto the workspace. The clusters are where I had data. The gaps are where v1 fell apart.
v1 on the real arm. Good when Kirby landed where the data was.
#
v2: more data, and two traps I did not see coming
The fix for v1 looked obvious. Fill the gaps. I recorded 40 more demonstrations aimed straight at the sparse corners the dashboard had exposed, taking the dataset from 60 to 100. By this point I had done this enough that my hands were better at it. The new batches came out 90 to 95% clean, much higher than my first nervous attempts. Operator skill compounds, which nobody warns you about. Your hundredth demonstration is genuinely better than your first.
Every recorded episode, scored automatically for trajectory quality. 74 of 100 came out clean, 26 flagged as outliers, none severe. Building the tooling to see your own data is half the work.
I retrained, ran 20 trials, and got 65% strict success. Better than v1, but only barely. Forty fresh demonstrations had bought me five points. Something else was going on.
Two things, it turned out.
The first was a trap I walked into without noticing. Between library versions, a default had changed, and my action chunk size had silently flipped from 50 to 100. I had not chosen that. ACT predicts a chunk of future actions at once, and a longer chunk means the model commits further ahead before it looks again. After this, I started writing down every hyperparameter as a deliberate decision rather than trusting defaults. A value you did not choose is a value that can quietly break you.
The second was worse, and it had been hiding under all of v1 too. My inference was slow. The control loop was running at about 8 cycles per second, when the system is meant to run near 30. A policy that reacts three times slower than it should will overshoot and lurch, because it keeps acting on stale decisions. I had been blaming the model for behavior that was really a speed problem.
The tell was simple once I looked for it. Plain teleoperation on the exact same Mac ran at 27 to 28 cycles per second. Same cameras, same machine. The only thing different during inference was the model's forward pass. The model was the bottleneck, not the hardware.
The fix was to stop running the model in PyTorch on the Mac's GPU and convert it to CoreML, Apple's own runtime, which spreads the work across the Neural Engine, GPU, and CPU. The forward pass dropped from 90 milliseconds to 63. Timing jitter dropped from plus or minus 5 milliseconds to plus or minus 1. The output drifted from the original model by about a quarter of a percent, far below the precision of the servos, so it was the same policy, just faster. The real loop went from 8 cycles per second to 11.5.
That is the second lesson. Inference speed is not a footnote. It is a confound that can make a perfectly good policy look broken, and you will read it as a model problem unless you measure the loop against a known baseline.
The training curves looked polished long before the arm agreed. The loss is not the score.
| Inference path | Forward pass | Timing jitter | Real control loop |
|---|---|---|---|
| PyTorch (MPS) | 90 ms | plus or minus 5 ms | about 8 Hz |
| CoreML | 63 ms | plus or minus 1 ms | about 11.5 Hz |
#
v3: the version that broke
Now I wanted to push past the plateau, and the right tool for that is DAgger. The idea is clean. Run the current policy on the arm. When it starts to fail, take over, demonstrate the correct recovery, and add those correction episodes to your data. You are collecting examples exactly where the model is weak, which is far more efficient than recording more random demonstrations.
The robot fought me first.
The smooth handover feature, where the leader arm drives itself to match the follower's pose so you can grab it mid-rollout, would not work on my kit. The leader produced zero torque. I wrote six diagnostic scripts over an evening, confirming that every software register was correct: torque enabled, position mode set, goal positions accepted, bus communication clean. The software was perfect and the arm was limp. It was a hardware problem with power reaching the leader's servos, not anything I could fix in code. I gave up on smooth handover and used the unglamorous workaround: the rollout, move the leader by hand to roughly match, hand control back. It worked fine. Sometimes the answer is to stop debugging the software and just move the arm with your hand.
I collected around 20 correction episodes this way, all targeting the spots where v2 stumbled. Then I fine-tuned v2 on those corrections and called it v3.
The 20 corrections, scored the same way as the demonstrations. Most of the interventions landed on a single joint, the shoulder lift.
v3 was a disaster.
The arm stopped doing the task at all. Instead of approaching Kirby, it swung straight to the edge of the workspace and stayed there, every time, like it had forgotten the center of the table existed. The model that had been getting two pickups in three now got essentially none. I had made it worse by trying to make it better.
This is catastrophic forgetting, and I diagnosed it the slow way, by going back and reading the original 2011 DAgger paper line by line. The whole thing turns on one detail. After you collect corrections, you are supposed to train on the corrections combined with all of your original data. The paper writes it as a single line, D ← D ∪ D_i: the new data gets added to the old, never swapped in for it. I had skipped that. I fine-tuned on the corrections alone, so the model optimized hard for the 20 correction episodes and quietly threw away everything the 100 base episodes had taught it. The corrections all sat near the edges, so the model decided the answer to everything was "go to the edge."
That is the third lesson, and it is the one I will never forget because the robot acted it out for me. Never fine-tune on corrections alone. Correction data is something you add to your dataset, not something you swap in for it.
v3 forgot the task existed. Every single trial, straight to the edge.
#
v4: DAgger done right
The fix followed directly from the diagnosis. Build one dataset that holds the 100 base episodes and the 20 corrections together, 120 episodes total, and train on that. This was fiddlier than it sounds, because the correction episodes carried an extra data column the base episodes did not have, so I had to strip that column before the two could be merged, and the obvious one-line way to combine datasets was not actually implemented in the version I had. A few quirks, nothing deep. I also set the image augmentation explicitly this time instead of trusting it to carry over, since I had learned that fine-tuning does not inherit the dataset settings on its own. That omission was part of what had hurt v3.
Then 20 trials.
v4 hit 90% strict success, eighteen out of twenty, a 25 point jump over v2.
The number I cared about most was not the headline though. It was the breakdown by location. Five of the 20 trials were placed in the exact zone where I had recorded the DAgger corrections, the hard zone that had been failing. v4 got three of those five, where v3 had gotten none and v2 had been shaky. The other 15 trials, in the normal zones, went 15 for 15. That is the pattern you hope to see: the corrections improved the hard region without wrecking everything else. v3 showed what getting it wrong looks like. v4 showed what getting it right looks like. Having both, back to back, made the lesson concrete in a way no paper could.
Two honest caveats, because I would want them if I were reading this. First, 20 trials is a small sample. The jump from 65% to 90% is real and large, but with numbers this small a formal test puts it just over the usual significance line, not comfortably past it. The effect is clear, the precision is not. Second, the 25 point gain is not all from DAgger. Some of it is the CoreML speed fix from the v2 stage finally paying off, and I cannot fully separate the two without more controlled runs. Better data and faster inference both helped. I would rather say that plainly than pretend I ran a clean ablation.
v4. Same arm, same task, done right.
The v4 evaluation, all 20 trials scored automatically. 18 success, 2 fail, 90 percent. The two failures both fell in the hard zone.
| Version | Episodes | Strict success (n=20) | What changed |
|---|---|---|---|
| v1 | 60 | 60% | first working model |
| v2 | 100 | 65% | filled sparse zones, CoreML inference fix |
| v3 | 100 + 20 corrections | broke | fine-tuned on corrections alone, forgot the task |
| v4 | 120 aggregated | 90% | corrections plus base data, done right |
#
What I would tell myself eight weeks ago
A few things I wish someone had said to me at the start, beyond the three lessons already above.
I never used a validation set. All four models were trained without holding out any episodes to check generalization, and I picked checkpoints by step count and gut feel. This is the clearest mistake of the whole project. v3's collapse was caught by the robot, in the real world, weeks after I trained it. A held-out validation curve would have shown the same collapse during training, around step 5,000, before I ever touched the arm. From the next project on, this is not optional. Log validation loss next to training loss, and choose the checkpoint with the lowest validation loss, not the last one.
Deployment is the only ground truth. My v1 loss said the model was polished while it failed one pickup in three. The loss measures fit, not success. If you only have time to trust one number, trust the success rate on the real hardware.
The boring parts are the parts that matter. Throw away your first few recording sessions, they are practice. Move slowly when you record. Buy spare servos before you need them. None of this is clever. It is all true anyway.
And the thing that surprised me most, coming from operations. The hard problems in robot learning are not really about robots. Collecting good data is a workforce and quality-control problem. Keeping a policy reliable enough to run all day is an uptime problem. Coverage, consistency, the distance between a demo and a system that works every time. I had been managing versions of these problems for seven years. The robot just made them physical.
#
What's next
The 90% is on the task as I trained it. The honest next step is to break it on purpose: move the bowl, change the lighting, put Kirby in poses I never demonstrated, swap the table surface, and see how far the policy holds outside the conditions it learned. That out-of-distribution test is the difference between a model that memorized my room and a model that learned the task.
After that, a harder task. Something with more than one step, or an object that does not hold its shape. And this time, validation loss from day one.
If you are coming at this from outside robotics, like I did, the short version is that it is more reachable than it looks and more humbling than you expect. But the first time a thing you trained reaches out on its own and drops Kirby in the bowl, with nobody touching it, you will understand why I kept going back to the desk. The whole task in one frame: the arm, Kirby, and the bowl.
#
Resources
- Models on the Hub: act_kirby_to_bowl_v1 ,v2 ,v3 ,v4
- Datasets: kirby_to_bowl_v1 (the demonstrations),kirby_to_bowl_v1_plus_dagger_iter1 (the aggregated v4 set), and the eval rollouts (v1 ,v3 ,v4 )
- Hardware and framework: [SO-ARM101](https://github.com/TheRobotStudio/SO-ARM100) ,[LeRobot](https://github.com/huggingface/lerobot)
- Papers: [ACT (Zhao et al. 2023)](https://arxiv.org/abs/2304.13705) ,[DAgger (Ross et al. 2011)](https://arxiv.org/abs/1011.0686)
- The writeups that taught me the most: Sherry Chen's ACT on SO-101 and the LeRobot team'srobot folding post