{"slug": "an-agentic-robotics-fable", "title": "An Agentic Robotics Fable", "summary": "Agentic policies that write and iterate on their own Python skills reached a 34% to 65% overall task success rate across ten bimanual tasks, beating end-to-end policies trained on 30 times more demonstrations on 8 of 10 tasks, according to a Poke & Wiggle robotics study. The best end-to-end policy needed 300 teleoperated demos to reach 37%, while agentic policies started from the same ten demonstrations, ran skills in simulation up to 30 times and requested up to 10 real trials. On extrapolated object layouts, the agentic model Fable 5.2 still reached 52%, while Rho trained on 300 demos fell from 53% on nominal layouts to 13%.", "body_md": "# Chapter 3:\n\nAn agentic robotics fable\n\nPoke & Wiggle ·\n\nIt’s all about closing the loop.\n\nHumans close it after every single trial. Miss the bin with a paper ball, and your next throw already corrects for it. We iterate, and in a sense we RL our way to performance. But doing that efficiently takes intelligence in the way we learn. That is what makes us such fast learners: we reason about why something failed, change exactly that, bring in everything else we know about the world, and go again.\n\nUp to now, the AI robotics space has mostly left that intelligence out of the loop. A policy that drops a screw doesn’t ask why. It waits for more demonstrations. For years, the field has obsessed over the fine print of control instead. Action chunks. Absolute, relative or delta poses. Binning or flow matching.\n\nSomewhere along the way, robotics forgot how useful general intelligence is. Language models learned to read, plan, write code and fix their own bugs, yet we rarely let them solve tasks in the real world.\n\n**So what happens when you give that intelligence hands?**\n\nWe follow a line of work that put coding agents on robots: [Code as Policies](https://arxiv.org/abs/2209.07753) (Jacky Liang, Andy Zeng and colleagues), [CaP-X](https://arxiv.org/abs/2603.22435) (Max Fu, Ken Goldberg, Jim Fan and colleagues), [ASPIRE](https://arxiv.org/abs/2607.00272) (Runyu Lu, Yuke Zhu, Jim Fan, Guanzhi Wang and colleagues) and [RAX](https://arxiv.org/abs/2610.22496) (Runhan Huang, Xiangcheng Zhang, Yilun Du and colleagues). What we add is the measurement: agents evaluated with the same rigor as our VLA evals, on the same real layouts, every trial rated by people.\n\nEach model gets ten bimanual tasks and our robot API. For each task, it does not start from nothing: it also gets our skill library, our motion planner, a simulator of the task and ten teleoperated demonstrations. It writes skills in Python, runs them in simulation up to 30 times, then asks for up to 10 real trials.\n\nOur fable has thirteen findings, in three parts.\n\n## Part oneAgentic compared to end-to-end policies\n\nSame ten tasks, same layouts, same robot. The agents write their own skills, in code, from ten demonstrations, and iterate on them in up to 30 sim and 10 real trials. The end-to-end policies on our leaderboard train on 10, 100 or 300.\n\n- **Agentic policy.** A frontier model learns the task by writing code. It reads ten teleop demos, writes Python skills on top of our skill library and fixes them between sim and real trials. On the robot the code runs on its own and calls the model only at decision points it wrote itself.\n- **End-to-end policy.** A neural network trained on 10, 100 or 300 teleop demos maps camera images and the robot’s state straight to short chunks of arm motion. What it learned lives in its weights.\n\n### Finding 1Agentic policies beat end-to-end policies trained on 30 times more data on 8 of 10 tasks\n\n**The gap is not subtle.** With the same ten demonstrations, every end-to-end policy on our leaderboard stays in the single digits. Agentic policies reach an overall task success rate of 34 to 65%. The best end-to-end policy needs 30 times more data to pass the low end of that range, and even then stops at 37%.\n\n### Finding 2Agentic policies hold up on OOD object placements\n\nOn extrapolated layouts Fable 5.2 still reaches 52%, while Rho trained on 300 demos drops from 53% on nominal ones to 13%.\n\n### Finding 3Agentic policies know when they fail\n\nThey win where they can check their work. After every training trial, they work out what went wrong and change the code. On the robot, that code measures whether a box is seated or a target is in reach, and almost never claims a success it didn’t have. This self-assessment is what end-to-end policies usually miss, and it enables a whole new set of insights into a policy and makes its solutions interpretable.\n\n### Finding 4Agentic policies recover from their own mistakes\n\nWhen something looks wrong, the agentic policy calls the model with the recoveries the agent wrote for that moment, and the model picks one or gives up. Because the skills read the scene instead of replaying the demos, they still work after a mistake: a connector that slips lands somewhere no demo covered, and the policy finds it and picks it up again, as Argon does in the video below. Their solutions are robust, and at decision points they use common sense.\n\n### Finding 5Agentic policies take 1.3 to 12 times a teleoperator’s execution time\n\nAveraged over the ten tasks, a successful trial takes a teleoperator 47 s and the end-to-end policies trained on 300 demos 46 s. The agentic policies take 124 to 177 s, 2.6 to 3.7 times as long. On Pour screws, the teleoperator takes 12 s, Rho 13 s and the agentic policies 33 to 49 s.\n\nAverage execution time\n\nReal robot, nominal layouts, time of a success, averaged over all ten tasks; an agent that never succeeds on a task is averaged over the tasks it solves.\n\nExecution time example\n\nPour screws into the funnel at real speed: a teleoperator, Rho trained on 300 demonstrations and the four agents, fastest first, each on its fastest success on this layout.\n\n### Finding 6End-to-end policies still win where the move is easier to imitate than to write down\n\nWhen loading two boxes into a container, the agentic policies get the first box in most trials but rarely the second: it fits with only a few millimetres to spare, and every time the code stops to measure, the box shifts slightly in the gripper. On the toolbox, Fable finds the latches by touch in a median 97 s on the demo layouts but 180 s on new ones, and runs out of time. In our comparison video, Rho trained on 300 demos opens it in 42 s.\n\nFinal evals on one layout, real speed. Rho trained on 300 demos is done in 42 s; Fable runs out of time and Opus fails to lift the lid.\n\n### Finding 7Training an agentic policy costs about as much as training an end-to-end policy\n\nTraining one task costs the agentic policies $10 to $230 in model tokens and the end-to-end policies $10 to $180 in GPU time. Fable 5.2 is the most expensive agent, mostly because it keeps working through its real trials. The cheapest policies, GR00T and DiT-Flow, are also among the weakest.\n\n## Part twoHow should an agent control a robot?\n\n### Finding 8An agent does better writing code than driving the robot live\n\nWe tried two ways and compare them with the same model, ChatGPT-6 Astra.\n\n- **Direct.** The model watches the cameras and picks every move live: about 12 calls and 35 images per trial.\n- **Skills harness.** The model writes its own skills in code, on top of our skill library, practises in our simulator and asks for about ten real trials. On the robot the code runs alone and calls the model only at decision points it wrote itself, about once per trial.\n\nWith skills, Astra wins Push box and Spray bottles, nearly ties Pour screws, and is four to eleven times faster: a successful Pour screws trial takes one minute instead of 11. Direct only kept up on Pour screws and almost never succeeded on Spray bottles.\n\n### Finding 9You can watch an agentic policy’s code run, skill by skill\n\n## Part threeThe agent models compared\n\n### Finding 10Fable 5.2 leads on success, generalisation and recovery\n\nFable 5.2 succeeds on 65% of all trials, against 34 to 39% for the other three. It keeps 87% of its success on OOD placements, gets 88% of steps right the first time, and later fixes 27% of the steps it fails. That lead has a price: Fable is the most expensive to train, at $232 per task, and writes the most code, about 6,300 lines per task. Argon 4 is the fastest, at 2.6 times a teleoperator’s time, and the cheapest to train, at $47, but its verdicts are the least reliable (85%). Astra 6 judges its own trials best, right 99% of the time.\n\nHow the four agents compare\n\nReal robot, final evals, all ten tasks, 900 trials per agent.\n\nSuccessof all trials\n\nGeneralisationkept on OOD placements\n\nFirst trysteps right first time\n\nRecoveryfailed steps later fixed\n\nSelf-assessmentits verdict is right\n\nExecution timevs teleop, lower is better\n\nTraining costper task, lower is better\n\nCode writtenlines per task, not a score\n\nImages viewedper task in sim, not a score\n\n- **Generalisation** : success on extrapolated layouts divided by success on nominal ones, median over the tasks with at least 20% nominal success (Fable 9, Opus 6, Argon 5, Astra 4).\n- **First try** : share of steps whose first attempt was rated a success.\n- **Recovery** : share of failed subtasks that a later attempt at the same subtask completed.\n- **Self-assessment** : share of trials where the agentic policy's own verdict matched the rating, over trials with a recorded verdict (Fable 713, Opus 670, Argon 530, Astra 800 of 900).\n- **Execution time** : time of a success divided by the teleoperator's, averaged over the tasks each agent solves (Astra 6 of 10).\n- **Training cost** : model tokens for sim and real trials.\n- **Code written** : lines of the final solution, mean per task.\n- **Images viewed** : camera frames, plots or renders the model opened in its sim phase, mean per task.\n\n### Finding 11Every agent improves in sim, and Fable 5.2 starts ahead\n\nTrial by trial, every model gets better in sim. Sim success does not carry straight over to the real robot, but the agent keeps adapting its skills during its 10 real world trials too: Opus, Astra and Argon drop to about 20% success rate in their first real trials and climb to about 40% by their last, and Fable wins 58% of its real trials, averaged over its 10 tasks.\n\nHow each model learns one task\n\nThe agent gets ten teleop demos, our skill library and a simulator. It writes skills in Python, runs up to 30 sim trials, then asks for up to 10 real ones, changing its code along the way. Its final code then runs 30 real trials per layout set (nominal, interpolated, extrapolated), each rated by people. Shown for Connect the DC jack; read each column top to bottom.\n\n### Finding 12Agents find solutions that differ from the teleoperators’ demos\n\nConnect the DC jack. In all ten teleop demos, the connectors are grasped at an angle, both arms lift them about 30 cm and join them in the middle. Opus does the same; Fable and Argon found their own ways.\n\n- **Teleoperator** , as in all ten demos: grasps both connectors at an angle, lifts them about 30 cm with both arms and joins them in the middle.\n- **Fable 5.2** does not bring both arms to the middle. Its left gripper holds one connector still on the table, and the right arm carries the other 29 cm over and plugs it in there, never more than 2 cm up.\n- **Opus 5.5** stays closest to the demos: it grasps the connectors at an angle, as in the demos, lifts both about 28 cm and joins them in the middle.\n- **Argon 4** grasps the connectors vertically, then rotates its grippers so the wrist cameras do not collide. In the demos the connectors are grasped at an angle, so they join without rotating. Argon lifts both only about 13 cm, then moves just one into the other.\n\n### Finding 13Fable 5.2 built its own world to practise in\n\nEvery model tests its code before it spends a trial, but the tests catch different things. On nine of its ten tasks, Fable wrote a fake station and ran its whole skill code on it, moves included. Opus did the same on two tasks. Astra replays old camera frames, which checks its vision but never its moves. Argon builds nothing to practise on: before its first sim trial it runs 60 to 120 one-off scripts over the demo recordings, and keeps none of them.\n\nFrom Fable 5.2's world to the real robot\n\n## Conclusion\n\n1. **Part one:** Agentic policies beat end-to-end policies on 8 of 10 tasks with 30 times fewer demonstrations, and hold up on out-of-distribution placements. End-to-end policies are still faster, and win where the final move is easier to imitate than to write down.\n2. **Part two:** Let the agent learn the task and write code to solve it.\n3. **Part three:** Models differ in the strategies they find and the methods they use, such as building their own world to practise in.\n\nWe think the future of AI robotics is agentic. The agent is the brain, and VLAs and WAMs might become its muscle memory: a tool call for the moves it should not have to think about. It sounds like a demotion. It isn’t. Muscle memory is what lets a pianist forget her fingers and play.\n\nWill it turn out that way? Who knows. But we know how to find out: give agents an API to the real world, and keep closing the loop.", "url": "https://wpnews.pro/news/an-agentic-robotics-fable", "canonical_source": "https://pokeandwiggle.com/blog/agentic-robotics#part-oneagentic-compared-to-end-to-end-policies", "published_at": "2026-11-13 16:03:45+00:00", "updated_at": "2026-11-13 16:20:26.912222+00:00", "lang": "en", "topics": ["robotics", "ai-agents", "ai-research", "machine-learning"], "entities": ["Poke & Wiggle", "Fable 5.2", "Rho", "Code as Policies", "CaP-X", "ASPIRE", "RAX"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/an-agentic-robotics-fable", "markdown": "https://wpnews.pro/news/an-agentic-robotics-fable.md", "text": "https://wpnews.pro/news/an-agentic-robotics-fable.txt", "jsonld": "https://wpnews.pro/news/an-agentic-robotics-fable.jsonld"}}