{"slug": "what-we-are-seeing-with-gpt-astra-in-robotics", "title": "What we are seeing with GPT Astra in Robotics", "summary": "OpenAI's self-reported results show GPT-6 Astra reaching 95.9% mean voxel IoU on the University of Virginia's BenchCAD benchmark, against 83.3% for GPT-5.6 Sol and 84.3% for Claude Fable 5.1 under a modified evaluation protocol, according to an analysis of the model's robotics, 3D, CAD, and simulation results. BenchCAD contains approximately 17,900 execution-verified CadQuery programs across 106 families of industrial parts, and the Astra score has not been independently re-graded by the BenchCAD team. The analysis argues the evidence does not support the claim that robotics has been solved by scaling foundation models, but also that dismissing the results as retrieval, contamination, or selected demos is insufficient.", "body_md": "*Some thoughts on robotics, 3D, structure, and what we should actually be measuring*\n\nOver the last few days, I have been looking more carefully at the emerging GPT-6 Astra results in robotics, 3D, CAD, simulation, and autonomous experimentation.\n\nI started from a rather skeptical position. There is a lot of hype around these demonstrations, and in robotics especially, we should be very careful about what exactly a successful video demonstrates. What information did the model receive? What was in the prompt? What tools and abstractions were available? Was the model directly producing the behavior, or was it constructing a system that eventually produced it? How familiar was the task from training, and what information could the model retrieve at test time?\n\nAfter looking more systematically at the results, however, I believe something interesting is happening. I do not think the evidence supports the claim that robotics has suddenly been solved by scaling foundation models. The physical manipulation results alone make this clear. At the same time, I also do not think that dismissing these results as retrieval, contamination, or a collection of carefully selected demos is sufficient.\n\nThere seems to be a real change in capability, but we need to be more precise about what that capability actually is, what gives rise to it, and what role structure is playing in producing it. By **capability**, I mean the set of skills or competencies that the system displays, for example, its ability to interpret visual input, reason about spatial relationships, generate actions, or construct code and workflows. **Structure**, in this context, refers both to the explicit computational frameworks and abstractions provided around the model, such as action spaces, tools, simulation environments, and controllers, and to structure within the model’s learned representations, such as inductive biases, equivariances, or relational encodings. Clarifying this distinction is important because the behavior we observe is a property of the complete system. Understanding how much of that behavior comes from what the model has learned, and how much becomes possible through the structure provided around it, is central to interpreting these results.\n\n### What is Astra actually doing?\n\nFrom observations to executable hypotheses\n\nTake some of the 3D results. In [BenchCAD](https://arxiv.org/abs/2605.10865), Astra receives multi-view renders of an object and is asked to generate executable CadQuery code reconstructing its geometry. With tools, Astra reaches 95.9% mean voxel IoU on BenchCAD in [OpenAI’s self-reported results](https://openai.com/index/gpt-6-astra/), against 83.3% for GPT-5.6 Sol and 84.3% for Claude Fable 5.1 under a modified evaluation protocol. BenchCAD itself is an independent benchmark from the University of Virginia, containing approximately 17,900 execution-verified CadQuery programs across 106 families of industrial parts. The Astra score has not, to my knowledge, been independently re-graded by the BenchCAD team.\n\nThe Blender demonstrations are a different setting. There, Astra can use Blender Python to construct a scene, render it, inspect the result, modify the scene, and continue iterating. We should not conflate this with the BenchCAD result, but there is an interesting commonality.\n\nIn both cases, the model can observe a problem, construct a hypothesis, make that hypothesis executable through code or tools, observe the result, and, if needed, revise either the hypothesis or its execution. The model does not necessarily have to produce the final solution directly. It can construct a process that produces a solution, observe what this process does, and intervene again.\n\nIn robotics, Astra can play very different roles\n\nNow let’s consider robotics. Here Astra appears in several quite different roles, and it is important to distinguish them rather than grouping all of them under the statement that “Astra controls robots.”\n\nIn some experiments, Astra is effectively used as the policy itself. [RoboCurve](https://openai.robocurve.org/gpt-6-astra/) provides a useful example because the setup is well documented. Astra receives three camera views, from a top camera and the two wrist cameras, together with proprioceptive state. It generates absolute end-effector poses and gripper commands for each arm, which are subsequently converted into joint angles through IK.\n\nOn the block-into-bowl task, Astra succeeds in 19 out of 20 trials, compared with 8 out of 20 for Claude Fable 5.1. On the much more precise puzzle-piece insertion into a groove, however, both models succeed in only 2 out of 20 trials and tend to stall at the same final stage. RoboCurve correctly notes that 20 trials are sufficient to distinguish 19 from 8, but not to establish a meaningful difference between 2 and 2.\n\nThese results suggest that there is indeed an interesting visuospatial-to-action capability inside the model itself. What exactly this capability represents is much less clear. Has the model acquired a sufficiently general representation connecting visual observations to spatial actions? How much is recomposition of task and trajectory structures encountered during training? How would the same capability behave under a new embodiment, different coordinate conventions, unfamiliar object geometries, or different interaction dynamics?\n\nEven in these direct-control examples, the model is not operating in an entirely unstructured action space. Predicting an end-effector pose is different from generating the low-level motor commands required to realize that motion. Kinematics, IK, and lower-level control already solve important parts of the physical control problem. This does not make the result less interesting, but it changes what the result tells us.\n\nThe pen-spinning example is something different again\n\nThe pen-spinning demonstration [reported by Wentao Zhu](https://x.com/walterzhu8/status/2100212420840989112?s=20) from an autonomous run by his student Chengyang Li received considerable attention. According to their account, Astra created the pen mesh, implemented a Sharpa-hand task in Isaac Lab, trained a PPO policy, and produced the final visualization over roughly a day and a half.\n\nThe obvious interpretation of the final video is that GPT-6 Astra can now perform dexterous manipulation. But this is not really what happened. Astra did not directly control the hand during the final behavior. It constructed the simulation, task, and learning setup through which a specialized policy was subsequently trained.\n\nThere is also an important precedent here. [Eureka](https://arxiv.org/abs/2310.12931) used GPT-4 to generate reward functions for reinforcement learning in Isaac Gym, and its headline demonstration was precisely pen spinning with a Shadow Hand. The general pattern of using an LLM as part of an RL design process is therefore not new, and pen spinning itself is certainly not a novel task. The interesting question is how much further than Eureka can Astra autonomously construct, integrate, debug, and execute the complete learning workflow.\n\nAstra’s role in this case was not to be the dexterous controller. It was to construct and operate the computational process that produced one.\n\nThis is a pattern we have also studied in [HARBOR](https://arxiv.org/abs/2606.08610), where we propose a harness for robot RL, where the task is decomposed into bounded stages including environment setup, reward design, algorithm tuning, and evaluation, with explicit validation gates and persistent experimental artifacts. Across six benchmarks and sixteen tasks in manipulation, locomotion, and bimanual dexterous control, the resulting policies match or improve over expert default configurations and transfer to real robots.\n\nI find this scientifically at least as interesting as direct control. A sufficiently capable general model may not need to directly acquire every specialized capability. It may instead become capable of identifying what machinery is needed, constructing it, evaluating what happens, and revising the process until a specialized solution emerges.\n\n### Scale has not made structure disappear\n\nAcross these examples, something is easy to miss. Many of the strongest results are not cases in which scale has made the existing computational structure disappear. Astra is operating through it.\n\nBenchCAD provides a structured language of geometric primitives, parameters, constraints, and operations. Blender provides objects, transformations, scene structure, cameras, materials, and a renderer. In robotics, Cartesian and SE(3) action spaces structure motion. Kinematics and IK encode relationships between task and configuration space. Simulators encode dynamics. Controllers structure how desired motions are realized. RL provides an algorithmic process for searching for specialized policies.\n\nBut there is another distinction that matters here. So far I have mostly described structure around the model. In robotics and machine learning, we have also spent decades studying structure inside learned representations.\n\nFor embodied systems, this can mean equivariance to the geometric symmetries of the workspace, representations respecting morphological symmetries of the robot, explicit relational representations of objects and contacts, physical constraints on learned dynamics, or representations grounded in multimodal interaction. These are not simply tools that a model calls after perception. They determine how observations are represented, which transformations the representation respects, and which information about an interaction is preserved in the first place.\n\nThe Astra demonstrations provide striking evidence that sufficiently capable models can exploit external computational structure. They tell us much less about whether structure inside learned representations has become unnecessary.\n\nI do not think these two forms of structure are interchangeable. An IK solver can translate an end-effector target into joint configurations, but it does not make the visual representation equivariant to a change of reference frame. A simulator can propagate a known state forward, but it does not tell the model which aspects of an observed interaction should be represented when state and physical parameters are uncertain.\n\nThis makes the relationship between structure and scale considerably more interesting than a simple opposition between hand-designed structure and end-to-end learning.\n\nThis connects to a related argument that [Ken Goldberg](https://x.com/Ken_Goldberg/status/2100986412762087909?s=20) has been making around what he calls agentic robotics. He points to these systems as a possible bridge between model-based and model-free robotics, where a general model can make use of classical models, control methods, learned policies, and other computational tools without requiring one of these paradigms to replace the other. I think this is an important perspective on the current results.\n\nMy question is perhaps one step further. If increasingly capable models can not only use existing structure, but select, construct, and eventually improve the structures through which they solve problems, then the boundary itself may become dynamic. Which structure should organize what the model learns and represents, which can remain external, and which can increasingly be discovered or constructed by the model itself?\n\nIn robotics there is an additional consideration. **Some structure needs to remain outside the model not because the model lacks capability, but because physical systems require independent mechanisms for robustness, verification, and safety.**\n\n### From static geometry to interactive physical structure\n\nUnderstanding this boundary becomes considerably harder when we move from static 3D geometry to physical interaction.\n\nBenchCAD and many of the Blender demonstrations primarily concern shape, pose, composition, and appearance. Robotics requires more. Geometry evolves in time under contact, friction, compliance, and the robot’s own actions. The relevant scene is therefore not simply 3D. It is an interactive 4D process, where intervention changes what happens next.\n\nThe current Astra results give us limited evidence about this level of physical understanding. In direct-control experiments, much of the physics is resolved by the world itself. The robot acts, the cameras report what happened, and the model reacts. In RL experiments, the physics is supplied by the simulator. Neither setting tells us whether the model has an internal representation of contact dynamics, whether it can infer mass, friction, or compliance from observation, or whether it can predict how these properties affect the consequences of an intervention.\n\n[StationeryBench](https://openai.robocurve.org/stationerybench/) is informative here. Astra completes only 7 out of 100 trials, although its mean progress score is 46 out of 100 compared with 12 for MolmoAct2. MolmoAct2, a vision-language-action model, was run zero-shot on scenes outside its training distribution, so the comparison says more about Astra’s absolute performance than about the two approaches. Per task, Astra succeeds 5 out of 20 times at uncapping a marker, 1 out of 20 at extracting the middle sticky note from a stack, 1 out of 20 at handing a ruler between the arms, 0 out of 20 at opening a box, removing an eraser, and closing the lid, and 0 out of 20 at pouring paper clips into a lifted bowl. The evaluation itself has limitations, but the failure modes are nevertheless interesting.\n\nPouring, handover, insertion, and articulated manipulation require more than estimating where objects are. They require anticipating how objects respond to force and contact.\n\nThis is where structured representations become particularly relevant. Geometric equivariance and symmetry, explicit object and interaction representations, contact constraints, physically structured dynamics, and multimodal information from vision, proprioception, and touch are all attempts to capture structure that matters for physical interaction.\n\nThe question is not whether every such prior should be hard-coded. It is whether sufficiently capable general models learn these structures implicitly, whether they can discover and use them when needed, whether some should organize learning as inductive biases, and which can instead remain external in simulators, controllers, and tools.\n\n### Safety, robustness, and interaction remain open\n\nThe benchmark results also expose questions that the current demonstrations largely do not address.\n\nRoboCurve is again useful because the experimental conditions are explicit. The Astra runs used a 25% speed cap, a 20-call budget, and default safety guardrails. Objects were reset manually between trials. These are entirely reasonable choices, but they mean that we are observing a model inside a system that already constrains how it can act.\n\nAstra is also not a high-frequency reactive controller. It observes, reasons, issues an end-effector command, and receives another observation. Reactive safety therefore remains the responsibility of lower-level control and the surrounding system. We know very little about what happens under unexpected contact, slipping objects, perturbations during execution, or distribution shifts.\n\nHuman-robot interaction is absent from these evaluations altogether. There is no person inside the workspace whose behavior changes in response to the robot. Yet interaction with people introduces additional requirements: bounded forces, predictable and legible motion, uncertainty about human actions and intent, and the ability to incorporate corrective feedback during execution.\n\nAgain, this is partly a question about where structure should reside. Some safety constraints should clearly remain externally enforceable regardless of how capable the model becomes. Other capabilities, such as representing uncertainty, anticipating the consequences of contact, or reasoning about another agent’s response, require something from the model itself.\n\nStrong visuospatial reasoning is not by itself evidence for safe interactive intelligence.\n\n### But what exactly was the experiment?\n\nThere is also a methodological issue that deserves more attention. For many viral single-run demonstrations, we simply do not know enough about the experimental conditions. The complete prompt is not always available, nor are the system instructions, accessible tools, retrieved context, intermediate interventions, or unsuccessful runs.\n\nThis criticism should not be applied indiscriminately. RoboCurve is close to the standard we should expect. It publishes its harness, prompts, per-trial transcripts, videos, and rerun files. For Astra, it reports medium thinking effort, the 20-call budget, the 25% speed cap, default safety guardrails, and the version of the open-source [Inspect Robots](https://github.com/robocurve/inspect-robots) harness used.\n\nThis information matters because the prompt and harness are part of the experiment.\n\nThe pen-spinning case illustrates this well. The published prompt did not simply ask for a robotic hand that spins a pen. It specified Isaac Lab as the simulator, the Sharpa hand as the embodiment, and that the pen mesh should be created by the agent, and it granted the agent permission to search the web and download papers. The task was therefore posed at a level of abstraction where the simulator, the platform, and access to the literature were given. That is a legitimate experiment, but it is a different experiment from one in which the agent must select these itself, and the final video would look the same in both cases while the inference required from the model is radically different.\n\nPrompt, context, retrieval, tool access, action interface, feedback, and harness configuration should therefore be treated as experimental methodology. Without this information, a successful demonstration establishes that the complete setup worked. It tells us considerably less about why.\n\nThis also matters for memorization and recomposition. I do not think we currently have evidence to claim that Astra is simply memorizing robot trajectories. But many demonstrations also do not separate a transferable spatial-action capability from recomposition of familiar task structures. That requires deliberately designed experiments.\n\n### Where does the capability live?\n\nThis leads to what I think is one of the deeper questions raised by these results.\n\nSuppose a successful behavior results from a foundation model operating with retrieved literature, existing code, a simulator, a robot model, an RL algorithm, IK, a controller, and substantial compute. Where exactly should we locate the capability?\n\nThe importance of the surrounding system is visible even outside robotics. [ARC Prize](https://arcprize.org/blog/astra) reports Astra scoring 62.7% on ARC-AGI-3 with its Standard harness and 99.9% with the Provider Adapter harness. The model is the same, while the interface around it changes substantially.\n\nOur experience with HARBOR and [Nautilus](https://arxiv.org/abs/2605.11665) has made us particularly sensitive to this issue. In Nautilus, we asked whether a general coding agent, given a single request such as evaluating a policy on a benchmark, could produce a correct and reproducible robot-learning workflow. Robot learning remains fragmented across policy families, benchmarks, and physical platforms, and each pairing encodes assumptions about action spaces, observations, control rates, and coordinate conventions. Typed interfaces, procedural priors, and validation gates can therefore substantially change whether the workflow succeeds at all.\n\nThis does not mean that the harness is the intelligence. It means that the model alone may increasingly be the wrong unit of analysis. We need to understand what is contributed by the model, what is contributed by the representations and algorithms around it, and how these components interact.\n\n### What experiments would actually tell us what we are seeing?\n\nRather than another collection of impressive videos, I would like to see experiments designed explicitly to separate these effects.\n\nFirst, we should vary prior exposure and information access. Give the same model no external information, documentation, papers, repositories, or unrestricted retrieval. Use private or procedurally generated tasks where possible.\n\nSecond, we should vary external structure while holding the model fixed. Compare joint-space control with end-effector control, then add IK, simulators, planners, motion primitives, pretrained policies, or other tools.\n\nThird, we should vary internal representational structure. Compare generic representations with geometrically equivariant ones, unstructured scene representations with explicit object and interaction representations, unconstrained learned dynamics with physically structured dynamics, and vision-only state with multimodal representations grounded in proprioception and touch.\n\nFinally, we should vary physical interaction itself. Change embodiment, coordinate conventions, object geometry, mass, friction, contact configuration, and dynamics. Inject perturbations during execution. Test physical-property inference and prediction under intervention. Extend the same controlled variation to safety and eventually to interaction with a human collaborator.\n\nThen we can ask much more meaningful questions. Does the model possess a transferable spatial-action representation? Which structures improve systematic generalization? Can the model discover the appropriate structure itself? When does it need a specialized policy? Which structures must organize the learned representation, and which can remain external?\n\nHolding scale fixed, what changes when we alter the available structure? Holding structure fixed, what changes as we scale the model?\n\nThose are experiments I would find considerably more informative. And I invite the academic community to collaborate to conduct these experiments and answer the deeper research questions.\n\n### Scientific provenance is also part of the problem\n\n[Toru Lin](https://x.com/ToruO_O/status/2101105891676168698?s=20), [Guanya Shi](https://x.com/GuanyaShi/status/2101161990475354616?s=20), [Jitendra Malik](https://x.com/JitendraMalikCV/status/2101158162305036429?s=20), and others have raised important questions about scientific credit and provenance when agents can search papers and repositories, integrate existing ideas, run experiments, and return a working result.\n\nThere is a direct connection to the methodological problem above. If an agent reads papers, inspects repositories, tries different formulations, runs experiments, rejects failures, and eventually produces something that works, then the retrieval and experimental trajectory are part of the evidence. We should know what it retrieved, what it reused, what it modified, what failed, and what was actually new.\n\nInterestingly, agentic systems could eventually make this more transparent than human research rather than less. Every retrieval, code modification, experiment, and revision could in principle be recorded. But for that to happen, we need to stop treating the prompt, retrieved context, and agent trajectory as implementation details.\n\n### So, what are we actually seeing?\n\nI started looking at these results because I wanted to understand whether the recent demonstrations really support some of the broader claims being made about robotics. I still do not think they show that scaling has solved physical intelligence. But I also think something important has changed.\n\nGeneral models are becoming surprisingly capable at moving between perception, spatial reasoning, action, code, tools, simulation, and learning. Sometimes they directly generate the action. Sometimes they exploit a structure we have provided. Sometimes they construct the process through which a specialized solution is found. These are different capabilities, and our evaluations need to distinguish them.\n\nFor a long time, structure and scale have often been discussed as competing approaches. I am wondering whether this is still the right framing.\n\nPerhaps scale does not make structure obsolete. Perhaps scale is becoming capable of using structure.\n\nBut that leaves the harder scientific question unresolved: which structure should be built into the learned representation as an inductive bias, which should be learned from data without prior constraint, which can be supplied externally through tools and algorithms, and how do these boundaries move as models keep scaling?\n\nFor embodied intelligence, this means concrete questions about geometric and morphological symmetry, object and contact representations, dynamics, uncertainty, multimodal interaction, safety, and eventually interaction with people.\n\nThe Astra results are exciting because they make these questions harder to ignore. Before deciding what has been solved, we should first understand what we are actually seeing.\n\n### References\n\nARC Prize. (2026, September). OpenAI’s GPT-6 Astra on ARC-AGI-3. [https://arcprize.org/blog/astra](https://arcprize.org/blog/astra)\n\nJin, Y., Guo, J., Jia, X., Deng, Y., Li, Z., Liu, H., Liao, W., Prasad, V., Franzius, M., Neumann, G., & Chalvatzaki, G. (2026). Nautilus: From One Prompt to Plug-and-Play Robot Learning. arXiv:2605.11665. [https://arxiv.org/abs/2605.11665](https://arxiv.org/abs/2605.11665)\n\nLi, Z., Jin, Y., Liu, X., Liu, P., Prasad, V., D’Eramo, C., & Chalvatzaki, G. (2026). HARBOR: A Harness Framework for Agentic Robot Reinforcement Learning. arXiv:2606.08610. [https://arxiv.org/abs/2606.08610](https://arxiv.org/abs/2606.08610)\n\nMa, Y. J., Liang, W., Wang, G., Huang, D.-A., Bastani, O., Jayaraman, D., Zhu, Y., Fan, L., & Anandkumar, A. (2024). Eureka: Human-Level Reward Design via Coding Large Language Models. ICLR 2024. arXiv:2310.12931. [https://arxiv.org/abs/2310.12931](https://arxiv.org/abs/2310.12931)\n\nMenon, A., Machcha, S., Zou, S., Chan, T. K., & Chooi, J. (2026, September 4). GPT-6 Astra on robotic manipulation. RoboCurve. [https://openai.robocurve.org/gpt-6-astra/](https://openai.robocurve.org/gpt-6-astra/)\n\nOpenAI. (2026, September 3). GPT-6 Astra: A new generation of intelligence. [https://openai.com/index/gpt-6-astra/](https://openai.com/index/gpt-6-astra/)\n\nRoboCurve. (2026). Inspect Robots (Version 0.58.0). [https://github.com/robocurve/inspect-robots](https://github.com/robocurve/inspect-robots)\n\nZhang, H., Liu, K., Chen, M., Li, L., Yang, S., Peng, C., & Chen, H. (2026). BenchCAD: A Comprehensive, Industry-Standard Benchmark for Programmatic CAD. arXiv:2605.10865. [https://arxiv.org/abs/2605.10865](https://arxiv.org/abs/2605.10865)\n\nZhang, Z. J., Machcha, S., Zou, S., Chan, T. K., & Chooi, J. (2026, September 10). GPT-6 Astra vs MolmoAct2 on bimanual robotic manipulation: StationeryBench. RoboCurve. [https://openai.robocurve.org/stationerybench/](https://openai.robocurve.org/stationerybench/)\n\nGoldberg, K. (2026, September 18). *Reflections on agentic robotics and the relationship between model-based and model-free robotics.*[X post](https://x.com/Ken_Goldberg/status/2100986412762087909?s=20).\n\nZhu, W. (2026, September 16). GPT-6 Astra test 4/n: Sharpa-hand pen-spinning RL in Isaac Lab, autonomous run by Chengyang Li [Post]. X. [https://x.com/walterzhu8/status/2100212420840989112](https://x.com/walterzhu8/status/2100212420840989112)", "url": "https://wpnews.pro/news/what-we-are-seeing-with-gpt-astra-in-robotics", "canonical_source": "https://twitter.com/GeorgiaChal/status/2101335513558929868", "published_at": "2026-09-19 20:22:52+00:00", "updated_at": "2026-09-19 20:54:25.224860+00:00", "lang": "en", "topics": ["artificial-intelligence", "robotics", "large-language-models", "ai-research", "generative-ai"], "entities": ["OpenAI", "GPT-6 Astra", "BenchCAD", "University of Virginia", "GPT-5.6 Sol", "Claude Fable 5.1", "CadQuery", "Blender"], "alternates": {"html": "https://wpnews.pro/news/what-we-are-seeing-with-gpt-astra-in-robotics", "markdown": "https://wpnews.pro/news/what-we-are-seeing-with-gpt-astra-in-robotics.md", "text": "https://wpnews.pro/news/what-we-are-seeing-with-gpt-astra-in-robotics.txt", "jsonld": "https://wpnews.pro/news/what-we-are-seeing-with-gpt-astra-in-robotics.jsonld"}}