We ask not only how capable GPT-6-Astra is, but what its capabilities imply for the field: we probe its limits on 12 benchmarks, dissect the harness behind it, validate it on eight real-robot platforms, and prototype what the next generation of embodied benchmarks should measure.
Probing the capability upper bound of GPT-6-Astra across 12 benchmarks
Sort by resultSort by gain over SOTAGroup by embodiment
GPT-6-Astra under a protocol that allows in-context learning on the training set and extends the execution budget up to 10×. Published baselines are reported under the standard budget, collected as of October 12, 2026. *Result with a VLA model. ‡Zero-shot, without access to training data.
In-context learning protocol
ICL
In-context learning from the training set
Successful demonstrations are conventionally accessible when training smaller task-specific policies, so GPT-6-Astra may use them too. It studies the training data and builds a frozen knowledge package of task guidance, motion and visual references, executable helpers, recovery procedures and metadata, then retrieves and adapts these resources to each test scene.
10×
A substantially larger execution budget
We allow up to 10× the official step limit, since standard step limits can significantly underestimate GPT-6-Astra. Its emerging capability for failure recovery lets it translate additional steps into higher success, nearly saturating SIMPLE and the LIBERO family. Extensions continue the same episode — no resets.
In-context learning: what GPT-6-Astra keeps Contents of an ICL knowledge package
Knowledge package Kwritten by GPT-6-Astra during ICL preparation · frozen before testing
Success rate versus step budget. The budget is expressed as a multiple of each benchmark's official step limit; dots show observed success rates and pink curves show fitted trends. We evaluate budgets up to 10× the official limit (1× for BEHAVIOR). The gray line marks the upper bound, which assumes that every episode that exhausts the evaluated budget without succeeding has the potential to succeed with an unlimited budget.
Limitations and failure modes
←1 / 5→
1LimitationTask-specific ICL2LimitationTime & cost3Failure modeVerifying physical success4Failure modeConsequential actions5Failure modeProgress over long horizons
Limitation
ICL packages are largely task-specific, which limits transfer
Packages are built from each benchmark's training data, so their benefit weakens once test tasks move away from it. Packages that generalize beyond their source tasks remain an open direction.
RoboDojo · Open group
Lowest SR
No task-specific demonstrations; it relies on a package studied from the other four groups. GPT-6-Astra + ICL still benefits from more interaction, but attains the lowest success of all groups.
RoboLab · no training set
69.2%
No task-level package can be built (zero-shot, 10× budget). GPT-6-Astra leads the best listed baseline (42.9%), yet its absolute success remains moderate. Close coverage
95%
Where test tasks follow the training distribution — LIBERO family, SIMPLE, VLN-CE — demonstrated motions or routes are retrieved almost directly and success approaches saturation: above 95% on the LIBERO family and SIMPLE, around 90% on VLN-CE at 10×.
SIMPLE · zero-shot → ICL
70.8 → 90.8%
Standard-budget success on the four common tasks, with shorter successful episodes under ICL.
Limitation
Time and cost remain high
Even nearly solved benchmarks are expensive: about 1M tokens and nearly an hour per LIBERO-Long++ episode, roughly 9M tokens and nearly two hours on LIBERO-Ultra-Long. Extending the budget adds more, concentrated in difficult episodes that often still fail: on RoboDojo, an extended-budget episode takes more than three hours on average.
About 98% of tokens are cached input, so 1M tokens ≈ 1 USD. Inference latency is not counted in the step budget, so this cost also limits deployment where real-time control is required.
Average per-task cost by benchmark
Tokens (M)
Time (min)
Failure mode
Verifying physical success
GPT-6-Astra can tell a task is unfinished but not which physical condition is unmet, so its corrections fail to converge. Looking done, or reporting done, is not the same as meeting the task's contact, support and alignment conditions.
Failure mode
Recovering from consequential actions
Each attempt changes the state that recovery starts from, and some changes cannot be undone. A failed prerequisite blocks every later subtask while retries drain the budget.
Recovery works when GPT-6-Astra changes strategy rather than repeating itself, as in DexJoCo Fold Glasses, where a side approach and a test lift complete the task.
←
Pour liquid into cupMulti-room navigation
→
Failure mode
Preserving progress over long horizons
Earlier progress must stay intact while later actions run, yet a later motion can undo it, and small errors in grasping, placement or navigation compound across stages. GPT-6-Astra has to keep rechecking what it has already done rather than treating completed subgoals as permanently resolved.
Next 2 · Time & cost→
02 · Harness Ablation
Harness Matters: An Ablation
Token cost versus success rate on RoboDojo
AllA · System 1–System 2B · Action interface
42 RoboDojo tasks; performance is the unweighted mean of category success rate. Dashed lines cross at the Agentic GPT-6-Astra baseline; upper-left is better on both axes. Standalone VLA (π0.5, 6.91%) uses no GPT tokens and is not plotted. Hover a point to compare it with the baseline.
A · System 1 – System 2 collaboration
Supervision pays off, but every review costs tokens
Letting the VLA run alone is insufficient; GPT-6-Astra reviewing its proposed actions works, but costs tokens. Reviewing four at a time saves 63.5% of tokens over per-action review, and adding ICL gives the best result — at about 6.9× the tokens of the best closed-loop setting without ICL.
5.83%open-loop25.83%closed-loop · L = 454.17%ICL + VLA B · Action representation & chunking
Interfaces should match how the model reasons
Directional commands help and joint angles hurt. Letting the model choose its chunk size beats a fixed chunk in success rate, at a higher token cost.
What Still Separates Benchmark Success from Deployment
01 / ZERO-SHOT
No demonstrations, no knowledge package
GPT-6-Astra performs as a generalist, without task- or robot-specific training. 02 / HUMAN PRIORS
Structured context written by a human
Tool schemas, joint limits, camera placement, safety constraints, hand presets and, on some platforms, task-level guidance such as admissible grasp points — no high-level semantic skills such as pick, place or insert.
03 / WALL CLOCK
Budgeted in time, judged by a person
Trials are normally limited to 60 minutes. An operator holds an emergency stop, and success is judged from the physical outcome, not the model's own claim.
Embodiment#Main questionTasks
Single arm, gripperFranka 3, Franka Panda, PiPER + Pika gripper
3Do simulation results transfer, from rigid to deformable and precise tasks?
Stacking and container nesting
Cloth and towel folding
Plug insertion
Dual arm, grippersAgileX Piper
1Can GPT-6-Astra coordinate two arms on harder zero-shot tasks?
Ordered folding
Flower insertion
Bimanual holding
Sorting
Dual arm, dexterous handsTianji M6S
1Can GPT-6-Astra control high-DoF hands? Toy and bottle grasping
Wheeled mobile manipulatorGalaxea R1 Pro, AgileX Tracer + xArm7
2Can GPT-6-Astra act over room-scale distances and long horizons?
Find washer and press button
Flower collection
Tray arrangement
Navigation and search
HumanoidUnitree G1
1Can GPT-6-Astra act as System 2 over low-level locomotion and arm control?
Tabletop pick-and-drop Turn, approach and stop
Findings and what changes in the real world
←1 / 4→
1FindingOne model, every embodiment2FindingActive vision3FindingSystem 2 over System 04What changesPhysical limits & missing demos
Finding 1
One model drives every embodiment; success depends on the task
Without platform-specific training, the same model controls all eight platforms. Rigid pick-and-place is reliable and matches simulation, whereas deformable objects, precise contact and long-horizon mobile tasks remain markedly harder. A second arm extends the range of zero-shot tasks, and GPT-6-Astra re-plans from execution feedback when the scene changes.
What GPT-6-Astra can do depends on what it can see
In flower insertion, the gripper holds the stem vertically, so its own wrist camera cannot see the vase mouth. On a single arm the insertion fails. On the dual-arm Piper, GPT-6-Astra moves the free arm until its wrist camera sees the vase mouth, then completes the insertion — the viewpoint becomes part of the action space. Teleoperated demonstrations rarely contain this active vision.
Finding 3
GPT-6-Astra as System 2 over existing System 0 motor control
On a real Unitree G1 with Wuji hands, GPT-6-Astra commands arm-joint targets and binary hand states for manipulation (2/3), and turns and walks through the vendor velocity interface (3/3) — without a task-trained System 1 policy. This establishes feasibility under slow, feedback-driven supervision rather than reliable real-time visuomotor control.
What changes
What changes in the real world
In simulation, extra steps and failed attempts cost nothing, and demonstrations of the test task exist. On real robots neither holds: physical constraints limit extended execution, and missing demonstrations reshape in-context learning.
Physical constraints
Extended execution is not free on real hardware
Extra steps and retries cost nothing in simulation. On real robots, protective stops and hardware faults end runs, and elapsed time adds tokens, risk, hardware stress and supervision.
2 / 3PiPER plug-insertion trials ended by a joint collision fault62 °Chand temperature guard~90 %of G1 time deliberating
Missing demonstrations
Context quality decides the outcome
Human-written context can stand in for demonstrations, but it is expert prompt engineering. Building context from other sources is an open problem.
Stated as a person would, e.g. “Could you make me breakfast?” Judged on the request (Intent) and on unstated obligations (Duty).
02 / NO DEMOS
Experience acquired at test time
No in-domain demonstrations. The agent learns the scene through its own exploration.
03 / WORLD CLOCK
A world that keeps running
Thinking takes world time, as on a real robot. The judge watches the whole trajectory, even after the agent says it is done.
1
Open-source assetspublicly available object assets
2
Layouta floor plan composed into an interactive home
3
Agent-assisted placementcoding agents place items at real-world scale on named supports, under human review
4
Physics bindingarticulation, liquids, cloth, contact, heating and cooking
5
Task designa task states only a need; judged on Intent and on unstated Duty
6
Evaluationonboard cameras only; deliberation advances world time
Interactive homeNot loaded
Load interactive scene
The scene couldn’t load.
Please reload the page.
Room tour: a camera fly-through of the apartment, from day to night.Laundry: carry the basket, load the washer, add detergent, start it.Coffee: brew a cup with the capsule machine.Breakfast: prepare a meal with the air fryer and toaster, while the world clock keeps running.
←→
30 interaction clips in the home: doors, drawers, appliances, cooking and tidying. Drag or use the arrows to browse.
←
3D ViewerRoom tourLaundry taskCoffee taskBreakfast taskGallery
→
Findings in the household and open challenges
←1 / 4→
1FindingLoses track of the home2FindingUnfamiliar fixtures, own body3FindingOutdated observations4Open challengesJudge, fidelity, time, cost
Finding 1
It attends to the object in hand and loses track of the rest of the home
Its own arm and the states it has changed seem to enter its decisions only when they block the next step. A final-state check would miss this, so the judge tracks the whole trajectory.
Finding 2
Without demonstrations, it does not work out how unfamiliar fixtures move or where its own body has to be
It plans for the gripper and handle, not its whole body, and repeats and adjusts the same motion. In Section 01, the demonstrations it distills in context show both, so this weakness does not appear there.
Finding 3
It bases decisions on observations that become outdated while it thinks
It rightly decides to turn off the burner once the egg sets, but thinking takes world time: the burner goes off about 90 s late and the egg chars.
World clock
Too late
The simulators of Section 01 wait for the model. This world does not, so the action arrives too late.
Open challenges
What a next-generation benchmark still has to solve
Measuring deployment rather than isolated tasks raises four open problems for the judge, the simulator and the evaluation budget.
01 / JUDGE
Evidence for the judge
Simulators model key physical facts heuristically; vision-language success detectors reach at most 60% balanced accuracy on contact-rich assembly.
02 / FIDELITY
Simulation fidelity
Modeling errors look like agent behavior; simulation has been shown to predict real-robot results only for short tabletop tasks.
03 / REAL TIME
Real-time evaluation
If the world keeps running, scores depend on latency, hardware and service load. 04 / COST
Evaluation cost
Hour-long deployments are expensive to repeat; even short robot trials are typically run too few times to separate policies reliably.
Next 2 · Unfamiliar fixtures, own body→
A clear leap in embodied intelligence — but not its endpoint.
Strong harness design with in-context learning and a sufficient execution budget can push many isolated benchmarks close to saturation. The remaining challenges shift toward transfer across tasks, long-horizon reasoning, failure recovery, and real-world constraints on time, cost and safety. The next frontier is not simply to solve more isolated tasks, but to build embodied systems that operate reliably and efficiently under realistic deployment conditions.