AA-Robotics v0.1

We are launching AA-Robotics v0.1, our research preview that measures frontier language model performance on understanding, planning, and acting in the physical world. In v0.1, each model is given control of a 6-axis robot arm (5 joints and a gripper) and a simple objective. We’re releasing results for our first eight tasks today.

Introducing AA-Robotics v0.1 #

Since the beginning of 2026, frontier language models have made enormous progress in their ability to control robot arms. We now believe that the future of robotics is likely to involve strong generalist agents, and conclude that tracking progress of language models for robotics tasks is going to be important - even if frontier language models are unsuitable for production robotics use on their own.

AA-Robotics v0.1 is our research preview of the simplest experiment we have designed to test this - language models in coding agent harnesses directly controlling a robot arm. Each model is given control of a 6-axis robot arm (5 joints and a gripper) and a simple objective for each task.

We find that frontier language models in coding agent harnesses have indeed recently become capable of directly manipulating robot arms for simple tasks. Today, they are far too slow and expensive for any kind of production usage - nearly two orders of magnitude worse than human baselines. We expect this to change quickly.

GPT-6 Astra places first in AA-Robotics v0.1 with a score of 69, followed by GPT-6.1 Sol, Gemini 4 Argon, Claude Opus 5.5 and Claude Fable 5.1. GPT-6 Astra took 30 minutes and cost $16.36 on average per task.

AA-Robotics Score

Each attempt is scored out of 100: completing the task scores 100, and a partial result earns points for the progress made.

The eight tasks #

AA-Robotics v0.1 initially includes eight tabletop tasks, modeled on everyday activities, to evaluate a range of robotics capabilities: control and dexterity, such as precise grasping and handling objects, spatial reasoning, such as understanding how objects interact with one another, and adaptability, measuring how models respond to the unexpected.

Press the red button

• Goal:
• Press the button with the gripper.
• Pass criteria:
• The button flashes.

Place the crocodile toy in the rectangle

• Goal:
• Place the crocodile inside the rectangle.
• Pass criteria:
• The crocodile is entirely within the rectangle.

Pour the drink into the cup

• Goal:
• Pour the drink from the bottle into the cup.
• Pass criteria:
• Drink is in the cup, none is spilled, and nothing is broken.

Tidy the blocks into the box

• Goal:
• Put all four blocks in the box, then put the lid back on.
• Pass criteria:
• All blocks are inside the box with the lid flush.

Complete the shape matching challenge

• Goal:
• Place the blocks into the board so that it matches the picture.
• Pass criteria:
• Blocks in the board match the picture.

Write hello world

• Goal:
• Write “hello world” on the paper with the marker, inside the black box.
• Pass criteria:
• “hello world” is legible and inside the black box.

Build the tallest tower

• Goal:
• Make the tallest tower out of the foam blocks.
• Pass criteria:
• Tower is freestanding and max achievable height.

Measure the box

• Goal:
• Measure the length, width, and height of the box.
• Pass criteria:
• Each dimension is within 2 mm of the correct size.

Results #

Results per Task

- Full score (100)
- Partial (1 to 99)
- Zero (0)

Each attempt is scored out of 100: completing the task scores 100, and a partial result earns points for the progress made.

GPT models run in Codex, Claude models in Claude Code, and Gemini 4 Argon in Antigravity, all at the high effort setting.

Task

Model

Press the button with the gripper.

Full score in 2:12

How AA-Robotics works #

Models run in an isolated container with brief instructions: how to reach the robot arm (via a simple HTTP API), which arm it is (a Hiwonder NexArm, with no further specifications), and how to submit an attempt as complete. They can view the scene through four cameras, three mounted around the table and one on the arm’s wrist, and can write and run code in the container to plan and send movements.

Once a model has submitted a task for review, the end state is graded against the task’s pass criteria. A pass scores 100, with points awarded for partial success in the case of an incomplete task. All task attempts are zero-shot.

Time & cost #

Today’s language models are roughly 100 times slower than a human on AA-Robotics v0.1, taking about half an hour per task compared with a human average of 20 seconds. In aggregate, four of the five models tested look similar, at about 30 minutes per task, while Gemini 4 Argon is the slowest at 52 minutes. However, these averages hide large differences between tasks.

GPT-6 Astra pressed the red button in 2.2 minutes, whereas Claude Fable 5.1 spent about ten minutes understanding and planning before its first action. The largest difference came on the crocodile placement task, where GPT-6.1 Sol declared itself complete after five minutes, compared with 45 for Claude Opus 5.5, a ninefold difference. Across all tasks, Gemini 4 Argon was the slowest to act, spending an average of 21 minutes observing and reasoning before its first action, nearly twice as long as any other model.

Time per Task

AA-Robotics Score vs. Time per Task

The gap between humans and models is narrower on cost: the most cost-effective model costs about ten times as much as a person paid the average US hourly wage, against nearly 100 times as long on time. Between models, GPT-6.1 Sol is the cheapest, at $1.93 per task on average, while GPT-6 Astra, the most expensive, costs $16.36, more than eight times as much as Sol, for a score only four points higher (69 against 65). Given the novelty of the arm and scene, and the large volume of visual information the models must process, most of the cost lands on input: 70% of the spend goes on context.

Cost per Task

AA-Robotics Score vs. Cost per Task

Human cost is the by-hand task time priced at $37.81 per hour, the average hourly earnings of all employees on US private nonfarm payrolls (US Bureau of Labor Statistics, preliminary data for September 2026).

How each model spent its time #

As expected, models spend a large portion of the initial task time calibrating the arm and understanding the scene, continuing this up until taking the first action. After this point, the majority of time is spent planning movement, with relatively short, precise arm movements between each longer planning step. Camera views and note writing are relatively small fractions of the overall time.

Interestingly, models often continue to refine their attempts long after reaching a passing state. GPT-6.1 Sol, for example, builds a passing tower after ~17 minutes, only to subsequently fail the task. It remains to be seen how models will improve their ability to reason about task completion in physical space.

Activity Through Each Run

0:00 elapsed0 of 5 finished

• Understanding the Scene, Arm, and Movement

• Planning Movements

• Reviewing Cameras

• Arm Moving

• Other

• First Action

• First Reached a Passing State

• Irrecoverable Mistake

Other covers waiting on running code, writing notes, and the final report.

The first action is the first press, grasp, push, pour, or pen stroke of a run.

An irrecoverable mistake means the model can no longer achieve full marks.

Gemini 4 Argon's activity times are estimated: Antigravity does not timestamp each step, so steps are placed between the arm commands, which are timed exactly, in proportion to their tokens.

Physical accuracy #

Grasping and fine manipulation were key skills to test in AA-Robotics v0.1, as they point the way not only to more precise control tasks, but more importantly to tool use and the far wider range of capabilities it unlocks.

GPT-6 Astra and Claude Fable 5.1 performed best with the highest proportion of successful attempts (~45%), indicating a degree of close control, whilst GPT-6.1 Sol, scored worst, at 28%. Gemini 4 Argon fell in between at 36%, with many of its misses closing just above or beside the block it was reaching for.

Outcome of Every Grasp Attempt

• Successful Grasp
• Missed, Partial, or Wrong-Object Grasp

Grasps were attempted in the Place Crocodile, Pour Drink, Tidy Box, Match Shapes, Build Tower, and Measure Box tasks. Successful is defined as holding the intended object through its lift or carry, and missed, partial, or wrong-object where the gripper did not close on the object, closed on it before it slipped, dropped, or could not be lifted, or took a different object by mistake.

Mistake wall #

Hover over a clip to see what went wrong and click a clip to enlarge it. We’ve highlighted a few of our favorite examples!

Misjudging success or failure - and when to stop

Poor physical reasoning

Missed grasps, misplacements and drops

Read the latest

First AA-AgentPerf-Local results for an NVIDIA RTX Spark N1X laptop

An NVIDIA RTX Spark N1X laptop completes our local agent workload on Qwen3.8 27B in 18.7 minutes.

October 15, 2026

NVIDIA Releases Nemotron 3.5 Super VL and Nemotron 3.5 Bolt 74B A9B

Nvidia’s Nemotron 3.5 Super VL is now the leading US open weight model, scoring 26 on the Artificial Analysis Intelligence Index

October 15, 2026

Artificial Analysis Decision Index v1.0

The Artificial Analysis Decision Index benchmarks decision models, a new category of models that answer structured decisions in a single call with a probability for each option. Version 1.0 combines four evaluations across classification, tool calling, knowledge, hallucination, and reasoning.

October 13, 2026