{"slug": "frontier-models-score-under-2-100-controlling-a-simulated-humanoid", "title": "Frontier models score under 2/100 controlling a simulated humanoid", "summary": "Robo-bench, a benchmark evaluating general-purpose AI agents on two robot platforms, found that no model solved any of its 11 humanoid tasks and none averaged more than 2 out of 100 on them, while GPT-6.1 Sol averaged 51 out of 100 on the 13 arm tasks and solved four. The benchmark runs a 7-DoF Franka Emika Panda arm and a 29-DoF Unitree G1 humanoid in MuJoCo, with each episode lasting three minutes of wall-clock time so deliberation consumes the episode; Grok 4.7 sent no motor command in 16 of its 24 episodes and Gemini 4 Argon in 15, while both GPT models moved the robot in every episode.", "body_md": "## What a run looks like\n\nThe three public tasks: two from the scored runs, the ninja course from a development run. Every frame is a replay of the motor commands the agent sent, re-rendered after the episode.\n\nThe held-out tasks are not shown, for the same reason they are not named.\n\n## Key Takeaways\n\n- The arm is partly within reach and the humanoid is not. GPT-6.1 Sol averages 51 out of 100 on the 13 arm tasks and solves four of them. No model solves any of the 11 humanoid tasks, and none averages more than 2 out of 100 on them.\n- Thinking time is part of the score. The simulation runs in real time from the moment the agent starts, so deliberation spends the episode. Grok 4.7 never sent a single motor command in 16 of its 24 episodes and Gemini 4 Argon in 15, Claude Opus 5.5, Claude Fable 5.1 and Muse Wonder Full in 6 each; both GPT models moved the robot in every episode.\n\n## Background\n\nThe agent gets everything a robot designer provides: the robot’s datasheet, its kinematic model, its sensors, motors and cameras, and nothing more. We want to evaluate whether a general-purpose agent can make a robot work on its own.\n\nWe evaluate agents across two robot platforms: a 7-DoF arm and a 29-DoF humanoid. Our tasks focus on core robotic capabilities such as perception, planning and low-level control.\n\n**24 tasks across two robot platforms:** 13 arm tasks and 11 humanoid tasks.\n\nRobo-bench’s arm tasks are performed on a 7-DoF Franka Emika Panda robot with a parallel-jaw gripper. The arm’s joints track position targets with the robot’s own servos; there is no motion planner or grasping software. The agent reads a six-axis force-torque sensor at the wrist and the gripper’s opening and grip force, which helps it understand the forces involved in the task: it can feel contact and weigh what it holds. As on the real arm, a protective stop halts the arm when a link presses on something with more than 45 N or the hand meets more than 80 N, and joint targets are rate-limited to the datasheet joint speeds. The robot is simulated in MuJoCo, but the agent interacts with it as it would with the physical hardware: through motor commands, sensor readings and camera images, never the simulator itself. The agent commands the arm at up to 50 Hz by sending joint position targets, plus a gripper width and force. In total, we evaluate 13 different arm manipulation tasks.\n\nRobo-bench’s humanoid tasks are performed on a 29-DoF Unitree G1 humanoid with two 7-DoF three-finger Dex3 hands. There is no balance or walking controller: the agent commands every motor itself, legs included. To keep controllers from exploiting weaknesses in the simulator, both robots are held to their real motor limits: every joint stops at the end of its range, the arm’s targets are rate-limited, and the humanoid’s motors follow a torque-speed curve, so torque commands cannot inject unbounded power. Unlike the arm, the humanoid does not take bare joint positions: as in Unitree’s low-level interface, each motor takes a target position and velocity together with a stiffness, a damping and a feed-forward torque, and the hands take joint position targets. In total, we evaluate 11 different humanoid locomotion and manipulation tasks.\n\n**Cameras: two on the arm, one on the humanoid.** We provide visual observation\nthrough the robots’ own cameras. The arm has a fixed overhead camera and a\nwrist camera; the humanoid has a head camera. Every camera returns color images\nand, on request, a depth image.\n\nBesides the written task, everything the agent learns about the scene comes through these cameras: it gets no object positions. It does know its own robot’s state (joint angles, velocities and torques; on the arm the gripper and the wrist force sensor, on the humanoid two IMUs, foot contact forces and its base position and velocity) and each camera’s calibration, so it can relate what it sees to the robot.\n\nEach episode lasts three minutes of wall-clock time, with no setup time: reading the task and planning count too. The physics keeps running while the model thinks, so a humanoid left standing on stiff joints will eventually tip over, and an agent that spends its time deliberating has less of it left to act. Balance needs a correction every second or so, far faster than a model’s turn, so any agent that succeeds has to write code that closes the loop on its own.\n\n### Private, with public examples\n\nRobo-bench is a held-out evaluation. Three tasks are public; the other 21 stay private, which keeps them out of training data and keeps agents and harnesses from being tuned to the test. The private tasks span assembly, dexterity, dynamics, reasoning, stacking and tool use on the arm, and multi-stage courses, dynamic obstacles, dynamics, loco-manipulation and perception on the humanoid.\n\n| Public task | Robot | What it takes | Ladder | \n|---|---|---|---|\n| Balance scale | arm | Balance a beam scale against a sealed load of unknown mass using labelled weights, either by weighing the load with the wrist force sensor or by trial and error. | weight on the right pan 10, beam off its stop 30, within 5° 50, within 2° 75, balanced and held for 2 s 100 | \n| Lego manual | arm | Build a five- or six-brick model on a studded baseplate by following an illustrated manual; bricks click together through their studs and must be pressed into place. | 15 per brick placed as the manual shows, 100 for the finished model | \n| Ninja course | humanoid | Climb five stairs, step onto a motor-driven swinging platform and ride it across a water pit, walk down a ramp, crawl through a 0.8 m crawl space on hands and knees, and reach the finish. | top of the stairs 20, on the swing 35, across the pit 50, down the ramp 60, through the crawl space 80, finish 100 | \n\nThe private tasks are scored the same way: milestones worth 100 in total, which only count in order. Seeds randomize object poses and hidden parameters such as the unknown mass.\n\n## Results\n\nScores are the mean over all 24 tasks, each out of 100. The arm and humanoid columns average their own suites, and the three public tasks are shown on their own. Cost is the estimated API cost of the whole run: all 24 tasks for the overall score, and only that column’s tasks for the others. The time column is each model’s average number of seconds from its launch to its first motor command, counting a task where the robot never moved as the full three minutes.\n\nEvery motor command an agent sends is logged with the tick it took effect, so when each model started acting can be read off directly. GPT-6 Astra and GPT-6.1 Sol moved the robot in every episode, a little over a minute in on average. The other five averaged more than two minutes before their first command, and Grok 4.7 and Gemini 4 Argon never moved it in most episodes.\n\n## What the agent does\n\nEach model runs in the coding agent its own vendor ships, at its highest reasoning setting: Codex for OpenAI models, Claude Code for Anthropic models, Antigravity for Gemini, Grok Build for Grok and Muse Code for Muse.\n\nThe agent works in a fresh workspace holding the task (`GOAL.md`), the\nrobot’s datasheet (`ROBOT.md`: command semantics, joint axes and limits, sensor\nframes, link masses, camera calibration), the same data as JSON, the robot’s\nMuJoCo model without the scene, and a small Python client. The client reads the\nlive robot state, sends motor commands, runs a function of the agent’s choosing\nevery control tick, and takes pictures. Everything else (perception, planning,\nthe controllers) the agent writes in its own shell during the episode.\n\n## Methodology\n\n- **Clock.** The simulation runs in real time from the agent’s launch, and every\nepisode ends at three minutes. Episodes where the simulation fell more than a\nsecond behind the clock are flagged.\n- **Episode rules.** A broken rule ends an arm episode. Humanoid episodes keep\nrunning: a fall is not a failure, and a broken rule only rules out success.\nA diverging simulation ends any episode as a failure.\n- **Isolation and scoring.** The agent runs as an unprivileged user in a sandbox\nthat reaches only its model provider. The simulator, which the agent cannot\nread, scores and signs each episode. Videos replay the logged motor commands.\n- **Validity.** Every task has a reference solution that succeeds on five of five\nseeds, and a do-nothing agent never succeeds. Simulator bugs that agents\nexposed, such as inverted cylinder contacts, are fixed.\n- **Cost.** Token counts reported by each provider, at list prices.\n- **One seed.** Each task runs once per model. Repeated runs moved a model’s\ntotal by up to about ten points, so small gaps are not significant.", "url": "https://wpnews.pro/news/frontier-models-score-under-2-100-controlling-a-simulated-humanoid", "canonical_source": "https://www.vals.ai/benchmarks/robo_bench", "published_at": "2026-10-14 23:20:51+00:00", "updated_at": "2026-10-15 05:11:33.074240+00:00", "lang": "en", "topics": ["robotics", "artificial-intelligence", "ai-agents", "ai-research"], "entities": ["Robo-bench", "GPT-6.1 Sol", "Grok 4.7", "Gemini 4 Argon", "Claude Opus 5.5", "Claude Fable 5.1", "Muse Wonder Full", "Unitree G1"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/frontier-models-score-under-2-100-controlling-a-simulated-humanoid", "markdown": "https://wpnews.pro/news/frontier-models-score-under-2-100-controlling-a-simulated-humanoid.md", "text": "https://wpnews.pro/news/frontier-models-score-under-2-100-controlling-a-simulated-humanoid.txt", "jsonld": "https://wpnews.pro/news/frontier-models-score-under-2-100-controlling-a-simulated-humanoid.jsonld"}}