{"slug": "robocolosseum-5699-real-robot-comparisons-across-124-sites", "title": "RoboColosseum: 5,699 Real-Robot Comparisons Across 124 Sites", "summary": "A new evaluation called RoboColosseum ran 5,699 blinded pairwise comparisons at 124 sites and found that frontier AI agents rival specialized robot policies, that policy rankings shift substantially across robot bodies and test types (Kendall's W of 0.24), and that no policy completed any of the five tasks on the G1 humanoid, with progress ranging from 13.3% to 23.2%. The study, arXiv 2611.02497, tested 5 VLA policies and 3 frontier AI agents on four robot bodies — SO-101, Franka, Bimanual-YAM and the G1 Humanoid — with GR00T N1.7 leading on SO-101 and pi0.5 leading on Franka and Bimanual-YAM after fine-tuning. The authors report rank reversals between zero-shot and fine-tuned rankings on all three shared robots, and the paper's client repository for Open Track runs had 2 GitHub stars at the time of writing.", "body_md": "RoboColosseum is a new evaluation that tests robot policies on real hardware at many locations instead of in one lab. Its first study ran 5,699 blinded pairwise comparisons at 124 sites and found three things: frontier AI agents rival specialized robot models, policy rankings change a lot between robots and between test types, and no policy completed a single task on the G1 humanoid. The paper is arXiv 2611.02497, and it ranked on the Hugging Face daily papers page with 23 upvotes when we checked.\n\nThis matters because robot demos are easy to stage and hard to compare. This paper is an attempt to make \"which model is better\" a question with a repeatable answer. The author list on the paper page has more than 25 names, including Dieter Fox and Siddhartha Srinivasa, but the page lists no affiliations. We could not open the project site at robocolosseum.ai (it returned a 401 error), so this post relies on the Hugging Face paper page and the public client repository.\n\n## TL;DR: what RoboColosseum found\n\n| Question | Answer (source) | \n|---|---|\n| What was tested? | 5 VLA policies and 3 frontier AI agents on 4 robot bodies ( [paper page](https://huggingface.co/papers/2611.02497?ref=explainx) ) | \n| Which robots? | SO-101, Franka, Bimanual-YAM and the G1 Humanoid | \n| How many trials? | 5,699 pairwise Open Track comparisons at 124 sites | \n| Who leads after fine-tuning? | GR00T N1.7 on SO-101; pi0.5 on Franka and Bimanual-YAM | \n| Do rankings agree across robots? | Weakly. Kendall's W is 0.24 | \n| Does zero-shot ranking predict fine-tuned ranking? | Poorly; the paper reports rank reversals on all three shared robots | \n| Did the humanoid finish any task? | No policy completed any of the five G1 tasks; progress ranged from 13.3% to 23.2% | \n| Is there code? | A client repo for Open Track runs, with 2 GitHub stars at the time of writing | \n\n## How does RoboColosseum work?\n\nThe framework solves a practical problem. Physical experiments vary: the table, the lighting, the objects and the operator all change. That variation makes trials informative, but it also makes them hard to reproduce. RoboColosseum accepts the variation and averages over a network of locally operated robots, with unified policy serving and standardized protocols so every site runs the same procedure.\n\n### The Open Track\n\nThe Open Track compares models without task-specific adaptation. Trials are blinded and pairwise: an operator gives an instruction, two policies each attempt it under matched local conditions, and the operator scores the outcome and states a preference. The client README shows the workflow. The client controls the robot, communicates with a policy server, records the trial, and uploads the result. In the YAM evaluation GUI, an operator reviews scores, picks a preference and adds observations, and the repo says Claude Haiku turns those notes into written feedback before the result is submitted.\n\nThe README also lists the LLM relay setup: API keys for OpenAI, xAI and Anthropic models are selected by model name (gpt, grok and claude prefixes). That strongly suggests the three frontier agents in the study come from those three families, but the abstract does not name them, so treat that as an inference.\n\n### The Fine-tune Track\n\nThe Fine-tune Track measures how well VLA policies adapt. Each policy receives the same 100 demonstrations for each of 20 tasks across the four robots. This separates \"works out of the box\" from \"adapts well,\" and the paper's main finding is that the two do not line up.\n\n## What did the results show?\n\n### Frontier agents rival specialized robot policies\n\nAcross the 5,699 Open Track comparisons, the authors report that frontier agents rival specialized VLA policies, \"though all systems struggle to complete open-world tasks reliably.\" That is a notable result for people who assumed a robot-specific model would win by default. It does not mean a chatbot can run a robot well. It means that, on these protocols, a general agent with tool access to a robot interface scored comparably on preference and progress.\n\n### Fine-tuning changes who wins\n\nAfter fine-tuning, GR00T N1.7 leads on SO-101, and pi0.5 leads on Franka and Bimanual-YAM. The abstract also says Open Track rankings poorly predict relative performance after fine-tuning, with rank reversals on all three embodiments shared between the tracks. In plain terms: the model that looks best on day one is not reliably the one that learns best from 100 demos. If you are choosing a policy for your own robot, a zero-shot demo is weak evidence.\n\nA gripper mirroring a hand, representing the 100 demonstrations per task used in the Fine-tune Track\n\n### Rankings depend on the robot\n\nPolicy rankings show weak agreement across embodiment-specific task suites, with Kendall's W of 0.24. W runs from 0 (no agreement) to 1 (perfect agreement), so 0.24 means the order of policies changes noticeably from one robot body to another. A single \"best robot model\" leaderboard would hide that.\n\n### The humanoid result\n\nOn the G1 Humanoid, no policy completes any of the five tasks. The authors then measure partial progress and find average progress ranging from 13.3% to 23.2%. Success rate alone would call every policy equal at zero. Partial credit separates them. This is the most useful methodological point in the paper, because most real-world robot tasks are multi-step and a pass/fail score throws away information. It also matches what we have seen in humanoid coverage: demos look strong, and measured results are modest. See our reading of the [Unitree H2 endurance and kung fu claims](https://explainx.ai/blog/unitree-h2-x-kung-fu-legend-guardian-8-hour-endurance-claimed-vs-verified-2026) for a case of the same gap between show and measurement.\n\n### Fewer trials with smart matchmaking\n\nOpen Track rankings stabilize as trials accumulate, and active matchmaking reaches comparable ranking agreement with substantially fewer physical trials than uniform sampling. Real robot trials are slow and expensive, so choosing which pairs to test next, as chess ratings do, is a practical contribution. For a lab with one robot and a limited number of operator hours, this matters more than the headline ranking. It means a team can reach a trustworthy order of policies without running every possible pairing. It also means a network like this can grow without the cost of trials growing at the same rate.\n\n## How does this compare with other robot benchmarks?\n\nMost robot evaluations run in simulation or in a single lab. RoboColosseum trades control for diversity: 124 sites mean many kinds of tables, lights and objects. Related items on explainx.ai show the same direction. Physical Intelligence opened a [pi API beta for robot fine-tuning](https://explainx.ai/blog/physical-intelligence-pi-api-beta-robot-fine-tuning-2026), the kind of service whose claims a shared evaluation could test. NVIDIA published a [browser demo of a Unitree G1 humanoid policy](https://explainx.ai/blog/nvidia-metatron-browser-unitree-g1-humanoid-policy-onnx-webgpu-open-model-license-2026). Unitree's [UnifoLM-X2 world model](https://explainx.ai/blog/unitree-unifolm-x2-world-model-humanoid-combat-2026) is another model that would need independent trials. And a recent open [Qwen-based robot agent with RoboDojo](https://explainx.ai/blog/uni-robot-agent-open-qwen-robodojo) sits in the same \"general agent controls a robot\" camp as the frontier agents here.\n\n| Benchmark style | Strength | Weakness | \n|---|---|---|\n| Simulation suites | Cheap, repeatable | Gap to real hardware | \n| Single-lab real tests | Controlled | Narrow scenes, hard to compare | \n| RoboColosseum (distributed, blinded, pairwise) | Diverse scenes, preference-based | Operator variation, early software | \n\n## Limitations to keep in mind\n\n- **Early release.** The client repository has 2 stars, and the Unitree G1 driver is listed in the README as not yet written. The humanoid result in the paper therefore does not come from a fully supported public path yet. Anyone who wants to reproduce the G1 numbers cannot do so with the public client today.\n- **Hardware validation pending.** The README lists Bimanual YAM as \"hardware validation pending,\" and Earth Rover Mini+ physical validation is also pending.\n- **Human judgment.** Pairwise preference and partial-progress scores come from operators. Blinding helps, but scoring is still subjective.\n- **Small policy set.** Five VLA policies and three agents is a narrow sample of a fast-moving field.\n- **We read the abstract and the repository, not the full paper.** Details such as exact per-policy scores were not available to us in the pages we could open, so we give only what the abstract states.\n- **Project site unavailable to us.** The robocolosseum.ai page returned an error, so we could not check its leaderboard or video.\n\n## What builders should take away\n\n1. **Do not choose a policy from a zero-shot demo.** The paper's rank reversals say fine-tuned performance is a separate question.\n2. **Measure partial progress.** If you evaluate your own robot, log sub-task progress. It reveals differences that task success misses.\n3. **Test on more than one body.** With W at 0.24, a model that tops one embodiment may not top another.\n4. **Treat humanoid claims carefully.** No policy finished a task on the G1 in this study.\n5. **Consider joining.** If you run an SO-101 or Franka, the client lets you contribute trials. Keep the physical emergency stop within reach, as the README warns that a software stop is not an emergency stop.\n\n*Details are accurate as of November 4, 2026. Results come from an unreviewed preprint abstract and public repository.*", "url": "https://wpnews.pro/news/robocolosseum-5699-real-robot-comparisons-across-124-sites", "canonical_source": "https://explainx.ai/blog/robocolosseum-physical-intelligence-eval-124-sites-5699-comparisons-gr00t-n1-7-pi0-5-g1-humanoid-2026?utm_source=rss&utm_medium=feed&utm_campaign=blog", "published_at": "2026-11-04 00:00:00+00:00", "updated_at": "2026-11-04 07:21:56.255352+00:00", "lang": "en", "topics": ["robotics", "ai-research", "ai-agents", "ai-safety"], "entities": ["RoboColosseum", "GR00T N1.7", "pi0.5", "G1 Humanoid", "SO-101", "Franka", "Bimanual-YAM", "Dieter Fox"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/robocolosseum-5699-real-robot-comparisons-across-124-sites", "markdown": "https://wpnews.pro/news/robocolosseum-5699-real-robot-comparisons-across-124-sites.md", "text": "https://wpnews.pro/news/robocolosseum-5699-real-robot-comparisons-across-124-sites.txt", "jsonld": "https://wpnews.pro/news/robocolosseum-5699-real-robot-comparisons-across-124-sites.jsonld"}}