{"slug": "cap-x-lms-first-physical-exam", "title": "CAP-X: LMs' First Physical Exam", "summary": "A new benchmark called CaP-Bench, integrated with LIBERO-PRO, Robosuite and BEHAVIOR, shows frontier large language models can generate executable robot control code zero-shot at over 30% average success, leaving a 56-point gap to human performance. On LIBERO-PRO's 30 perturbed manipulation tasks, state-of-the-art Vision-Language-Action models OpenVLA and π0 scored 0% while the best VLA, π0.5, reached 13%, and the training-free coding agent CaP-Agent0 reached 18%. Using CaP-RL reinforcement learning on the coding agent, a 7B Qwen 2.5 Coder model rose from 20% to 72% average success in simulation after 50 training iterations and transferred to a real Franka Emika robot, reaching 84% on cube lifting and 76% on cube stacking.", "body_md": "Today's off-the-shelf LMs have incredible generalization, reasoning, and planning capabilities.\n          Agentic harnesses in **CaP-Agent0** unleash their potential in the physical world.\n        \n\nClick on each task to see the agent in action.\n\n**CaP-Bench** provides the first comprehensive benchmark for evaluating how well large language model agents\n          can write code to control robots. Integrated with hundreds of manipulation tasks across multiple\n          robot learning benchmarks (LIBERO-PRO, Robosuite, BEHAVIOR), **CaP-Bench** tests both LLM and VLM models on their\n          ability to generate executable robot control policies from natural language instructions.\n        \n\n```\nKey Findings\n\n1\nFrontier models achieve meaningful zero-shot success on robotic manipulation\n\n            Without any task-specific training, today's best frontier models can directly generate executable robot control\n            code and achieve over 30% average success — a sharp contrast to the prior belief that only specially trained\n            models (VLAs) can perform manipulation. Yet a 56-point gap to human performance remains, marking this as\n            one of AI's most important open challenges.\n          \n\nLoading chart...\n\n2\nTraining-free CaP-Agent0 outperforms state-of-the-art VLAs on perturbed tasks\n\n            On LIBERO-PRO — 30 manipulation tasks with position and instruction perturbations — state-of-the-art\n            Vision-Language-Action models (OpenVLA, π0) score 0% across the board.\n            Even the best VLA (π0.5) reaches only 13% average success. CaP-Agent0, a training-free\n            coding agent, achieves 18% without any task-specific training, demonstrating that\n            code-generation agents generalize where end-to-end learned policies break down.\n          \n\nLoading chart...\n\n3\nCaP-RL: Post-training on code dramatically boosts robot performance — and transfers sim-to-real\n\n            Using CaP-RL, we apply reinforcement learning with environment rewards directly on the coding agent.\n            A 7B model (Qwen 2.5 Coder) jumps from 20% to 72% average success in simulation after\n            just 50 training iterations. The learned policies transfer to a real Franka Emika robot with\n            minimal sim-to-real gap — reaching 84% on cube lifting and 76% on cube stacking,\n            approaching human expert performance.\n          \n\nLoading chart...\n\n4\nHigher abstraction boosts all models — and dramatically closes the gap for smaller ones\n\n            As API abstraction increases from raw primitives (S4) to high-level pick-and-place (S1),\n            all models improve substantially — but the gains are most pronounced for\n            weaker and open-source models, whose compilation rates collapse at low abstraction levels.\n            This suggests a promising path: pair a lightweight LM for high-level planning\n            with a visual-motor policy (e.g., a VLA) that handles low-level control, letting even\n            smaller models achieve strong task performance through the right division of labor.\n          \n\nLoading chart...\n```\n\n", "url": "https://wpnews.pro/news/cap-x-lms-first-physical-exam", "canonical_source": "https://capgym.github.io/", "published_at": "2026-10-05 21:59:12+00:00", "updated_at": "2026-10-05 22:20:04.171838+00:00", "lang": "en", "topics": ["robotics", "large-language-models", "ai-agents", "ai-research", "machine-learning"], "entities": ["CaP-Bench", "CaP-Agent0", "CaP-RL", "LIBERO-PRO", "Robosuite", "BEHAVIOR", "OpenVLA", "π0.5"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/cap-x-lms-first-physical-exam", "markdown": "https://wpnews.pro/news/cap-x-lms-first-physical-exam.md", "text": "https://wpnews.pro/news/cap-x-lms-first-physical-exam.txt", "jsonld": "https://wpnews.pro/news/cap-x-lms-first-physical-exam.jsonld"}}