{"slug": "refactor-vla-unsupervised-library-learning-of-typed-motor-programs", "title": "REFACTOR-VLA: Unsupervised Library Learning of Typed Motor Programs", "summary": "Researchers Riyaaz Shaik and Chandru Venkataraman introduced REFACTOR-VLA, a system that learns reusable motor skills for vision-language-action models using a wake/sleep architecture with a Behavioral-Equivalence Kernel and typed lambda terms. Evaluated on the LIBERO benchmark, it found that scaling the world model from 188M to 430M parameters worsened performance on all 4 suites, while adding an InfoNCE contrastive loss during warmup improved skill clustering, achieving NMI scores up to 0.915 on the Goal suite.", "body_md": "[content type paper](/research/)published September 2026\n\nREFACTOR-VLA: Unsupervised Library Learning of Typed Motor Programs\n\nAuthorsRiyaaz Shaik, Chandru Venkataraman\n\nREFACTOR-VLA: Unsupervised Library Learning of Typed Motor Programs\n\nAuthorsRiyaaz Shaik, Chandru Venkataraman\n\nMost current vision-language-action (VLA) models—such as OpenVLA, π0, RT-2, and RDT-1B—are “monolithic.” This means they generate raw motor commands or very short sequences of actions, without organizing behaviors into reusable, well-defined abstractions. As a result, these models perform poorly on long-horizon (multi-step) tasks, and it’s difficult to interpret what they have learned. Existing approaches for discovering skills often avoid the core problem of deciding when two action sequences are “behaviorally equivalent.” For example, AtomicVLA and AtomSkill group action sequences by clustering their contrastive embeddings. In contrast, BLADE and LRLL rely on a large language model (LLM) to judge whether two sequences are equivalent, but these LLMs are not calibrated to the robot’s own dynamics. We introduce REFACTOR-VLA, a system that learns reusable skills using a “wake/sleep” architecture. In the sleep phase, the system clusters segments of motor programs using a Behavioral-Equivalence Kernel (BEK). This BEK is based on the outcomes of rolling out actions in a learned latent world model, Mφ. In the wake phase, the system generates typed lambda terms (simple, structured programs) from a vocabulary inspired by the Hindley–Milner type system. These lambda terms are then used by a library-conditioned rectified-flow action decoder to produce actions. Only abstractions that pass both a Minimum Description Length (MDL) criterion and a return-preservation gate are accepted as skills. To train REFACTOR-VLA, we use a three-phase schedule: • Phase A (World-model warmup): The latent world model Mφ is trained. • Phase B (Wake-phase policy optimization): The policy that uses the library of skills is optimized. • Phase C (Sleep-phase skill discovery): The system clusters action fragments into reusable skills. We evaluated REFACTOR-VLA on the full LIBERO benchmark suite. Our results show two main findings. First, simply increasing the size of the world model—from 188 million to 430 million parameters—worsened performance on 4 out of 4 benchmark suites, disproving the idea that just making the world model bigger always helps. Second, changing the training objective makes a big difference: adding an auxiliary supervised contrastive loss (specifically, InfoNCE loss) during the world-model warmup (Phase A) greatly improved the quality of skill clustering in the sleep phase (Phase C). We measured this using Normalized Mutual Information (NMI) under n = 3 multi-seeding: • Object suite: 0.462 ± 0.021 • Spatial suite: 0.867 ± 0.025 • Goal suite: 0.915 ± 0.013 • LIBERO-10 suite: 0.754 ± 0.010\n\nGrounding Multimodal Large Language Models in Actions\n\nFebruary 20, 2025[research area Computer Vision](/research/?domain=Computer%20Vision), [research area Methods and Algorithms](/research/?domain=Methods%20and%20Algorithms)[conference NeurIPS](/research/?event=NeurIPS)\n\nMultimodal Large Language Models (MLLMs) have demonstrated a wide range of capabilities across many domains, including Embodied AI. In this work, we study how to best ground a MLLM into different embodiments and their associated action spaces, with the goal of leveraging the multimodal world knowledge of the MLLM. We first generalize a number of methods through a unified architecture and the lens of action space adaptors. For continuous actions,…\n\nKV-Runahead: Scalable Causal LLM Inference by Parallel Key-Value Cache Generation\n\nMay 14, 2024[research area Methods and Algorithms](/research/?domain=Methods%20and%20Algorithms), [research area Speech and Natural Language Processing](/research/?domain=Speech%20and%20Natural%20Language%20Processing)[conference ICML](/research/?event=ICML)\n\nLarge Language Model or LLM inference has two phases, the prompt (or prefill) phase to output the first token and the extension (or decoding) phase to the generate subsequent tokens. In this work, we propose an efficient parallelization scheme, KV-Runahead to accelerate the prompt phase. The key observation is that the extension phase generates tokens faster than the prompt phase because of key-value cache (KV-cache). Hence, KV-Runahead…", "url": "https://wpnews.pro/news/refactor-vla-unsupervised-library-learning-of-typed-motor-programs", "canonical_source": "https://machinelearning.apple.com/research/refactor-vla-motor-programs", "published_at": "2026-09-02 00:00:00+00:00", "updated_at": "2026-09-02 15:55:31.228523+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "robotics", "ai-research"], "entities": ["REFACTOR-VLA", "Riyaaz Shaik", "Chandru Venkataraman", "LIBERO", "OpenVLA", "AtomicVLA", "BLADE"], "alternates": {"html": "https://wpnews.pro/news/refactor-vla-unsupervised-library-learning-of-typed-motor-programs", "markdown": "https://wpnews.pro/news/refactor-vla-unsupervised-library-learning-of-typed-motor-programs.md", "text": "https://wpnews.pro/news/refactor-vla-unsupervised-library-learning-of-typed-motor-programs.txt", "jsonld": "https://wpnews.pro/news/refactor-vla-unsupervised-library-learning-of-typed-motor-programs.jsonld"}}