{"slug": "learning-hierarchical-skill-policies-with-offline-quality-diversity-learning", "title": "Learning Hierarchical Skill Policies with Offline Quality-Diversity Reinforcement Learning", "summary": "Researchers introduced QDOS (Quality-Diversity Offline Skill learning), a unified pipeline for offline-to-online reinforcement learning that uses an Advantage-Weighted Quality-Diversity pretraining objective to extract diverse and high-value skills from pre-collected datasets. In experiments, QDOS significantly outperformed strong baselines in structured manipulation and unstructured locomotion tasks, accelerating exploration and improving final returns in sparse-reward domains.", "body_md": "arXiv:2608.19684v1 Announce Type: new\nAbstract: Recent studies investigate how to leverage pre-collected datasets to improve the policy performance and sample efficiency of RL. One promising approach to achieve this goal is to employ a two-stage strategy: In the first stage, diverse skills are extracted as a low-level policy from a given dataset, and a high-level policy is trained to solve a specific task in the second stage. Typically, extraction of the low-level policy is performed based on unsupervised learning such as trajectory VAE. However, a limitation of this approach is that the quality of the low-level policy highly depends on the quality of the dataset. To address this issue, we introduce QDOS (Quality-Diversity Offline Skill learning), a unified pipeline for robust offline-to-online learning. Our approach incorporates an Advantage-Weighted Quality-Diversity pretraining objective, which weights the skill extraction and diversity objectives by the estimated advantage of each trajectory segment. This approach allows the model to extract diverse and high-value skills. By providing robust and task-relevant skill representations, QDOS significantly improves the quality of the embedded skill space used by the low-level policy. We further integrate this with a dual dataset reuse strategy, where offline data is used both for skill pretraining and for populating the online replay buffer via pseudo-labeling. Experiments demonstrate that QDOS significantly outperforms strong baselines in structured manipulation tasks and unstructured locomotion tasks, confirming its ability to accelerate exploration and improve final returns in challenging sparse-reward domains.", "url": "https://wpnews.pro/news/learning-hierarchical-skill-policies-with-offline-quality-diversity-learning", "canonical_source": "https://arxiv.org/abs/2608.19684", "published_at": "2026-08-21 04:00:00+00:00", "updated_at": "2026-08-21 04:12:52.256538+00:00", "lang": "en", "topics": ["machine-learning", "artificial-intelligence"], "entities": ["QDOS"], "alternates": {"html": "https://wpnews.pro/news/learning-hierarchical-skill-policies-with-offline-quality-diversity-learning", "markdown": "https://wpnews.pro/news/learning-hierarchical-skill-policies-with-offline-quality-diversity-learning.md", "text": "https://wpnews.pro/news/learning-hierarchical-skill-policies-with-offline-quality-diversity-learning.txt", "jsonld": "https://wpnews.pro/news/learning-hierarchical-skill-policies-with-offline-quality-diversity-learning.jsonld"}}