ARDY: Autoregressive Diffusion for Human Motion Generation

NVIDIA researchers introduced ARDY, an autoregressive diffusion model for interactive human motion generation that supports online text prompting and flexible long-horizon kinematic constraints in real time. The model, presented at SIGGRAPH 2026, bridges the gap between offline controllability and online synthesis, enabling applications in animation, simulation, and humanoid robotics.

ARDY: Autoregressive Diffusion with Hybrid Representation for Interactive Human Motion Generation ACM Transactions on Graphics · SIGGRAPH 2026 ARDY is an autoregressive diffusion model designed for interactive motion generation, supporting online text prompting and flexible long-horizon kinematic constraints root paths/waypoints, full-body keyframes, and sparse joint positions/rotations with real-time responsiveness. Generating realistic 3D human motions in real-time within interactive applications is key for animation, simulation, and humanoid robotics. While recent offline motion generation approaches like Kimodo https://research.nvidia.com/labs/sil/projects/kimodo/ offer precise control via text and kinematic constraints, they lack the inference speed required for interactive settings. Conversely, existing online methods enable real-time synthesis but often sacrifice controllability or struggle with complex text semantics and long-horizon goals due to limited context windows. In this work, we introduce ARDY, a streaming generation framework that bridges this gap by enabling high-fidelity motion generation controllable via online text prompts and flexible kinematic constraints. ARDY employs a hybrid representation that combines explicit root features with a latent body embedding, balancing precise trajectory control with efficient generative learning. We propose a two-stage autoregressive transformer denoiser that features variable history context and supports conditioning on flexible, long-horizon kinematic constraints. By training on a large-scale motion capture dataset and being directly conditioned on text labels and kinematic constraints sampled from ground truth poses, ARDY natively learns controllable generation that supports online prompting and flexible long-horizon goals. Extensive evaluations demonstrate strong motion quality and constraint adherence, and we present an interactive demo with dynamic text control, keyframe constraints, path following, and real-time locomotion control. Key Capabilities of ARDY Online Text-to-Motion Generation ARDY supports interactive text-conditioned motion generation across a wide range of behaviors. Kinematically Constrained Motion Generation ARDY supports flexible kinematic constraints, including root trajectories or waypoints, full-body keyframes, end-effector joint positions and rotations, as well as arbitrary combinations of these constraints. Constraints can also be specified far into the future beyond the current generation window to enable long-horizon goal reaching. Root Trajectory 1 Root Trajectory 2 Root Waypoint 1 Root Waypoint 2 Fullbody Keyframe 1 Fullbody Keyframe 2 End Effector Position End Effector Position+Orientaion Long-Horizon Goal 1 Long-Horizon Goal 2 Compositionality Kinematic Constraints Chain Application: Interactive Humanoid Control ARDY enables online motion synthesis for interactive applications, which can be valuable for game character control, downstream robotics, and simulation workflows. It supports real-time locomotion control via mouse waypoint editing and keyboard velocity commands. Humanoid Robot Control By combining ARDY’s real-time humanoid motion generation with the SONIC https://nvlabs.github.io/SONIC/ physical tracking policy, we enable interactive robot motion control with streaming constraints and user inputs. We demonstrate applications on the Unitree G1 robot. Method ARDY is an autoregressive diffusion model for interactive motion generation. It is built around two key ideas: 1 a hybrid motion representation that combines explicit global root motion with a compact latent embedding of body motion, and 2 an autoregressive two-stage transformer denoiser that generates motion in a streaming fashion while conditioning on online text prompts and flexible, spatiotemporally sparse kinematic constraints over long horizons. Motion Tokenizer. The encoder first embeds the patchified body motion into a latent representation. This latent body motion is concatenated with the patchified global root motion to form our hybrid representation, which is decoded back to reconstruct the body motion. This hybrid representation balances precise, interpretable root control useful for global scene-space constraints with efficient generative learning in a lower-dimensional latent space for body motion. Autoregressive Two-Stage Transformer Denoiser. Left Conditioned on a variable-length history context and optional spatial goal constraints, the autoregressive denoiser predicts a sequence of C clean motion tokens within the current generation window. Spatial goal constraints can be arbitrarily sparse and may be located within or beyond the current motion generation window. Right The two-stage denoiser first predicts clean global root motion, which then conditions the second stage to predict clean latent body tokens, together forming the complete hybrid motion prediction. The denoiser supports a variable-length history context to capture longer-term semantics, and conditions on masked kinematic constraints that can be sparse in both time and joints. These constraints can span beyond the current generation window to enable long-horizon goals e.g., far-future waypoints . Our interleaved two-stage design predicts root first and then body conditioned on root, helping maintain motion fidelity while satisfying online text prompts and diverse spatial controls such as root trajectories/waypoints, full-body keyframes, and sparse joint positions/rotations. More details can be found in the paper. Humanoid Motion at NVIDIA ARDY is part of a broader effort to support interactive humanoid motion generation and downstream robotics/animation applications. Related projects include: SOMA Body Model https://github.com/NVlabs/SOMA-X A parametric body model used across NVIDIA humanoid motion projects. BONES-SEED Dataset https://huggingface.co/datasets/bones-studio/seed A public dataset with production-quality motion capture data. ProtoMotions https://github.com/NVlabs/ProtoMotions A framework for training physics-based humanoid policies. SOMA Retargeter https://github.com/NVIDIA/soma-retargeter Tools for retargeting motion data across humanoid skeletons. GEM https://github.com/NVlabs/GEM-X A motion diffusion model for reconstructing motion from monocular videos. Kimodo https://research.nvidia.com/labs/sil/projects/kimodo/ An offline controllable motion diffusion model for high-quality 3D motion authoring. MotionBricks https://nvlabs.github.io/motionbricks/ Scalable real-time motions with modular latent generative model and smart primitives. GEAR SONIC https://nvlabs.github.io/SONIC/ A framework for humanoid whole-body tracking and control. Acknowledgments We would like to thank Edy Lim, Eugene Jeong, Sam Wu, Ehsan Hassani, Michael Huang, and Jin-Bey Yu for their help with data processing and cleaning, and Cyrus Hogg, Simon Yuen, Lindsey Pavao, Jenna Diamond, Rizwan Khan, Samantha Shinagawa, and Akanksha Shukla for their efforts on data acquisition and labeling. We also thank the anonymous reviewers for their valuable feedback. BibTeX @article{zhao2026ardy, title = {ARDY: Autoregressive Diffusion with Hybrid Representation for Interactive Human Motion Generation}, author = {Zhao, Kaifeng and Petrovich, Mathis and Zhang, Haotian and Wang, Tingwu and Tang, Siyu and Rempe, Davis}, journal = {ACM Transactions on Graphics TOG }, year = {2026}, volume = {45}, number = {4}, articleno = {86}, doi = {10.1145/3811284} }