SalsaAgent: A multimodal embodied language model for interactive dance generation

Researchers have developed SalsaAgent, a multimodal embodied language model that generates expressive, full-body salsa dance motions in response to a human leader and synchronized to music. The system extends a large language model's vocabulary to process discrete motion tokens, pairwise relation tokens, and audio, enabling nonverbal interaction through motion token passing. Evaluations show SalsaAgent significantly outperforms baselines in motion quality, music and partner coordination, and consistent two-person spatial behavior, advancing socially aware robotics and interactive virtual agents.

arXiv:2605.29219v1 Announce Type: new Abstract: Interaction between humanoids involves bidirectional and nonverbal reactivity, coordination and synchrony. Toward socially aware robots and interactive virtual agents, we present SalsaAgent, a language model that generates expressive, full-body salsa dance motions in reaction to a human leader and against a contextual music backdrop. We formulate interaction as nonverbal motion token passing, extending the vocabulary of a large language model LLM to process discrete motion tokens, pairwise relation tokens, and audio. Our contributions include new tokens for full-body and motion relations, LLM fine-tuning using automatically derived text descriptions of skeleton dynamics for token grounding, and a two-stage token-to-diffusion pipeline. Subjective and objective evaluations demonstrate the effectiveness of our approach in terms of motion quality, music and partner coordination, and consistent two-person spatial behavior, with significant improvements over baselines.