iFLYTEK-Embodied-Omni Technical Report

IFLYTEK-Embodied-Omni, a unified multimodal foundation model, jointly models vision, language, and action within a single framework to enable general-purpose embodied agents to understand instructions, predict environment evolution, and produce precise control actions. The model uses a brain-cerebellum architecture with shared multimodal self-attention, combining vision-language, video-generation, and action-generation components. A four-stage training strategy leverages action-annotated and action-free embodied videos, human demonstrations, and robot interactions to develop these capabilities.

arXiv:2607.02542v1 Announce Type: new Abstract: General-purpose embodied agents must understand multimodal instructions, anticipate how their environment will evolve, and produce precise control actions over extended horizons. Existing approaches typically specialize in visual-language reasoning, video-based world modeling, or action generation, while cascaded pipelines that first synthesize future observations and then infer actions can introduce interface bottlenecks and compound prediction errors. We present iFLYTEK-Embodied-Omni, a unified multimodal foundation model that jointly models vision videos and images , language, and action within a single Omni framework. Its modality-specific visual-language, video-generation, and action-generation components communicate through shared multimodal self-attention. This design establishes brain-cerebellum collaboration: the vision-language modeland video generation model form a high-level brain for instruction understanding, task planning, progress tracking, and future visual-state prediction, whereas the action generation modelserves as a low-level cerebellum that directly converts planned subgoals and shared multimodal context into executable action chunks. To develop these capabilities, we combine action-annotated and action-free embodied videos from human demonstrations and robot interactions with embodied reasoning, embodied perception, and general-purpose image-text data to construct a comprehensive dataset. We further adopt a four-stage strategy that progressively trains the VLM, VGM, and AGM before jointly fine-tuning the complete model.