Personalizing Embodied Multimodal Large Language Model Agents over Long-term User Interactions

Researchers have developed POLAR, a multimodal memory-augmented framework that enables embodied AI agents to personalize assistance over long-term user interactions by organizing prior experiences into a knowledge graph. The system retrieves relevant memories to interpret implicit user requests and guide task execution, with evaluations showing consistent performance improvements across multiple AI backbones. The memory mechanism proves especially effective when agents must reason across multiple interactions, perform multi-hop inference, or track evolving user-specific context over time.

arXiv:2605.26256v1 Announce Type: new Abstract: Multimodal large language model MLLM -based embodied agents have shown strong potential for solving complex tasks in physical environments. However, personalized assistance requires more than following generic instruction or recognizing object categories. In real-world scenarios, the intended target is often specified only implicitly through prior interactions, requiring agents to leverage personalized context accumulated over time. In this work, we propose POLAR, a multiomodal memory-augmented framework for personalized embodied agents over long-term user interactions. POLAR organizes prior interactions into a multimodal knowledge graph that captures semantic memory for personalized context and visual concepts, and episodic memory for embodied experiences such as agent trajectories. To execute embodied tasks, POLAR retrieves relevant memories to interpret the current request and guide task execution. We evaluate POLAR across multiple MLLM backbones and diverse evaluation scenarios to study the role of memory in long-term personalization. Results show that the proposed memory mechanism consistently improves performance by enabling more effective use of information accumulated over prior interactions. The gains are especially pronounced when the agents are required to reason across multiple interactions, perform multi-hop inference, or tracking updates in user-specific context over time.