{"slug": "alphaavatar-v0-6-4-a-self-hostable-realtime-multimodal-assistant-runtime-with", "title": "AlphaAvatar v0.6.4: a self-hostable realtime multimodal assistant runtime with shared perception and ENV memory", "summary": "AlphaAvatar v0.6.4, an open-source self-hostable runtime for realtime multimodal AI assistants, has been released with a new architecture separating core perception, RTC adapters, and agent orchestration. The update introduces a shared perception pipeline that allows multiple runtime modules to process the same media streams independently, and positions the project as a reusable runtime layer for persistent agents rather than just an avatar frontend.", "body_md": "Hi everyone\n\nA few weeks ago, I shared an early introduction to **AlphaAvatar**, an open-source and self-hostable realtime full-multimodal personal AI assistant runtime.\n\nThe discussion around that post was very helpful. One particularly useful point was that AlphaAvatar may be more valuable when positioned less as another avatar frontend, and more as a reusable runtime layer for persistent realtime multimodal agents.\n\nThat is increasingly how I see the project as well.\n\nAlphaAvatar does not try to own every ASR, TTS, RTC, model, or avatar renderer. Instead, it aims to provide the stateful runtime connecting them:\n\nrealtime multimodal perception\n\nsession and interaction state\n\npersistent Memory and Persona\n\nidentity continuity\n\ntools, MCP, RAG, and DeepResearch\n\nstatus and workflow feedback\n\navatar and channel outputs\n\nself-hosted storage and provider control\n\nSince the previous post, the project has gone through several architecture-focused releases, culminating in **v0.6.4**.\n\nThe new architecture is organized around six main areas:\n\n`avatar-core`\n\ncontains transport-independent multimodal primitives:\n\n`EnvObservation`\n\n`EnvAnnotation`\n\n`MediaPayload`\n\nraw and annotated payload views\n\ngeneric video frame buffers\n\ntyped perception streams\n\nshared timelines\n\nconsumer-specific observation windows\n\nThe goal is to let Memory, Persona, Vision, and future runtime modules consume the same multimodal observation model without depending directly on LiveKit or another RTC backend.\n\n`avatar-rtc`\n\nacts as the adapter boundary between external realtime communication systems and AlphaAvatar Core.\n\nLiveKit remains the default RTC integration today, but LiveKit-specific frame and track types are converted at the adapter boundary instead of leaking into the core runtime.\n\nThe main design principle is:\n\nRTC is an adapter, not business logic.\n\nFuture native WebRTC, `aiortc`\n\n, or other realtime backends should be able to publish the same core observations.\n\n`avatar-agents`\n\nowns orchestration and session execution.\n\nThe runtime is now separated into parallel components:\n\n`SessionRuntime`\n\n`ContextRuntime`\n\n`PerceptionRuntime`\n\nIO Runtime\n\nAgent Runtime\n\n`AvatarEngine`\n\nas the high-level orchestrator\n\nThis moves more responsibilities out of `AvatarEngine`\n\nand gives plugins clearer runtime dependencies and lifecycle boundaries.\n\nCross-cutting assistant capabilities are implemented as plugins:\n\nMemory\n\nPersona\n\nStatus\n\nCharacter\n\nInteraction Router\n\nfuture Reflection, Planning, and Behavior plugins\n\nPlugins consume runtime context and perception streams instead of independently subscribing to RTC tracks or rebuilding their own session state.\n\nTool and knowledge capabilities remain independently replaceable:\n\nMCP\n\nRAG\n\nDeepResearch\n\nfuture sandbox and external workspace integrations\n\nThese tools are also connected to Memory and Status so that long-running actions can remain visible and useful across future sessions.\n\nThe same assistant runtime can be exposed through:\n\nthe realtime web application\n\navatar interfaces\n\nfuture Discord, Slack, mobile, and external application integrations\n\nThe channel should change how the assistant is accessed, not recreate the assistant itself.\n\nOne of the main changes in v0.6.4 is the introduction of a shared perception path:\n\n```\nRTC / Device Input\n        ↓\nEnvObservation + MediaPayload\n        ↓\nPerceptionRuntime\n  ├── Typed Streams\n  ├── Shared Timeline\n  └── Window Builder\n        ↓\nPersona / Vision / Memory / Future Routers\n```\n\nPreviously, different modules could subscribe to the LiveKit video track independently.\n\nNow, an RTC adapter publishes the media once. Multiple runtime consumers can process it using their own cursors, sampling policies, buffers, and temporal windows.\n\nFor example:\n\nPersona can perform face detection and identity matching.\n\nVision can select recent frames for the current LLM turn.\n\nMemory can consume a longer observation window.\n\nFuture routers can use motion, presence, screen, or speaker events.\n\nThese consumers do not have to block one another or use the same sampling rate.\n\nv0.6.4 also introduces the first version of **online ENV Memory**.\n\nTraditional assistant memory usually begins with conversation text:\n\n```\nuser message\n    ↓\nmemory extraction\n    ↓\nlong-term memory\n```\n\nENV Memory extends this to continuously observed multimodal context:\n\n```\nlive visual observations\n        ↓\nordered perception window\n        ↓\nmultimodal ENV extraction\n        ↓\nstructured environment memory\n        ↓\nfuture visual-history retrieval\n```\n\nExamples of ENV memories include:\n\na person remaining at a desk during a session\n\na cup being placed on a table\n\na plant repeatedly appearing near the user\n\na screen showing a coding workflow\n\na person entering or leaving the visible environment\n\nan object changing position across observation windows\n\nThe goal is not to store every frame or generate endless image captions.\n\nInstead, AlphaAvatar samples observations, aligns annotations, extracts concise episodic memories, and stores only useful structured results.\n\nRaw video frames remain runtime-only and are not written directly into the vector database or Markdown memory storage.\n\nPersona can now attach face annotations to shared observations.\n\nAn observation may contain multiple representations:\n\nraw video frame\n\nraw JPEG\n\nannotated video frame\n\nannotated JPEG\n\nfuture derived representations\n\nMemory can prefer an annotated JPEG when extracting ENV context, while Vision can resolve the most recent annotated frame when building model context.\n\nThe raw representation remains unchanged.\n\nThis allows different modules to enrich the same observation without copying or overwriting the original media.\n\nThe previous v0.6.3 release introduced graph-aware Memory foundations:\n\nmulti-object memory ownership\n\ngraph node mentions\n\nsession-scoped local entity keys\n\nalias-ready graph lookup\n\nLanceDB graph-node indexing\n\ngraph-aware semantic retrieval\n\nENV Memory now plugs into the same system.\n\nA visual memory may be connected to concrete observed entities such as:\n\na visible person\n\na face track\n\na screen\n\na document\n\na plant\n\na vehicle\n\na physical object\n\na distinguishable room\n\nThe memory text remains the source of truth, while graph nodes act as sparse retrieval anchors.\n\nAnother recent change is the task-based Provider Layer.\n\nMemory, Persona, embeddings, and multimodal extraction tasks can be configured independently.\n\nFor example:\n\none model may extract conversation memory\n\nanother multimodal model may process ENV observations\n\na local embedding model may index memories\n\nanother provider may handle the realtime assistant response\n\nThe personal data and storage layer can remain self-hosted even when optional external inference providers are used.\n\nIt is possible that more orchestration will eventually become model-native.\n\nModels may increasingly handle:\n\ncontext compression\n\ntool planning\n\nmultimodal state tracking\n\nmemory selection\n\nuser modeling\n\nHowever, I still think an explicit user-owned runtime remains valuable for:\n\nprivacy boundaries\n\npersistent personal storage\n\nmodel and provider swapping\n\nobservability\n\nreplay and evaluation\n\nidentity management\n\ntool permissions\n\ndebugging\n\nuser-editable Memory and Persona\n\ncross-channel continuity\n\nThe model can become more capable without requiring personal state, tools, and system behavior to disappear into an opaque kernel.\n\nAlphaAvatar is still developer-first and under active development.\n\nCurrent limitations include:\n\nLiveKit is still the primary RTC implementation.\n\nENV Memory currently focuses mainly on sampled visual observations.\n\nVisual-history retrieval is still relatively basic.\n\nMulti-user visual and speaker routing requires more work.\n\nReflection, Planning, Behavior, and Interaction Router are not complete.\n\nUser-facing Memory and Persona inspection controls are still being developed.\n\nThe next major directions include:\n\naudio and screen-based ENV Memory\n\nricher object-, event-, identity-, and time-aware retrieval\n\ncross-window event consolidation\n\nmulti-user face and speaker alignment\n\nalternative RTC adapters\n\ninteraction routing and proactive behavior\n\nReflection and Planning\n\nreplay and evaluation tooling\n\nuser-controlled Memory and Persona editing\n\nI would be interested in feedback on a few design questions:\n\nWhich assistant capabilities should remain explicit runtime components, and which should eventually become model-native?\n\nWhat is the right evaluation framework for long-running ENV Memory?\n\nShould multimodal memory primarily store textual episodes, structured events, graph relations, selected media artifacts, or a combination?\n\nWhat runtime interface would make AlphaAvatar useful to other voice-agent, digital-human, AI companion, or AI VTuber projects?\n\nThanks to everyone who provided feedback on the original post. The architecture is still evolving, and contributions, criticism, comparisons with related projects, and implementation discussions are very welcome.", "url": "https://wpnews.pro/news/alphaavatar-v0-6-4-a-self-hostable-realtime-multimodal-assistant-runtime-with", "canonical_source": "https://discuss.huggingface.co/t/alphaavatar-v0-6-4-a-self-hostable-realtime-multimodal-assistant-runtime-with-shared-perception-and-env-memory/177684#post_1", "published_at": "2026-07-11 10:42:14+00:00", "updated_at": "2026-07-11 10:45:06.358861+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-products", "ai-tools", "ai-infrastructure", "ai-agents"], "entities": ["AlphaAvatar", "LiveKit", "MCP", "RAG", "DeepResearch"], "alternates": {"html": "https://wpnews.pro/news/alphaavatar-v0-6-4-a-self-hostable-realtime-multimodal-assistant-runtime-with", "markdown": "https://wpnews.pro/news/alphaavatar-v0-6-4-a-self-hostable-realtime-multimodal-assistant-runtime-with.md", "text": "https://wpnews.pro/news/alphaavatar-v0-6-4-a-self-hostable-realtime-multimodal-assistant-runtime-with.txt", "jsonld": "https://wpnews.pro/news/alphaavatar-v0-6-4-a-self-hostable-realtime-multimodal-assistant-runtime-with.jsonld"}}