Hi everyone
A few weeks ago, I shared an early introduction to AlphaAvatar, an open-source and self-hostable realtime full-multimodal personal AI assistant runtime.
The discussion around that post was very helpful. One particularly useful point was that AlphaAvatar may be more valuable when positioned less as another avatar frontend, and more as a reusable runtime layer for persistent realtime multimodal agents.
That is increasingly how I see the project as well.
AlphaAvatar does not try to own every ASR, TTS, RTC, model, or avatar renderer. Instead, it aims to provide the stateful runtime connecting them:
realtime multimodal perception
session and interaction state
persistent Memory and Persona
identity continuity
tools, MCP, RAG, and DeepResearch
status and workflow feedback
avatar and channel outputs
self-hosted storage and provider control
Since the previous post, the project has gone through several architecture-focused releases, culminating in v0.6.4.
The new architecture is organized around six main areas:
avatar-core
contains transport-independent multimodal primitives:
EnvObservation
EnvAnnotation
MediaPayload
raw and annotated payload views
generic video frame buffers
typed perception streams
shared timelines
consumer-specific observation windows
The goal is to let Memory, Persona, Vision, and future runtime modules consume the same multimodal observation model without depending directly on LiveKit or another RTC backend.
avatar-rtc
acts as the adapter boundary between external realtime communication systems and AlphaAvatar Core.
LiveKit remains the default RTC integration today, but LiveKit-specific frame and track types are converted at the adapter boundary instead of leaking into the core runtime.
The main design principle is:
RTC is an adapter, not business logic.
Future native WebRTC, aiortc
, or other realtime backends should be able to publish the same core observations.
avatar-agents
owns orchestration and session execution.
The runtime is now separated into parallel components:
SessionRuntime
ContextRuntime
PerceptionRuntime
IO Runtime
Agent Runtime
AvatarEngine
as the high-level orchestrator
This moves more responsibilities out of AvatarEngine
and gives plugins clearer runtime dependencies and lifecycle boundaries.
Cross-cutting assistant capabilities are implemented as plugins:
Memory
Persona
Status
Character
Interaction Router
future Reflection, Planning, and Behavior plugins
Plugins consume runtime context and perception streams instead of independently subscribing to RTC tracks or rebuilding their own session state.
Tool and knowledge capabilities remain independently replaceable:
MCP
RAG
DeepResearch
future sandbox and external workspace integrations
These tools are also connected to Memory and Status so that long-running actions can remain visible and useful across future sessions.
The same assistant runtime can be exposed through:
the realtime web application
avatar interfaces
future Discord, Slack, mobile, and external application integrations
The channel should change how the assistant is accessed, not recreate the assistant itself.
One of the main changes in v0.6.4 is the introduction of a shared perception path:
RTC / Device Input
↓
EnvObservation + MediaPayload
↓
PerceptionRuntime
├── Typed Streams
├── Shared Timeline
└── Window Builder
↓
Persona / Vision / Memory / Future Routers
Previously, different modules could subscribe to the LiveKit video track independently.
Now, an RTC adapter publishes the media once. Multiple runtime consumers can process it using their own cursors, sampling policies, buffers, and temporal windows.
For example:
Persona can perform face detection and identity matching.
Vision can select recent frames for the current LLM turn.
Memory can consume a longer observation window.
Future routers can use motion, presence, screen, or speaker events.
These consumers do not have to block one another or use the same sampling rate.
v0.6.4 also introduces the first version of online ENV Memory.
Traditional assistant memory usually begins with conversation text:
user message
↓
memory extraction
↓
long-term memory
ENV Memory extends this to continuously observed multimodal context:
live visual observations
↓
ordered perception window
↓
multimodal ENV extraction
↓
structured environment memory
↓
future visual-history retrieval
Examples of ENV memories include:
a person remaining at a desk during a session
a cup being placed on a table
a plant repeatedly appearing near the user
a screen showing a coding workflow
a person entering or leaving the visible environment
an object changing position across observation windows
The goal is not to store every frame or generate endless image captions.
Instead, AlphaAvatar samples observations, aligns annotations, extracts concise episodic memories, and stores only useful structured results.
Raw video frames remain runtime-only and are not written directly into the vector database or Markdown memory storage.
Persona can now attach face annotations to shared observations.
An observation may contain multiple representations:
raw video frame
raw JPEG
annotated video frame
annotated JPEG
future derived representations
Memory can prefer an annotated JPEG when extracting ENV context, while Vision can resolve the most recent annotated frame when building model context.
The raw representation remains unchanged.
This allows different modules to enrich the same observation without copying or overwriting the original media.
The previous v0.6.3 release introduced graph-aware Memory foundations:
multi-object memory ownership
graph node mentions
session-scoped local entity keys
alias-ready graph lookup
LanceDB graph-node indexing
graph-aware semantic retrieval
ENV Memory now plugs into the same system.
A visual memory may be connected to concrete observed entities such as:
a visible person
a face track
a screen
a document
a plant
a vehicle
a physical object
a distinguishable room
The memory text remains the source of truth, while graph nodes act as sparse retrieval anchors.
Another recent change is the task-based Provider Layer.
Memory, Persona, embeddings, and multimodal extraction tasks can be configured independently.
For example:
one model may extract conversation memory
another multimodal model may process ENV observations
a local embedding model may index memories
another provider may handle the realtime assistant response
The personal data and storage layer can remain self-hosted even when optional external inference providers are used.
It is possible that more orchestration will eventually become model-native.
Models may increasingly handle:
context compression
tool planning
multimodal state tracking
memory selection
user modeling
However, I still think an explicit user-owned runtime remains valuable for:
privacy boundaries
persistent personal storage
model and provider swapping
observability
replay and evaluation
identity management
tool permissions
debugging
user-editable Memory and Persona
cross-channel continuity
The model can become more capable without requiring personal state, tools, and system behavior to disappear into an opaque kernel.
AlphaAvatar is still developer-first and under active development.
Current limitations include:
LiveKit is still the primary RTC implementation.
ENV Memory currently focuses mainly on sampled visual observations.
Visual-history retrieval is still relatively basic.
Multi-user visual and speaker routing requires more work.
Reflection, Planning, Behavior, and Interaction Router are not complete.
User-facing Memory and Persona inspection controls are still being developed.
The next major directions include:
audio and screen-based ENV Memory
richer object-, event-, identity-, and time-aware retrieval
cross-window event consolidation
multi-user face and speaker alignment
alternative RTC adapters
interaction routing and proactive behavior
Reflection and Planning
replay and evaluation tooling
user-controlled Memory and Persona editing
I would be interested in feedback on a few design questions:
Which assistant capabilities should remain explicit runtime components, and which should eventually become model-native?
What is the right evaluation framework for long-running ENV Memory?
Should multimodal memory primarily store textual episodes, structured events, graph relations, selected media artifacts, or a combination?
What runtime interface would make AlphaAvatar useful to other voice-agent, digital-human, AI companion, or AI VTuber projects?
Thanks to everyone who provided feedback on the original post. The architecture is still evolving, and contributions, criticism, comparisons with related projects, and implementation discussions are very welcome.