Hi everyone
In my previous AlphaAvatar v0.6.6 post, I shared how the project was moving toward shared multimodal perception, event-driven Memory, unified inference runners, and more explicit runtime contracts.
Since then, v0.6.7 introduced AlphaAvatar-owned multimodal turn-taking and semantic addressing, reducing the dependence on transport-specific interaction policies.
With AlphaAvatar v0.6.8, the project reaches a larger architectural milestone.
AlphaAvatar now has its own native realtime Agent runtime, independent of the LiveKit Agents SDK.
This does not mean removing LiveKit. LiveKit remains the first supported realtime communication backend. What changes is that LiveKit no longer owns the Agent lifecycle, model execution, tool orchestration, or the application’s runtime architecture.
This release also introduces:
A native AvatarHost responsible for application and execution lifecycle management
Processor-owned Persona, Memory, and Router capabilities
A runtime capability system that goes beyond prompt descriptions
A native streaming model and tool execution pipeline
Streaming PCM speech with playback-synchronized transcripts
Unified workspace organization, diagnostics, and Python 3.13 tooling
For me, the most important part is not any individual feature.
It is that the Agent, its execution, its capabilities, and its realtime connection now have more explicit ownership boundaries.
Earlier versions of AlphaAvatar used LiveKit Agents for more than realtime communication.
The SDK also managed parts of application startup, worker execution, Agent sessions, and foreground model orchestration.
This was useful for building the initial system, but it gradually created architectural constraints.
For example:
Should a realtime connection determine whether an Agent execution exists?
Should model execution be tied to a particular RTC SDK?
Who should own shared inference runners used by multiple components?
How should an Agent eventually continue working after a user disconnects?
Can the same Agent runtime support a server, desktop application, mobile device, or another future environment?
These questions are difficult to answer cleanly when communication, execution, and application lifecycle are coupled.
In v0.6.8, AlphaAvatar introduces a standalone avatar-host package.
The architecture is now approximately:
alphaavatar dev / start
|
v
AvatarHost
|
+------------+-------------+
| | |
Application Execution Session
Services Management Management
| | |
| | RTC Backend
| | |
| | LiveKit
| |
+------------+
|
v
AvatarEngine
|
+------------+-------------+
| | |
Core Foundation Plugins
Perception Services & Tools
| | |
+------------+-------------+
|
v
Native Model / Tool Loop
|
v
Output / Voice Runtime
|
v
RTC Media & Transcripts
The Host coordinates application initialization, managed executions, shared services, process supervision, and session attachments.
The Agent runtime remains responsible for perception, turn handling, model/tool orchestration, and output production.
The RTC integration is handled through a separate avatar-rtc package and Host session adapters.
Importantly, LiveKit remains the implemented RTC backend, but livekit.agents is no longer part of the active Agent execution path.
This is an architectural separation rather than a claim that all RTC transports are already supported.
One distinction I wanted to establish is that several concepts previously treated as closely related are actually different:
Application lifecycle
|
+-- Managed Agent execution
| |
| +-- Model / Tool Runs
| |
| +-- Runtime services
|
+-- Interactive sessions
|
+-- RTC connections
+-- Media transport
An interactive session represents access to an Agent through a particular connection.
A model Run represents a bounded model/tool execution.
A managed Agent execution represents a larger runtime lifecycle.
These should not be interchangeable identities.
For example, a user disconnecting from a browser does not necessarily imply that every shared runtime service should be destroyed.
Conversely, a completed model response does not imply that generated speech has finished playing.
v0.6.8 establishes these boundaries in the Host and runtime contracts.
However, I want to be precise about what is implemented.
Connection-independent, persistent Agent execution is not yet provided by this release. A configured execution may still end when its session terminates.
Durable Episodes, persistent Context recovery, execution resumption, and autonomous background operation remain future work.
The purpose of this release is to establish the ownership model needed to implement those features without restructuring the entire application again.
Another substantial change is how plugins are structured.
Previously, some plugin functionality was still implemented directly in the Agent package, while parts of the plugin lifecycle and configuration were managed at a broader level.
In v0.6.8, Persona and Memory have been reorganized around independently managed processors.
Persona Plugin
|
+-- Profiler Processor
+-- Speaker Processor
+-- Face Processor
Memory Plugin
|
+-- Conversation Processor
+-- Tool Processor
+-- Environment Processor
+-- Retrieval Processor
Each processor can own its configuration, providers, internal implementation, and lifecycle.
The Agent package retains reusable interfaces and runtime composition rather than embedding plugin-specific business logic.
Router has also moved further toward processor-owned organization for addressing and turn-taking.
This follows a principle I have been gradually adopting across AlphaAvatar:
A plugin should not merely register a feature. It should own the implementation and lifecycle of that feature.
This becomes increasingly important as plugins become more complex.
For example, Persona may have different models for speaker recognition, face recognition, and profile extraction.
Memory may have different processing policies for conversations, tool outcomes, environment observations, and retrieval.
Those capabilities should not need to share one large lifecycle or configuration implementation simply because they belong to the same plugin.
The workspace now organizes plugins into three groups:
Foundation: Context, Loop, Provider, Voice
Perception: Character, Memory, Persona, Router, Status
Tools: DeepResearch, MCP, RAG
These are organizational boundaries; they do not require changing public Python import paths to include the group names.
In v0.6.6, I introduced a way for plugins to describe their capabilities to the Avatar.
At that point, the primary goal was to let the model understand which features existed without hard-coding every possible plugin into the system prompt.
But there was an important limitation:
Describing a capability is not the same as making it executable.
A runtime needs to distinguish between:
A capability being registered
Its availability in the current runtime
Its visibility to the model
Whether a particular invocation is authorized
Whether it can actually execute
What happened after execution
v0.6.8 promotes capabilities into typed, runtime-registered definitions with callable bindings.
Conceptually:
Plugin / Processor
|
v
Capability Registration
|
v
Runtime Capability Registry
|
+-- Identity / Schema
+-- Visibility / Scope
+-- Authorization
+-- Invocation
|
v
Native Model / Tool Loop
|
v
Validated Execution Result
This also allows the native Loop to invoke tools from MCP, RAG, DeepResearch, and other registered capabilities through a consistent execution boundary.
There is an additional detail I think is important for practical Agents.
Cancellation does not imply that an external action never happened.
If a tool call has already been dispatched, interrupting the Agent does not automatically make it safe to execute the same action again.
The native runtime preserves these distinctions instead of assuming that failed or interrupted executions are always retryable.
Removing the LiveKit Agents execution dependency also required AlphaAvatar to own its model and tool pipeline.
v0.6.8 introduces a native Loop that coordinates:
Capturing the query’s runtime context
Preparing structured model input
Consuming streaming model responses
Validating and authorizing tool calls
Executing capabilities
Returning tool results to the model
Publishing canonical records and user-facing outputs
The Provider layer now separates shared model contracts from specific SDK implementations.
The release includes a native streaming implementation for the OpenAI Responses API, with structured input/output events, function calls, continuation handling, and explicit protocol validation.
Other provider integrations continue to exist, although not every task backend implements the same foreground streaming capabilities.
One useful outcome of this change is that the runtime can distinguish several states that are often conflated:
User input accepted
|
v
Committed Turn
|
v
Model / Tool Execution
|
v
Canonical Output Record
|
v
Output Processing
|
v
Audio / Text Delivery
For a realtime assistant, these distinctions matter.
A model completing its response does not mean the user has heard it.
A generated answer does not mean its audio synthesis succeeded.
An output being submitted to an RTC transport does not necessarily mean it reached the physical speaker.
Similarly, Memory processors should consume authoritative input and model/tool records rather than reconstructing conversation history from incidental transport events.
v0.6.8 makes more of those boundaries explicit through committed-turn events, canonical records, execution observations, and output delivery state.
Realtime speech is another major focus of this release.
Previously, parts of the output path relied more heavily on sentence-complete synthesis and delivery.
v0.6.8 moves to a native streaming PCM pipeline.
The simplified flow is:
Streaming LLM output
|
v
Incremental Text Segmentation
|
v
Asynchronous TTS
|
v
Streaming PCM / Bounded Buffers
|
+------> RTC Audio Output
|
v
Playback Progress
|
v
Transcript Synchronization
|
v
RTC Text / Captions
Speech can begin before the entire model response or sentence sequence has been generated.
Text segmentation balances early speech startup against natural phrase boundaries.
The synthesis pipeline handles partial audio, bounded buffering, cancellation, and finite retry policies.
An additional improvement is playback-paced transcript synchronization.
Instead of releasing all generated text as soon as the LLM produces it, AlphaAvatar coordinates transcript delivery with local audio playout progress.
The intention is to make captions more closely follow the speech actually being delivered.
At the same time, caption delivery must not block PCM playback.
Slow transcript consumers or transport feedback should not become the pacing authority for audio frames.
The pipeline also includes configurable first-audio hedging for slow TTS starts. This can improve startup behavior in some conditions, but may issue additional synthesis requests and increase provider costs.
I do not consider these changes a universal latency guarantee.
Real provider response times, network behavior, RTC buffering, and device playback still need to be measured separately.
One further addition is support for model-generated commentary before appropriate tool operations.
For example, the assistant can briefly explain what it is about to investigate while a longer tool operation proceeds.
This is generated through the normal model/output pipeline rather than being a separate hidden reasoning request.
It is optional model behavior, not a mandatory spoken status message for every tool call.
There are also several infrastructure improvements.
The project now targets CPython 3.13, with a unified uv workspace, updated dependency lockfile, packaging checks, Ruff configuration, and CI validation.
The public CLI has also moved to the Host:
alphaavatar dev <config.yaml>
alphaavatar start <config.yaml>
Runtime diagnostics now include more structured information about Host execution, process ownership, model/tool activity, interruption, and output delivery.
The Host also supports structured JSONL logs and configurable developer-facing conversation previews.
I think observability is especially important for realtime Agents because a perceived delay can originate from very different places:
Perception or turn admission
Model response generation
Tool execution
TTS startup
Audio buffering
RTC transport
Client playback
Without separating these stages, it is easy to attribute a problem to the model when the actual cause is elsewhere.
For anyone trying v0.6.8, I currently recommend installing from the matching GitHub source tag and lockfile, rather than assuming every workspace package is available on PyPI.
The project contains 16 Python workspace packages, but the GitHub Release and PyPI publication processes are separate.
I have been thinking about AlphaAvatar less as an avatar application and more as a reusable runtime for persistent, multimodal personal Agents.
An avatar is one way to interact with that runtime.
A browser, desktop application, mobile device, or future device-native environment may be another.
Likewise, LiveKit is one way to connect realtime inputs and outputs.
But none of those should necessarily define how the Agent reasons, maintains Memory, invokes tools, or manages its execution.
My longer-term mental model is becoming:
Applications / Devices
|
v
Interaction Layer
|
v
AvatarHost
|
v
Agent Runtime
|
+------------+------------+
| | |
Perception Memory Persona
| | |
+------------+------------+
|
Context / Model
|
Tool / Action
|
Runtime Outcomes
|
v
Output Delivery
The runtime should provide stable contracts for identity, state, capabilities, execution, and outputs.
The model should provide the reasoning and decision-making capabilities that connect those components.
This separation does not mean the runtime should attempt to reproduce intelligence already provided by the model.
Instead, I see its role as making model capabilities operationally useful: configurable, observable, replaceable, and eventually persistent across different execution environments.
There are still several important limitations.
First, persistent Agent execution remains unfinished.
The new Host establishes separation between execution and interactive connection ownership, but it does not yet provide durable Episodes, automatic task recovery, or independent long-running background execution.
Second, LiveKit is currently the implemented RTC backend.
The architecture is designed to accommodate other backends, but support for arbitrary RTC implementations has not been demonstrated.
Third, the Character frontend is still under validation.
The current AIRI integration is optional, and its complete renderer/RTC behavior is not yet considered validated.
Fourth, broader reliability and performance testing is still needed.
CI checks cover packaging, installation, imports, and important lifecycle contracts, but these are not substitutes for long-duration, multi-participant, high-load, or cross-device testing.
Finally, developer onboarding still needs work, particularly around configuration, installation diagnostics, provider setup, and platform-specific dependencies.
These areas are part of the next development steps.
I would be interested in how others building realtime or persistent Agents approach these problems:
1. Where should the boundary between an interactive session and an Agent execution be?
Should an Agent always be able to continue after the connection disappears, or should background execution require an explicit separate lifecycle?
2. How do you represent completion in streaming Agent systems?
Model completion, tool completion, canonical output publication, audio synthesis, and actual playback can all occur at different times. Which of these should be first-class runtime contracts?
3. How should runtime capabilities be exposed to a model?
Now that capabilities can be registered and invoked through a runtime registry, should a model receive the complete active capability list, or only a small dynamically selected subset?
4. What is the best approach to streaming speech and transcript synchronization?
In particular, how do you handle partial TTS output, interruption, output backpressure, and the difference between server-side playout estimates and actual client playback?
I would be especially interested in real operational experiences, architectural criticism, and failure cases from people building voice Agents, multimodal assistants, or persistent Agent runtimes.
GitHub: GitHub - AlphaAvatar/AlphaAvatar: An Omni Real-Time Assistant for Everyone · GitHub
v0.6.8 Release: Release AlphaAvatar v0.6.8 · AlphaAvatar/AlphaAvatar · GitHub
Full Release Notes: AlphaAvatar/docs/releases/v0.6.8.md at v0.6.8 · AlphaAvatar/AlphaAvatar · GitHub
Previous v0.6.6 Discussion: AlphaAvatar v0.6.6: event-driven multimodal memory, unified runtimes, and cleaner agent contracts
Thanks again to everyone who commented on the earlier AlphaAvatar posts.
Some of those discussions, particularly around runtime boundaries, capability semantics, observation provenance, and persistent state, have been useful in shaping the direction of the project.
AlphaAvatar is still evolving quickly, and I would welcome feedback from anyone exploring similar architectural problems.