AlphaAvatar v0.6.8: Native Agent Runtime, Processor-Owned Plugins, and Streaming Voice AlphaAvatar released v0.6.8, giving the open-source avatar project its own native realtime Agent runtime that no longer depends on the LiveKit Agents SDK to own the Agent lifecycle, model execution, or tool orchestration. The release adds a standalone `avatar-host` package with a native AvatarHost for application and execution lifecycle management, processor-owned Persona, Memory and Router capabilities, a runtime capability system, a native streaming model and tool execution pipeline, streaming PCM speech with playback-synchronized transcripts, and Python 3.13 tooling. LiveKit remains the first supported realtime communication backend, now isolated in a separate `avatar-rtc` package and Host session adapters, so that realtime connections no longer determine whether an Agent execution exists. Hi everyone In my previous AlphaAvatar v0.6.6 post https://discuss.huggingface.co/t/alphaavatar-v0-6-6-event-driven-multimodal-memory-unified-runtimes-and-cleaner-agent-contracts/178766 , I shared how the project was moving toward shared multimodal perception, event-driven Memory, unified inference runners, and more explicit runtime contracts. Since then, v0.6.7 introduced AlphaAvatar-owned multimodal turn-taking and semantic addressing, reducing the dependence on transport-specific interaction policies. With AlphaAvatar v0.6.8 , the project reaches a larger architectural milestone. AlphaAvatar now has its own native realtime Agent runtime, independent of the LiveKit Agents SDK. This does not mean removing LiveKit. LiveKit remains the first supported realtime communication backend. What changes is that LiveKit no longer owns the Agent lifecycle, model execution, tool orchestration, or the application’s runtime architecture. This release also introduces: A native AvatarHost responsible for application and execution lifecycle management Processor-owned Persona, Memory, and Router capabilities A runtime capability system that goes beyond prompt descriptions A native streaming model and tool execution pipeline Streaming PCM speech with playback-synchronized transcripts Unified workspace organization, diagnostics, and Python 3.13 tooling For me, the most important part is not any individual feature. It is that the Agent, its execution, its capabilities, and its realtime connection now have more explicit ownership boundaries. Earlier versions of AlphaAvatar used LiveKit Agents for more than realtime communication. The SDK also managed parts of application startup, worker execution, Agent sessions, and foreground model orchestration. This was useful for building the initial system, but it gradually created architectural constraints. For example: Should a realtime connection determine whether an Agent execution exists? Should model execution be tied to a particular RTC SDK? Who should own shared inference runners used by multiple components? How should an Agent eventually continue working after a user disconnects? Can the same Agent runtime support a server, desktop application, mobile device, or another future environment? These questions are difficult to answer cleanly when communication, execution, and application lifecycle are coupled. In v0.6.8, AlphaAvatar introduces a standalone avatar-host package. The architecture is now approximately: alphaavatar dev / start | v AvatarHost | +------------+-------------+ | | | Application Execution Session Services Management Management | | | | | RTC Backend | | | | | LiveKit | | +------------+ | v AvatarEngine | +------------+-------------+ | | | Core Foundation Plugins Perception Services & Tools | | | +------------+-------------+ | v Native Model / Tool Loop | v Output / Voice Runtime | v RTC Media & Transcripts The Host coordinates application initialization, managed executions, shared services, process supervision, and session attachments. The Agent runtime remains responsible for perception, turn handling, model/tool orchestration, and output production. The RTC integration is handled through a separate avatar-rtc package and Host session adapters. Importantly, LiveKit remains the implemented RTC backend, but livekit.agents is no longer part of the active Agent execution path. This is an architectural separation rather than a claim that all RTC transports are already supported. One distinction I wanted to establish is that several concepts previously treated as closely related are actually different: Application lifecycle | +-- Managed Agent execution | | | +-- Model / Tool Runs | | | +-- Runtime services | +-- Interactive sessions | +-- RTC connections +-- Media transport An interactive session represents access to an Agent through a particular connection. A model Run represents a bounded model/tool execution. A managed Agent execution represents a larger runtime lifecycle. These should not be interchangeable identities. For example, a user disconnecting from a browser does not necessarily imply that every shared runtime service should be destroyed. Conversely, a completed model response does not imply that generated speech has finished playing. v0.6.8 establishes these boundaries in the Host and runtime contracts. However, I want to be precise about what is implemented. Connection-independent, persistent Agent execution is not yet provided by this release. A configured execution may still end when its session terminates. Durable Episodes, persistent Context recovery, execution resumption, and autonomous background operation remain future work. The purpose of this release is to establish the ownership model needed to implement those features without restructuring the entire application again. Another substantial change is how plugins are structured. Previously, some plugin functionality was still implemented directly in the Agent package, while parts of the plugin lifecycle and configuration were managed at a broader level. In v0.6.8, Persona and Memory have been reorganized around independently managed processors. Persona Plugin | +-- Profiler Processor +-- Speaker Processor +-- Face Processor Memory Plugin | +-- Conversation Processor +-- Tool Processor +-- Environment Processor +-- Retrieval Processor Each processor can own its configuration, providers, internal implementation, and lifecycle. The Agent package retains reusable interfaces and runtime composition rather than embedding plugin-specific business logic. Router has also moved further toward processor-owned organization for addressing and turn-taking. This follows a principle I have been gradually adopting across AlphaAvatar: A plugin should not merely register a feature. It should own the implementation and lifecycle of that feature. This becomes increasingly important as plugins become more complex. For example, Persona may have different models for speaker recognition, face recognition, and profile extraction. Memory may have different processing policies for conversations, tool outcomes, environment observations, and retrieval. Those capabilities should not need to share one large lifecycle or configuration implementation simply because they belong to the same plugin. The workspace now organizes plugins into three groups: Foundation: Context, Loop, Provider, Voice Perception: Character, Memory, Persona, Router, Status Tools: DeepResearch, MCP, RAG These are organizational boundaries; they do not require changing public Python import paths to include the group names. In v0.6.6, I introduced a way for plugins to describe their capabilities to the Avatar. At that point, the primary goal was to let the model understand which features existed without hard-coding every possible plugin into the system prompt. But there was an important limitation: Describing a capability is not the same as making it executable. A runtime needs to distinguish between: A capability being registered Its availability in the current runtime Its visibility to the model Whether a particular invocation is authorized Whether it can actually execute What happened after execution v0.6.8 promotes capabilities into typed, runtime-registered definitions with callable bindings. Conceptually: Plugin / Processor | v Capability Registration | v Runtime Capability Registry | +-- Identity / Schema +-- Visibility / Scope +-- Authorization +-- Invocation | v Native Model / Tool Loop | v Validated Execution Result This also allows the native Loop to invoke tools from MCP, RAG, DeepResearch, and other registered capabilities through a consistent execution boundary. There is an additional detail I think is important for practical Agents. Cancellation does not imply that an external action never happened. If a tool call has already been dispatched, interrupting the Agent does not automatically make it safe to execute the same action again. The native runtime preserves these distinctions instead of assuming that failed or interrupted executions are always retryable. Removing the LiveKit Agents execution dependency also required AlphaAvatar to own its model and tool pipeline. v0.6.8 introduces a native Loop that coordinates: Capturing the query’s runtime context Preparing structured model input Consuming streaming model responses Validating and authorizing tool calls Executing capabilities Returning tool results to the model Publishing canonical records and user-facing outputs The Provider layer now separates shared model contracts from specific SDK implementations. The release includes a native streaming implementation for the OpenAI Responses API, with structured input/output events, function calls, continuation handling, and explicit protocol validation. Other provider integrations continue to exist, although not every task backend implements the same foreground streaming capabilities. One useful outcome of this change is that the runtime can distinguish several states that are often conflated: User input accepted | v Committed Turn | v Model / Tool Execution | v Canonical Output Record | v Output Processing | v Audio / Text Delivery For a realtime assistant, these distinctions matter. A model completing its response does not mean the user has heard it. A generated answer does not mean its audio synthesis succeeded. An output being submitted to an RTC transport does not necessarily mean it reached the physical speaker. Similarly, Memory processors should consume authoritative input and model/tool records rather than reconstructing conversation history from incidental transport events. v0.6.8 makes more of those boundaries explicit through committed-turn events, canonical records, execution observations, and output delivery state. Realtime speech is another major focus of this release. Previously, parts of the output path relied more heavily on sentence-complete synthesis and delivery. v0.6.8 moves to a native streaming PCM pipeline. The simplified flow is: Streaming LLM output | v Incremental Text Segmentation | v Asynchronous TTS | v Streaming PCM / Bounded Buffers | +------ RTC Audio Output | v Playback Progress | v Transcript Synchronization | v RTC Text / Captions Speech can begin before the entire model response or sentence sequence has been generated. Text segmentation balances early speech startup against natural phrase boundaries. The synthesis pipeline handles partial audio, bounded buffering, cancellation, and finite retry policies. An additional improvement is playback-paced transcript synchronization . Instead of releasing all generated text as soon as the LLM produces it, AlphaAvatar coordinates transcript delivery with local audio playout progress. The intention is to make captions more closely follow the speech actually being delivered. At the same time, caption delivery must not block PCM playback. Slow transcript consumers or transport feedback should not become the pacing authority for audio frames. The pipeline also includes configurable first-audio hedging for slow TTS starts. This can improve startup behavior in some conditions, but may issue additional synthesis requests and increase provider costs. I do not consider these changes a universal latency guarantee. Real provider response times, network behavior, RTC buffering, and device playback still need to be measured separately. One further addition is support for model-generated commentary before appropriate tool operations . For example, the assistant can briefly explain what it is about to investigate while a longer tool operation proceeds. This is generated through the normal model/output pipeline rather than being a separate hidden reasoning request. It is optional model behavior, not a mandatory spoken status message for every tool call. There are also several infrastructure improvements. The project now targets CPython 3.13 , with a unified uv workspace, updated dependency lockfile, packaging checks, Ruff configuration, and CI validation. The public CLI has also moved to the Host: alphaavatar dev