#
Posting this in Research since it’s really a question about model coordination more than a specific library issue.
The problem we were solving: an agent needs to hold a full-duplex voice conversation while also doing complex visual reasoning over a screen and executing device actions in the background without either one blocking the other.
Full-duplex speech models like Moshi (parallel-stream, processing user and system audio simultaneously) and the dual-tower approach in Generative Spoken Dialogue Language Modeling are genuinely good at natural turn-taking and simple tool calls. What they’re not necessarily built for is complex, long-horizon visual reasoning over a device task, which is the other half of what we needed.
So instead of trying to get one model to do both jobs well, we split it: a real-time voice model stays present in the conversation, a separate and stronger model handles the actual device-control task in the background, and the two coordinate asynchronously through a task queue rather than sharing one agent loop.
A few design decisions that came out of this that might be relevant to others working on similar foreground/background coordination problems:
- Task completion and task success are tracked as genuinely separate signals.
Completedjust means execution reached an end state: it says nothing about whether the goal was achieved. Keeping these distinct means the conversational layer never has to guess whether a “finished” backend task actually succeeded. - Results reaching the foreground go through a queue with a 500ms aggregation window: if a second result lands while the window’s open, it extends and the results get merged before delivery, rather than firing off multiple competing responses in a row.
- Two runtime-specific message types get converted to ordinary
UserMessageobjects at the model boundary: one carries environment state changes (avoiding a full system-prompt rewrite, and the cache invalidation that comes with it, every time something shifts), the other carries runtime events like loop-detection corrections or backend task results, through one consistent channel. - Device tasks are executed strictly serially by design: most tasks need exclusive ownership of a screen and input path, and we decided the ambiguity of concurrent execution wasn’t worth trying to solve yet.
Full technical writeup: https://aidenai.io/blog/when-voice-meets-the-physical-world-inside-aidens-full-duplex-agent-architecture/
This is dev-board stage, not validated across every device/OS/audio configuration. Interested in how others here have approached coordinating a real-time conversational model with a separate, more capable model for the actual task, particularly curious about interruption handling when the backend is mid-task and new voice input arrives, since that’s the part we’re least confident is fully solved.
The async task-queue split makes sense, but I’d add a third signal next to completed and succeeded: still relevant. One failure mode this design invites is a task succeeding after the user has already revised or abandoned the intent, so the foreground delivers a now-wrong result.
I’d key each task and result to a conversation-state generation. The 500ms merge window still helps with presentation timing, but does the queue have any reverse path for the foreground to supersede or cancel work that’s already in flight?