GPT-Live-1 Tool Delegation: Keep Voice Agents Honest During Slow Work

OpenAI's GPT-Live-1, a full-duplex voice model that can delegate reasoning and tool use to a backend agent, requires a typed "delegation contract" to keep voice agents from talking over customers, repeating actions, or losing the thread during slow tool runs, according to a guide for AI builders. OpenAI's launch material names pauses, interruptions, backchannels, and spoken self-corrections as evaluation targets, and developers on Reddit's LocalLLaMA forum report that the sequential speech-to-text, LLM, and text-to-speech pipeline works for demos but breaks down once interruptions and streaming arrive. The guide argues GPT-Live-1 should serve as the conversation runtime only, with a separate backend owning facts, permissions, tool execution, durable state, and proof that an action happened.

A full-duplex model can make a voice demo feel magical. A typed delegation contract is what keeps the same agent from talking over customers, repeating an action, or losing the thread when a tool takes ten seconds. For AI builders, product engineers, and teams moving voice agents beyond a scripted demo. The conversation needs to remain responsive even when the work behind it is not. Voice agents used to have a simple failure mode: they sounded slow. The microphone stopped, speech became text, a model thought, text became speech, and the caller waited through every handoff. GPT-Live-1 changes that shape. It can listen and speak at the same time, react to an interruption, and hand deeper work to a back-end agent. That does not make the application architecture disappear. It makes its weakest decisions more obvious. A caller says, “Actually, make it Thursday,” while a booking tool is still running. Should the agent stop talking? Cancel the pending request? Ask one question? Let the tool finish and then check whether its result is stale? Those are product and control decisions, not voice-quality settings. The useful shift is this: treat GPT-Live-1 as the conversation runtime , not as the owner of your business workflow. Its job is to keep the exchange humane. A separate backend should own facts, permissions, tool execution, durable state, and the proof that an action happened. The boundary between those layers is a delegation contract — the real subject of this guide. The practical payoff: people can interrupt naturally, while your system still knows which request is active, which tool run is safe to commit, and what must be confirmed before anything changes. In a classic speech-to-text, LLM, and text-to-speech chain, each component needs a fairly clean turn boundary. That is awkward, but it gives engineers a crude source of truth: the user has stopped, so now the system may act. A full-duplex conversation has no such luxury. A pause may mean “I am thinking,” “I expect you to acknowledge me,” or “I am done.” Background speech may be a colleague, television audio, or a customer correcting themselves. OpenAI describes GPT-Live-1 as a full-duplex voice model that can delegate reasoning and tool use to a backend agent. Its launch material explicitly calls out pauses, interruptions, backchannels, and spoken requests with self-corrections as evaluation targets. That is a strong reason to move the speech interaction out of a fragile client-side state machine. It is not a reason to let the model directly mutate a booking, account, or database. Developers on Reddit are describing the same gap from the implementation side. A recent LocalLLaMA discussion called the familiar sequential pipeline adequate for demos but painful once interruptions and streaming arrive. Another thread on a local voice orchestrator argued that deciding when a person has actually stopped talking can take more engineering time than the orchestration itself. The pain is not “which voice sounds best?” It is ownership of a live, changing intent. A dependable design separates responsibilities. You do not need four microservices on day one. You do need four clear boundaries. This is the low-latency path: microphone capture, playback, connection health, echo control, device changes, and session transport. Its success metric is continuity. It should know that audio is flowing, but it should not decide that a customer is eligible for a refund. This is where GPT-Live-1 belongs. It interprets conversational signals, chooses whether to listen or respond, and explains what is happening in plain language. Keep it focused on the latest stated intent. If a user interrupts, it should quickly acknowledge the correction without claiming that the old request was cancelled unless the control plane confirms it. The work plane runs retrieval, rules, reasoning, and external tools. It converts a conversational request into a typed task: look up an order, draft a reply, find an appointment, or calculate an estimate. It may take seconds. That is fine. It must return a structured result, not merely a paragraph that the voice model has to guess how to use. This is the small but critical layer that tracks intent versions, authorization, idempotency keys, approval state, and the lifecycle of each tool run. It decides whether a delayed result is still relevant. It is also the only layer allowed to turn “please cancel it” into a real cancellation. A voice model should coordinate the conversation, while typed tasks and commitments stay outside the audio loop. Transcripts are helpful evidence. They are a poor transaction log. In a live call, the transcript can be revised, partial, or ambiguous. Build a compact task record whenever a user makes a request that could cause work or a side effect. type VoiceTask = { taskId: string; sessionId: string; intentVersion: number; intent: "find slot" | "book slot" | "cancel booking"; status: "proposed" | "running" | "needs confirmation" | "committed" | "superseded"; idempotencyKey: string; arguments: Record<string, unknown ;}; // A new correction advances the version.// Tool results may only commit if their version is still current.if result.intentVersion == currentTask.intentVersion { markSuperseded result.taskId ; return { deliverToVoice: false };} Suppose the caller asks for a 4 p.m. slot, then says “No, Thursday morning.” Create a new intent version. If the 4 p.m. search returns later, it can still be logged for diagnosis, but it must not steer the conversation or reserve anything. This one rule removes a surprising amount of “the agent ignored me” behavior. For state-changing actions, add a confirmation boundary. The work plane may prepare a cancellation or a reservation. The voice model should summarize the exact commitment in natural language. The control plane commits only after the confirmation is unambiguous and the request still has the current version. “Barge-in” is often described as a playback feature: stop the audio when the user speaks. In a production agent it is a state transition across several systems. If you stop only the speaker, a slow backend job can still return and pull the conversation backward. When the user begins a meaningful interruption, do four things: Not every sound is an interruption. A good product makes this a policy, with tests. For a phone-order assistant, a quiet “mm-hm” should probably not cancel the response. “Wait, no onions” should. A second person speaking in the room should not become a new customer request. The exact classifier may live in the voice model, an audio layer, or both. The product contract belongs to you. Full-duplex voice creates an easy temptation: fill every tool delay with friendly chatter. Resist it. Empty reassurance can be worse than a short pause, especially when an action affects money, health, travel, or a customer record. The agent should distinguish between three states: Return an explicit tool contract to the voice layer. It should include a user-safe status message, a canonical result, what confidence or validation occurred, and whether the result requires confirmation. Do not make the voice model infer a commitment from an unstructured tool response. { "task id": "task 82", "intent version": 7, "outcome": "available slot found", "speak now": "I found an opening Thursday at 10:30 a.m.", "next action": "ask confirmation", "commit allowed": false, "evidence": {"calendar id": "clinic-east", "slot id": "sl 104"}} This keeps the friendly layer honest. It can make the message warmer, but it cannot silently promote a lookup into a booking. The common failure is subtle. A voice model says, “I’ll check that,” then a backend receives a loose natural-language summary. The backend guesses the action, returns a loose paragraph, and the voice model decides what it means. That loop feels flexible until a caller revises one detail or a tool returns partial data. A delegation contract replaces guesswork with a small shared vocabulary. Every handoff should carry five things: the task type, a versioned argument object, an effect class, a delivery rule, and an evidence object. The effect class can be read, prepare, or commit. A delivery rule says whether a result may be spoken immediately, must wait for user confirmation, or is diagnostic-only because it is stale. Evidence contains the canonical IDs and validation facts that make a later audit possible. That structure lets different backends evolve without changing the conversation model’s safety promise. You may route a simple lookup to a fast service, a complex decision to a reasoning agent, and a sensitive transaction to a deterministic workflow. Each one returns the same contract. The voice layer stays free to sound natural, but it never has to invent the state of the world. Useful design test: if a delayed tool response can be read aloud without its task ID, intent version, effect class, and delivery rule, the contract is not strict enough for an interruptible voice experience. Voice sessions fail in ways a text chat hides: a Bluetooth route changes, a caller loses signal, the browser background-throttles, or someone hangs up while a tool is working. Treat the media session as disposable. Treat task state as durable. Persist the minimum viable state outside the live connection: session identity, current intent version, active task IDs, confirmed commitments, and a compact event trail. On reconnect, the agent can say, “We were checking Thursday morning. I found one option at 10:30. Do you want to continue?” It should not replay a long internal transcript or resume a side effect without a fresh confirmation. Use idempotency keys for any tool that changes state. A reconnect must never turn one spoken confirmation into two appointments. For tools you cannot cancel safely, let them finish, mark the result superseded if needed, and prevent it from being delivered as current. The tests that matter simulate overlap, correction, tool delay, and reconnection — not only a clean question-and-answer exchange. Text-agent tests often compare a final answer with a reference. Voice systems need a timeline. A good test fixture includes audio events, intent hypotheses, tool latency, tool results, and the expected commitment state. You are evaluating the system’s behavior under overlap, not just its prose. Start with these scenarios: Measure more than end-to-end latency. Track time to stop speech after a meaningful interruption, time to the first useful acknowledgement, stale-result suppression rate, duplicate-commit rate, completion rate, and how often a human reviewer judges the agent as having talked past the caller. The Reddit discussion about voice evaluation gets this right: naturalness, latency, turn-taking, and task success are separate dimensions. A fast answer can still be an unusable conversation. Start with read-only tasks: order status, availability search, policy lookup, or guided triage. Run the voice model with your real tool contracts, but require a human or a conventional UI to make the final change. Listen to replays for interruption behavior, not just bad answers. Next, allow low-risk commits with explicit confirmation and idempotency. Only then consider higher-impact work, and keep a visible escape hatch: transfer to a person, send a text recap, or continue in a web interface. Voice is an excellent intent interface. It is not proof that a user understood a sensitive transaction. “Natural” should mean the caller can change their mind without the system losing control — not that the model sounds confident while it guesses. GPT-Live-1 reduces the awkwardness of the old chained voice stack. That is real progress. But the architecture that earns trust is still simple: let the model own the moment-to-moment conversation; let typed tasks own work; let a durable control layer own permissions and commitments. If you build that boundary first, your agent can interrupt gracefully, delegate without pretending, recover after a dropped call, and make fewer promises than it can prove. That is how a voice agent stops sounding like a demo and starts behaving like a product. GPT-Live-1 is OpenAI’s full-duplex voice model for real-time conversations. It can listen and speak simultaneously, handle conversational interruptions, and delegate deeper reasoning or tool work to a backend. It can reduce the amount of custom turn detection you need, but your product still needs policies for acknowledgements, background speech, confirmations, and when a spoken correction supersedes active work. Stop or duck playback, record the new intent, advance an intent version when the request changes, and prevent incompatible delayed results from being presented as current. Cancel the old job only when cancellation is safe and useful. Use a backend control layer for business tools. It should validate arguments, enforce authorization, assign idempotency keys, and require confirmation before a voice request produces a sensitive side effect. Track interruption stop time, first useful response time, task completion, stale-result suppression, duplicate commitments, tool errors, reconnect recovery, and human-rated conversational quality. One latency number cannot describe the experience. Keep lookup and commitment separate. Show or speak a precise summary, obtain an unambiguous confirmation, use idempotency for the final request, and provide a clear human handoff or non-voice fallback for exceptions. Sources and further reading: OpenAI’s GPT-Live-1 API announcement , the GPT-Live-1 model documentation , and the GPT-Live system card . GPT-Live-1 Tool Delegation: Keep Voice Agents Honest During Slow Work https://pub.towardsai.net/gpt-live-1-tool-delegation-keep-voice-agents-honest-during-slow-work-f3037b6f92ca was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.