Make the AI Wait: Build Explicit Floor Control for a Tencent RTC Voice Companion

A developer built a TypeScript coordinator for Tencent RTC voice companions that gives users explicit control over conversational turn-taking, addressing the problem of automatic endpoint detection interrupting users mid-thought. The implementation supports two input modes—automatic and hold—and rejects stale LLM responses, stops output on interruption, preserves drafts across disconnects, and exposes recoverable failures. The coordinator is provider-agnostic, integrating with Tencent RTC's real-time voice layer and multiple LLM providers.

A real-time voice companion can produce an answer quickly and still be exhausting to use. The problem appears when a user pauses to think. Automatic endpoint detection interprets the silence as the end of a turn, sends an incomplete thought to the LLM, and starts speaking just as the user finds the next sentence. For developers using AI as a rubber duck, tutor, or creative companion, this creates a surprisingly personal tension: the machine's fluency starts setting the pace. A pause feels like a mistake even though pausing is often where the useful reasoning happens. The counterintuitive fix is not to make every stage faster. It is to give the user explicit control of the conversational floor. In this tutorial, we'll build a TypeScript coordinator with two input modes: The implementation will also reject stale LLM responses, stop output on interruption, preserve drafts across disconnects, and expose recoverable failures instead of silently resetting the conversation. Tencent RTC provides the real-time voice layer for Conversational AI scenarios and can be connected to multiple LLM providers. Its Conversational AI overview is the starting point for the media and AI integration. This tutorial keeps the application policy independent from provider-specific callbacks. These acceptance cases are more useful than a vague requirement such as “support natural turn-taking”: | Situation | Required behavior | |---|---| | User pauses in Automatic mode | A verified endpoint may submit the draft | | User pauses in Hold mode | Keep collecting; do not call the LLM | | User selects Ask AI | Commit exactly the visible draft | | User speaks while the agent is answering | Stop playback and invalidate the old turn | | An invalidated LLM response arrives later | Ignore it | | The network disconnects | Cancel active work but retain the unsubmitted draft | | The LLM fails | Show Retry and Edit controls | | Speech synthesis fails | Keep the accepted text available for reading or replay | The important distinction is that silence is an observation, not always consent to submit . A production voice companion has several independently fallible parts: microphone │ ▼ RTC/media transport │ ▼ speech recognition ──► application turn coordinator │ ▼ moderation/policy │ ▼ OpenAI or another LLM │ ▼ speech synthesis │ ▼ RTC playback Do not represent this entire pipeline with one isTalking boolean. The RTC layer transports media. Speech recognition produces text and endpoint observations. The LLM generates a candidate answer. Speech synthesis produces output audio. Your application owns whether a draft may be submitted and whether a late result still belongs to the active turn. Tencent RTC's Large Language Model configuration guide https://trtc.io/document/68338 documents connecting OpenAI-compatible models and using request identification for routing and observability. Keep those identifiers aligned with your application turn IDs, but do not let provider configuration become your source of conversational state. mkdir patient-voice-companion cd patient-voice-companion npm init -y npm install openai npm install --save-dev typescript tsx @types/node npx tsc --init mkdir src npm pkg set scripts.test="tsx --test src/ .test.ts" The coordinator below has no dependency on a specific RTC, recognition, or synthesis callback name. Integration adapters will translate provider events into this application-owned vocabulary. Create src/turn.ts : export type InputMode = "automatic" | "hold"; export type Phase = | "offline" | "capturing" | "thinking" | "speaking" | "recoverable"; export type PendingTurn = { epoch: number; prompt: string; requestId: string; attempt: number; answer?: string; }; export type TurnState = { sessionId: string; phase: Phase; mode: InputMode; epoch: number; draft: string; pending?: PendingTurn; error?: "llm" | "synthesis"; }; export type Event = | { type: "CONNECTED" } | { type: "DISCONNECTED" } | { type: "SET MODE"; mode: InputMode } | { type: "TRANSCRIPT FINAL"; text: string } | { type: "ENDPOINT DETECTED" } | { type: "ASK" } | { type: "USER SPEECH STARTED" } | { type: "LLM OK"; epoch: number; answer: string } | { type: "LLM FAILED"; epoch: number } | { type: "SPEECH FINISHED"; epoch: number } | { type: "SPEECH FAILED"; epoch: number } | { type: "RETRY" } | { type: "DISCARD" }; export type Effect = | { kind: "requestLLM"; epoch: number; prompt: string; requestId: string; } | { kind: "cancelLLM"; epoch: number } | { kind: "speak"; epoch: number; text: string } | { kind: "stopPlayback" } | { kind: "publishStatus"; message: string }; export type Transition = { state: TurnState; effects: Effect ; }; export const initialState = sessionId: string : TurnState = { sessionId, phase: "offline", mode: "automatic", epoch: 0, draft: "" } ; epoch is the admission token for asynchronous work. An LLM or synthesis result is valid only when its epoch matches the current state. A request ID is for routing and diagnostics; the epoch determines whether an answer may still be used. Those are related concerns, but they are not interchangeable. Continue in src/turn.ts : js function submit state: TurnState : Transition { const prompt = state.draft.trim ; if prompt { return { state, effects: { kind: "publishStatus", message: "Nothing to ask yet." } }; } const epoch = state.epoch + 1; const requestId = ${state.sessionId}:turn-${epoch}:attempt-1 ; return { state: { ...state, phase: "thinking", epoch, draft: "", error: undefined, pending: { epoch, prompt, requestId, attempt: 1 } }, effects: { kind: "requestLLM", epoch, prompt, requestId } }; } function interrupt state: TurnState : Transition { const effects: Effect = ; if state.phase === "thinking" && state.pending { effects.push { kind: "cancelLLM", epoch: state.pending.epoch } ; } if state.phase === "speaking" { effects.push { kind: "stopPlayback" } ; } return { state: { ...state, phase: "capturing", epoch: state.epoch + 1, pending: undefined, error: undefined }, effects }; } export function transition state: TurnState, event: Event : Transition { switch event.type { case "CONNECTED": return { state: { ...state, phase: "capturing" }, effects: }; case "DISCONNECTED": { const stopped = interrupt state ; return { state: { ...stopped.state, phase: "offline" }, effects: stopped.effects }; } case "SET MODE": return { state: { ...state, mode: event.mode }, effects: { kind: "publishStatus", message: event.mode === "hold" ? "Holding the floor. Silence will not submit." : "Automatic turn submission enabled." } }; case "TRANSCRIPT FINAL": return { state: { ...state, draft: state.draft, event.text.trim .filter Boolean .join " " }, effects: }; case "ENDPOINT DETECTED": return state.mode === "automatic" ? submit state : { state, effects: }; case "ASK": return submit state ; case "USER SPEECH STARTED": return state.phase === "thinking" || state.phase === "speaking" ? interrupt state : { state, effects: }; case "LLM OK": if event.epoch == state.epoch || state.pending { return { state, effects: }; } return { state: { ...state, phase: "speaking", pending: { ...state.pending, answer: event.answer } }, effects: { kind: "speak", epoch: event.epoch, text: event.answer } }; case "LLM FAILED": if event.epoch == state.epoch || state.pending { return { state, effects: }; } return { state: { ...state, phase: "recoverable", error: "llm" }, effects: { kind: "publishStatus", message: "The AI did not answer. Retry, edit, or discard this turn." } }; case "SPEECH FINISHED": if event.epoch == state.epoch return { state, effects: }; return { state: { ...state, phase: "capturing", pending: undefined, error: undefined }, effects: }; case "SPEECH FAILED": if event.epoch == state.epoch || state.pending?.answer { return { state, effects: }; } return { state: { ...state, phase: "recoverable", error: "synthesis" }, effects: { kind: "publishStatus", message: "Audio playback failed. The text answer is still available." } }; case "RETRY": { if state.pending || state.error == "llm" { return { state, effects: }; } const attempt = state.pending.attempt + 1; const requestId = ${state.sessionId}:turn-${state.epoch}:attempt-${attempt} ; const pending = { ...state.pending, attempt, requestId }; return { state: { ...state, phase: "thinking", pending, error: undefined }, effects: { kind: "requestLLM", epoch: state.epoch, prompt: pending.prompt, requestId } }; } case "DISCARD": return interrupt state ; } } There are two deliberate asymmetries here: That second rule matters because cancellation is not proof that remote computation stopped. The response may still arrive. The effect runner connects the deterministic policy to real services: export interface VoicePorts { complete input: { prompt: string; requestId: string; } : Promise<string ; cancelCompletion epoch: number : Promise<void ; speak input: { text: string; epoch: number } : Promise<void ; stopPlayback : Promise<void ; publishStatus message: string : void; } export async function runEffect effect: Effect, ports: VoicePorts, dispatch: event: Event = void : Promise<void { switch effect.kind { case "requestLLM": try { const answer = await ports.complete { prompt: effect.prompt, requestId: effect.requestId } ; dispatch { type: "LLM OK", epoch: effect.epoch, answer } ; } catch { dispatch { type: "LLM FAILED", epoch: effect.epoch } ; } return; case "cancelLLM": await ports.cancelCompletion effect.epoch .catch = undefined ; return; case "speak": try { await ports.speak { text: effect.text, epoch: effect.epoch } ; dispatch { type: "SPEECH FINISHED", epoch: effect.epoch } ; } catch { dispatch { type: "SPEECH FAILED", epoch: effect.epoch } ; } return; case "stopPlayback": await ports.stopPlayback .catch = undefined ; return; case "publishStatus": ports.publishStatus effect.message ; } } Notice that failed cancellation is tolerated. Safety comes from rejecting the stale result, not from assuming cancellation always wins the race. If your selected OpenAI-compatible endpoint supports the Chat Completions shape, a server-side adapter can look like this: python import OpenAI from "openai"; import type { VoicePorts } from "./turn.js"; const client = new OpenAI { apiKey: process.env.LLM API KEY, baseURL: process.env.LLM BASE URL } ; export const llmPort: Pick<VoicePorts, "complete" = { async complete { prompt, requestId } { console.info "llm.request", { requestId } ; const result = await client.chat.completions.create { model: process.env.LLM MODEL , messages: { role: "system", content: "You are a voice companion. Answer the submitted thought; do not pretend to have heard unsent audio." }, { role: "user", content: prompt } } ; const answer = result.choices 0 ?.message?.content?.trim ; if answer throw new Error "Empty model response" ; console.info "llm.response", { requestId } ; return answer; } }; Keep credentials on the server. Confirm the exact model and request format supported by the endpoint you configure; “OpenAI-compatible” should not be treated as a promise that every optional OpenAI feature behaves identically. The requestId above is application-owned correlation data. Carry the same identifier into the routing and observability configuration supported by your selected Tencent RTC Conversational AI setup rather than placing entire transcripts in logs. Do not copy hypothetical callback names into your integration. Map the actual events from your Tencent RTC, recognition, and synthesis setup into the coordinator: // Application-level mappings, not Tencent RTC API names. recognitionAdapter.onFinalText text = dispatch { type: "TRANSCRIPT FINAL", text } ; recognitionAdapter.onEndpoint = dispatch { type: "ENDPOINT DETECTED" } ; recognitionAdapter.onSpeechStarted = dispatch { type: "USER SPEECH STARTED" } ; ui.onHoldChanged hold = dispatch { type: "SET MODE", mode: hold ? "hold" : "automatic" } ; ui.onAsk = dispatch { type: "ASK" } ; ui.onRetry = dispatch { type: "RETRY" } ; ui.onDiscard = dispatch { type: "DISCARD" } ; A visible control is preferable to relying only on phrases such as “let me think.” Recognition can mishear the phrase, and users should not have to remember a magic incantation to control whether their speech is submitted. A voice command can be an additional convenience, but the current mode should remain visible and directly reversible. Create src/turn.test.ts : python import test from "node:test"; import assert from "node:assert/strict"; import { initialState, transition } from "./turn.js"; function connected { return transition initialState "session-a" , { type: "CONNECTED" } .state; } test "silence does not submit while the user holds the floor", = { let state = connected ; state = transition state, { type: "SET MODE", mode: "hold" } .state; state = transition state, { type: "TRANSCRIPT FINAL", text: "The race might be in the cache" } .state; const result = transition state, { type: "ENDPOINT DETECTED" } ; assert.equal result.state.phase, "capturing" ; assert.equal result.state.draft, "The race might be in the cache" ; assert.deepEqual result.effects, ; } ; test "Ask submits the exact accumulated draft", = { let state = connected ; state = transition state, { type: "SET MODE", mode: "hold" } .state; state = transition state, { type: "TRANSCRIPT FINAL", text: "First part." } .state; state = transition state, { type: "TRANSCRIPT FINAL", text: "Second part." } .state; const result = transition state, { type: "ASK" } ; assert.equal result.state.phase, "thinking" ; assert.equal result.state.pending?.prompt, "First part. Second part." ; assert.equal result.effects 0 ?.kind, "requestLLM" ; } ; test "a response from an interrupted turn is ignored", = { let state = connected ; state = transition state, { type: "TRANSCRIPT FINAL", text: "Explain this design" } .state; state = transition state, { type: "ASK" } .state; const oldEpoch = state.epoch; state = transition state, { type: "USER SPEECH STARTED" } .state; const result = transition state, { type: "LLM OK", epoch: oldEpoch, answer: "This answer is now stale" } ; assert.equal result.state.phase, "capturing" ; assert.equal result.state.pending, undefined ; assert.deepEqual result.effects, ; } ; test "disconnect retains an unsubmitted draft", = { let state = connected ; state = transition state, { type: "TRANSCRIPT FINAL", text: "Do not lose this thought" } .state; state = transition state, { type: "DISCONNECTED" } .state; assert.equal state.phase, "offline" ; assert.equal state.draft, "Do not lose this thought" ; } ; Run the suite: npm test These tests do not prove microphone quality or network behavior. They verify the application invariant under callback orderings that are difficult to reproduce manually. Explicit handoff is not universally better. It exchanges conversational speed for control. | Experience | Better default | Reason | |---|---|---| | Short factual assistant commands | Automatic | The expected turn is brief and bounded | | Coding rubber duck | Hold | Developers often pause inside one thought | | Language pronunciation drill | Automatic | Fast repetition may be part of the exercise | | Reflective or wellbeing companion | Hold | Silence may be intentional and sensitive | | Hands-busy interaction | Automatic, with an accessible voice override | A screen control may be unavailable | | Noisy social environment | Hold or push-to-talk | Endpoint observations may be unreliable | The hidden cost of Hold mode is interaction overhead. The hidden cost of Automatic mode is accidental submission, interruption, and pressure to speak continuously. Measure both rather than optimizing only model latency: Avoid storing raw audio or complete transcripts merely to obtain these measurements. Use consent, retention limits, bounded identifiers, and stage-level events. Users should be told when audio is being processed by AI and should always have visible mute, stop, and exit controls. They must be ignored for submission. Switching modes should change policy immediately; it should not depend on restarting recognition. Both LLM OK and USER SPEECH STARTED may be queued close together. Test both callback orders. Only the current epoch may reach audible playback. The remote OpenAI-compatible request may continue. Keep the stale-result guard even if your provider exposes cancellation. The simple accumulator in this tutorial will duplicate them. In production, normalize recognition results using stable segment identity supplied by the recognition layer, if available. Do not deduplicate only by text; a user can intentionally repeat a sentence. The accepted prompt remains in pending , so Retry can use the same text. The UI should also offer Edit , which copies that prompt back into a draft and invalidates the failed turn. Do not ask the LLM again. Preserve and display pending.answer , then offer text reading or a synthesis-only retry. Keep the unsubmitted draft locally for a bounded period, mark the session offline, and require a successful reconnection before submission. Do not let reconnection automatically send speech the user never explicitly committed. A companion that returns a polished explanation demonstrates language generation over the context it received. It does not demonstrate that it heard the unsent part of a thought, understood why the user paused, or made a better engineering judgment than the user. That distinction matters for developer confidence. Fast generated prose can make slower human reasoning feel obsolete, but the durable engineering work is elsewhere: The practical next step is small: add a visible Hold control and measure accidental submissions before changing models or chasing lower latency. If interruptions fall but users dislike the extra action, offer both modes and remember the preference with clear consent. Before shipping, verify that: Tencent RTC also describes AI virtual companions and character dialogue within its Social Entertainment solution https://trtc.io/solutions/social-entertainment . The floor-control policy here is especially relevant to those longer, less command-like conversations: natural interaction does not mean removing control. Sometimes the most useful thing a voice companion can do is wait. Where would you default to explicit handoff rather than automatic endpointing? More importantly, what evidence would persuade you to change that default? Disclosure: I have a content relationship with Tencent RTC. Official Tencent RTC documentation was used as the implementation reference for this article.