{"slug": "make-the-ai-wait-build-explicit-floor-control-for-a-tencent-rtc-voice-companion", "title": "Make the AI Wait: Build Explicit Floor Control for a Tencent RTC Voice Companion", "summary": "A developer built a TypeScript coordinator for Tencent RTC voice companions that gives users explicit control over conversational turn-taking, addressing the problem of automatic endpoint detection interrupting users mid-thought. The implementation supports two input modes—automatic and hold—and rejects stale LLM responses, stops output on interruption, preserves drafts across disconnects, and exposes recoverable failures. The coordinator is provider-agnostic, integrating with Tencent RTC's real-time voice layer and multiple LLM providers.", "body_md": "A real-time voice companion can produce an answer quickly and still be exhausting to use.\n\nThe problem appears when a user pauses to think. Automatic endpoint detection interprets the silence as the end of a turn, sends an incomplete thought to the LLM, and starts speaking just as the user finds the next sentence.\n\nFor developers using AI as a rubber duck, tutor, or creative companion, this creates a surprisingly personal tension: the machine's fluency starts setting the pace. A pause feels like a mistake even though pausing is often where the useful reasoning happens.\n\nThe counterintuitive fix is not to make every stage faster. It is to give the user explicit control of the conversational floor.\n\nIn this tutorial, we'll build a TypeScript coordinator with two input modes:\n\nThe implementation will also reject stale LLM responses, stop output on interruption, preserve drafts across disconnects, and expose recoverable failures instead of silently resetting the conversation.\n\nTencent RTC provides the real-time voice layer for Conversational AI scenarios and can be connected to multiple LLM providers. Its\n\n[Conversational AI overview]is the starting point for the media and AI integration. This tutorial keeps the application policy independent from provider-specific callbacks.\n\nThese acceptance cases are more useful than a vague requirement such as “support natural turn-taking”:\n\n| Situation | Required behavior |\n|---|---|\n| User pauses in Automatic mode | A verified endpoint may submit the draft |\n| User pauses in Hold mode | Keep collecting; do not call the LLM |\n| User selects Ask AI | Commit exactly the visible draft |\n| User speaks while the agent is answering | Stop playback and invalidate the old turn |\n| An invalidated LLM response arrives later | Ignore it |\n| The network disconnects | Cancel active work but retain the unsubmitted draft |\n| The LLM fails | Show Retry and Edit controls |\n| Speech synthesis fails | Keep the accepted text available for reading or replay |\n\nThe important distinction is that **silence is an observation, not always consent to submit**.\n\nA production voice companion has several independently fallible parts:\n\n```\nmicrophone\n    │\n    ▼\nRTC/media transport\n    │\n    ▼\nspeech recognition ──► application turn coordinator\n                              │\n                              ▼\n                       moderation/policy\n                              │\n                              ▼\n                         OpenAI or another LLM\n                              │\n                              ▼\n                       speech synthesis\n                              │\n                              ▼\n                         RTC playback\n```\n\nDo not represent this entire pipeline with one `isTalking`\n\nboolean.\n\nThe RTC layer transports media. Speech recognition produces text and endpoint observations. The LLM generates a candidate answer. Speech synthesis produces output audio. Your application owns whether a draft may be submitted and whether a late result still belongs to the active turn.\n\nTencent RTC's [Large Language Model configuration guide](https://trtc.io/document/68338) documents connecting OpenAI-compatible models and using request identification for routing and observability. Keep those identifiers aligned with your application turn IDs, but do not let provider configuration become your source of conversational state.\n\n```\nmkdir patient-voice-companion\ncd patient-voice-companion\nnpm init -y\nnpm install openai\nnpm install --save-dev typescript tsx @types/node\nnpx tsc --init\nmkdir src\nnpm pkg set scripts.test=\"tsx --test src/*.test.ts\"\n```\n\nThe coordinator below has no dependency on a specific RTC, recognition, or synthesis callback name. Integration adapters will translate provider events into this application-owned vocabulary.\n\nCreate `src/turn.ts`\n\n:\n\n```\nexport type InputMode = \"automatic\" | \"hold\";\nexport type Phase =\n  | \"offline\"\n  | \"capturing\"\n  | \"thinking\"\n  | \"speaking\"\n  | \"recoverable\";\n\nexport type PendingTurn = {\n  epoch: number;\n  prompt: string;\n  requestId: string;\n  attempt: number;\n  answer?: string;\n};\n\nexport type TurnState = {\n  sessionId: string;\n  phase: Phase;\n  mode: InputMode;\n  epoch: number;\n  draft: string;\n  pending?: PendingTurn;\n  error?: \"llm\" | \"synthesis\";\n};\n\nexport type Event =\n  | { type: \"CONNECTED\" }\n  | { type: \"DISCONNECTED\" }\n  | { type: \"SET_MODE\"; mode: InputMode }\n  | { type: \"TRANSCRIPT_FINAL\"; text: string }\n  | { type: \"ENDPOINT_DETECTED\" }\n  | { type: \"ASK\" }\n  | { type: \"USER_SPEECH_STARTED\" }\n  | { type: \"LLM_OK\"; epoch: number; answer: string }\n  | { type: \"LLM_FAILED\"; epoch: number }\n  | { type: \"SPEECH_FINISHED\"; epoch: number }\n  | { type: \"SPEECH_FAILED\"; epoch: number }\n  | { type: \"RETRY\" }\n  | { type: \"DISCARD\" };\n\nexport type Effect =\n  | {\n      kind: \"requestLLM\";\n      epoch: number;\n      prompt: string;\n      requestId: string;\n    }\n  | { kind: \"cancelLLM\"; epoch: number }\n  | { kind: \"speak\"; epoch: number; text: string }\n  | { kind: \"stopPlayback\" }\n  | { kind: \"publishStatus\"; message: string };\n\nexport type Transition = {\n  state: TurnState;\n  effects: Effect[];\n};\n\nexport const initialState = (sessionId: string): TurnState => ({\n  sessionId,\n  phase: \"offline\",\n  mode: \"automatic\",\n  epoch: 0,\n  draft: \"\"\n});\n```\n\n`epoch`\n\nis the admission token for asynchronous work. An LLM or synthesis result is valid only when its epoch matches the current state.\n\nA request ID is for routing and diagnostics; the epoch determines whether an answer may still be used. Those are related concerns, but they are not interchangeable.\n\nContinue in `src/turn.ts`\n\n:\n\n``` js\nfunction submit(state: TurnState): Transition {\n  const prompt = state.draft.trim();\n\n  if (!prompt) {\n    return {\n      state,\n      effects: [{ kind: \"publishStatus\", message: \"Nothing to ask yet.\" }]\n    };\n  }\n\n  const epoch = state.epoch + 1;\n  const requestId = `${state.sessionId}:turn-${epoch}:attempt-1`;\n\n  return {\n    state: {\n      ...state,\n      phase: \"thinking\",\n      epoch,\n      draft: \"\",\n      error: undefined,\n      pending: { epoch, prompt, requestId, attempt: 1 }\n    },\n    effects: [{ kind: \"requestLLM\", epoch, prompt, requestId }]\n  };\n}\n\nfunction interrupt(state: TurnState): Transition {\n  const effects: Effect[] = [];\n\n  if (state.phase === \"thinking\" && state.pending) {\n    effects.push({ kind: \"cancelLLM\", epoch: state.pending.epoch });\n  }\n\n  if (state.phase === \"speaking\") {\n    effects.push({ kind: \"stopPlayback\" });\n  }\n\n  return {\n    state: {\n      ...state,\n      phase: \"capturing\",\n      epoch: state.epoch + 1,\n      pending: undefined,\n      error: undefined\n    },\n    effects\n  };\n}\n\nexport function transition(state: TurnState, event: Event): Transition {\n  switch (event.type) {\n    case \"CONNECTED\":\n      return { state: { ...state, phase: \"capturing\" }, effects: [] };\n\n    case \"DISCONNECTED\": {\n      const stopped = interrupt(state);\n      return {\n        state: { ...stopped.state, phase: \"offline\" },\n        effects: stopped.effects\n      };\n    }\n\n    case \"SET_MODE\":\n      return {\n        state: { ...state, mode: event.mode },\n        effects: [\n          {\n            kind: \"publishStatus\",\n            message:\n              event.mode === \"hold\"\n                ? \"Holding the floor. Silence will not submit.\"\n                : \"Automatic turn submission enabled.\"\n          }\n        ]\n      };\n\n    case \"TRANSCRIPT_FINAL\":\n      return {\n        state: {\n          ...state,\n          draft: [state.draft, event.text.trim()].filter(Boolean).join(\" \")\n        },\n        effects: []\n      };\n\n    case \"ENDPOINT_DETECTED\":\n      return state.mode === \"automatic\"\n        ? submit(state)\n        : { state, effects: [] };\n\n    case \"ASK\":\n      return submit(state);\n\n    case \"USER_SPEECH_STARTED\":\n      return state.phase === \"thinking\" || state.phase === \"speaking\"\n        ? interrupt(state)\n        : { state, effects: [] };\n\n    case \"LLM_OK\":\n      if (event.epoch !== state.epoch || !state.pending) {\n        return { state, effects: [] };\n      }\n\n      return {\n        state: {\n          ...state,\n          phase: \"speaking\",\n          pending: { ...state.pending, answer: event.answer }\n        },\n        effects: [{ kind: \"speak\", epoch: event.epoch, text: event.answer }]\n      };\n\n    case \"LLM_FAILED\":\n      if (event.epoch !== state.epoch || !state.pending) {\n        return { state, effects: [] };\n      }\n\n      return {\n        state: { ...state, phase: \"recoverable\", error: \"llm\" },\n        effects: [\n          {\n            kind: \"publishStatus\",\n            message: \"The AI did not answer. Retry, edit, or discard this turn.\"\n          }\n        ]\n      };\n\n    case \"SPEECH_FINISHED\":\n      if (event.epoch !== state.epoch) return { state, effects: [] };\n      return {\n        state: {\n          ...state,\n          phase: \"capturing\",\n          pending: undefined,\n          error: undefined\n        },\n        effects: []\n      };\n\n    case \"SPEECH_FAILED\":\n      if (event.epoch !== state.epoch || !state.pending?.answer) {\n        return { state, effects: [] };\n      }\n\n      return {\n        state: { ...state, phase: \"recoverable\", error: \"synthesis\" },\n        effects: [\n          {\n            kind: \"publishStatus\",\n            message: \"Audio playback failed. The text answer is still available.\"\n          }\n        ]\n      };\n\n    case \"RETRY\": {\n      if (!state.pending || state.error !== \"llm\") {\n        return { state, effects: [] };\n      }\n\n      const attempt = state.pending.attempt + 1;\n      const requestId = `${state.sessionId}:turn-${state.epoch}:attempt-${attempt}`;\n      const pending = { ...state.pending, attempt, requestId };\n\n      return {\n        state: {\n          ...state,\n          phase: \"thinking\",\n          pending,\n          error: undefined\n        },\n        effects: [\n          {\n            kind: \"requestLLM\",\n            epoch: state.epoch,\n            prompt: pending.prompt,\n            requestId\n          }\n        ]\n      };\n    }\n\n    case \"DISCARD\":\n      return interrupt(state);\n  }\n}\n```\n\nThere are two deliberate asymmetries here:\n\nThat second rule matters because cancellation is not proof that remote computation stopped. The response may still arrive.\n\nThe effect runner connects the deterministic policy to real services:\n\n```\nexport interface VoicePorts {\n  complete(input: {\n    prompt: string;\n    requestId: string;\n  }): Promise<string>;\n\n  cancelCompletion(epoch: number): Promise<void>;\n  speak(input: { text: string; epoch: number }): Promise<void>;\n  stopPlayback(): Promise<void>;\n  publishStatus(message: string): void;\n}\n\nexport async function runEffect(\n  effect: Effect,\n  ports: VoicePorts,\n  dispatch: (event: Event) => void\n): Promise<void> {\n  switch (effect.kind) {\n    case \"requestLLM\":\n      try {\n        const answer = await ports.complete({\n          prompt: effect.prompt,\n          requestId: effect.requestId\n        });\n        dispatch({ type: \"LLM_OK\", epoch: effect.epoch, answer });\n      } catch {\n        dispatch({ type: \"LLM_FAILED\", epoch: effect.epoch });\n      }\n      return;\n\n    case \"cancelLLM\":\n      await ports.cancelCompletion(effect.epoch).catch(() => undefined);\n      return;\n\n    case \"speak\":\n      try {\n        await ports.speak({ text: effect.text, epoch: effect.epoch });\n        dispatch({ type: \"SPEECH_FINISHED\", epoch: effect.epoch });\n      } catch {\n        dispatch({ type: \"SPEECH_FAILED\", epoch: effect.epoch });\n      }\n      return;\n\n    case \"stopPlayback\":\n      await ports.stopPlayback().catch(() => undefined);\n      return;\n\n    case \"publishStatus\":\n      ports.publishStatus(effect.message);\n  }\n}\n```\n\nNotice that failed cancellation is tolerated. Safety comes from rejecting the stale result, not from assuming cancellation always wins the race.\n\nIf your selected OpenAI-compatible endpoint supports the Chat Completions shape, a server-side adapter can look like this:\n\n``` python\nimport OpenAI from \"openai\";\nimport type { VoicePorts } from \"./turn.js\";\n\nconst client = new OpenAI({\n  apiKey: process.env.LLM_API_KEY,\n  baseURL: process.env.LLM_BASE_URL\n});\n\nexport const llmPort: Pick<VoicePorts, \"complete\"> = {\n  async complete({ prompt, requestId }) {\n    console.info(\"llm.request\", { requestId });\n\n    const result = await client.chat.completions.create({\n      model: process.env.LLM_MODEL!,\n      messages: [\n        {\n          role: \"system\",\n          content:\n            \"You are a voice companion. Answer the submitted thought; do not pretend to have heard unsent audio.\"\n        },\n        { role: \"user\", content: prompt }\n      ]\n    });\n\n    const answer = result.choices[0]?.message?.content?.trim();\n    if (!answer) throw new Error(\"Empty model response\");\n\n    console.info(\"llm.response\", { requestId });\n    return answer;\n  }\n};\n```\n\nKeep credentials on the server. Confirm the exact model and request format supported by the endpoint you configure; “OpenAI-compatible” should not be treated as a promise that every optional OpenAI feature behaves identically.\n\nThe `requestId`\n\nabove is application-owned correlation data. Carry the same identifier into the routing and observability configuration supported by your selected Tencent RTC Conversational AI setup rather than placing entire transcripts in logs.\n\nDo not copy hypothetical callback names into your integration. Map the actual events from your Tencent RTC, recognition, and synthesis setup into the coordinator:\n\n```\n// Application-level mappings, not Tencent RTC API names.\nrecognitionAdapter.onFinalText(text =>\n  dispatch({ type: \"TRANSCRIPT_FINAL\", text })\n);\n\nrecognitionAdapter.onEndpoint(() =>\n  dispatch({ type: \"ENDPOINT_DETECTED\" })\n);\n\nrecognitionAdapter.onSpeechStarted(() =>\n  dispatch({ type: \"USER_SPEECH_STARTED\" })\n);\n\nui.onHoldChanged(hold =>\n  dispatch({ type: \"SET_MODE\", mode: hold ? \"hold\" : \"automatic\" })\n);\n\nui.onAsk(() => dispatch({ type: \"ASK\" }));\nui.onRetry(() => dispatch({ type: \"RETRY\" }));\nui.onDiscard(() => dispatch({ type: \"DISCARD\" }));\n```\n\nA visible control is preferable to relying only on phrases such as “let me think.” Recognition can mishear the phrase, and users should not have to remember a magic incantation to control whether their speech is submitted.\n\nA voice command can be an additional convenience, but the current mode should remain visible and directly reversible.\n\nCreate `src/turn.test.ts`\n\n:\n\n``` python\nimport test from \"node:test\";\nimport assert from \"node:assert/strict\";\nimport { initialState, transition } from \"./turn.js\";\n\nfunction connected() {\n  return transition(initialState(\"session-a\"), { type: \"CONNECTED\" }).state;\n}\n\ntest(\"silence does not submit while the user holds the floor\", () => {\n  let state = connected();\n  state = transition(state, { type: \"SET_MODE\", mode: \"hold\" }).state;\n  state = transition(state, {\n    type: \"TRANSCRIPT_FINAL\",\n    text: \"The race might be in the cache\"\n  }).state;\n\n  const result = transition(state, { type: \"ENDPOINT_DETECTED\" });\n\n  assert.equal(result.state.phase, \"capturing\");\n  assert.equal(result.state.draft, \"The race might be in the cache\");\n  assert.deepEqual(result.effects, []);\n});\n\ntest(\"Ask submits the exact accumulated draft\", () => {\n  let state = connected();\n  state = transition(state, { type: \"SET_MODE\", mode: \"hold\" }).state;\n  state = transition(state, {\n    type: \"TRANSCRIPT_FINAL\",\n    text: \"First part.\"\n  }).state;\n  state = transition(state, {\n    type: \"TRANSCRIPT_FINAL\",\n    text: \"Second part.\"\n  }).state;\n\n  const result = transition(state, { type: \"ASK\" });\n\n  assert.equal(result.state.phase, \"thinking\");\n  assert.equal(result.state.pending?.prompt, \"First part. Second part.\");\n  assert.equal(result.effects[0]?.kind, \"requestLLM\");\n});\n\ntest(\"a response from an interrupted turn is ignored\", () => {\n  let state = connected();\n  state = transition(state, {\n    type: \"TRANSCRIPT_FINAL\",\n    text: \"Explain this design\"\n  }).state;\n  state = transition(state, { type: \"ASK\" }).state;\n  const oldEpoch = state.epoch;\n\n  state = transition(state, { type: \"USER_SPEECH_STARTED\" }).state;\n  const result = transition(state, {\n    type: \"LLM_OK\",\n    epoch: oldEpoch,\n    answer: \"This answer is now stale\"\n  });\n\n  assert.equal(result.state.phase, \"capturing\");\n  assert.equal(result.state.pending, undefined);\n  assert.deepEqual(result.effects, []);\n});\n\ntest(\"disconnect retains an unsubmitted draft\", () => {\n  let state = connected();\n  state = transition(state, {\n    type: \"TRANSCRIPT_FINAL\",\n    text: \"Do not lose this thought\"\n  }).state;\n\n  state = transition(state, { type: \"DISCONNECTED\" }).state;\n\n  assert.equal(state.phase, \"offline\");\n  assert.equal(state.draft, \"Do not lose this thought\");\n});\n```\n\nRun the suite:\n\n```\nnpm test\n```\n\nThese tests do not prove microphone quality or network behavior. They verify the application invariant under callback orderings that are difficult to reproduce manually.\n\nExplicit handoff is not universally better. It exchanges conversational speed for control.\n\n| Experience | Better default | Reason |\n|---|---|---|\n| Short factual assistant commands | Automatic | The expected turn is brief and bounded |\n| Coding rubber duck | Hold | Developers often pause inside one thought |\n| Language pronunciation drill | Automatic | Fast repetition may be part of the exercise |\n| Reflective or wellbeing companion | Hold | Silence may be intentional and sensitive |\n| Hands-busy interaction | Automatic, with an accessible voice override | A screen control may be unavailable |\n| Noisy social environment | Hold or push-to-talk | Endpoint observations may be unreliable |\n\nThe hidden cost of Hold mode is interaction overhead. The hidden cost of Automatic mode is accidental submission, interruption, and pressure to speak continuously.\n\nMeasure both rather than optimizing only model latency:\n\nAvoid storing raw audio or complete transcripts merely to obtain these measurements. Use consent, retention limits, bounded identifiers, and stage-level events. Users should be told when audio is being processed by AI and should always have visible mute, stop, and exit controls.\n\nThey must be ignored for submission. Switching modes should change policy immediately; it should not depend on restarting recognition.\n\nBoth `LLM_OK`\n\nand `USER_SPEECH_STARTED`\n\nmay be queued close together. Test both callback orders. Only the current epoch may reach audible playback.\n\nThe remote OpenAI-compatible request may continue. Keep the stale-result guard even if your provider exposes cancellation.\n\nThe simple accumulator in this tutorial will duplicate them. In production, normalize recognition results using stable segment identity supplied by the recognition layer, if available. Do not deduplicate only by text; a user can intentionally repeat a sentence.\n\nThe accepted prompt remains in `pending`\n\n, so Retry can use the same text. The UI should also offer **Edit**, which copies that prompt back into a draft and invalidates the failed turn.\n\nDo not ask the LLM again. Preserve and display `pending.answer`\n\n, then offer text reading or a synthesis-only retry.\n\nKeep the unsubmitted draft locally for a bounded period, mark the session offline, and require a successful reconnection before submission. Do not let reconnection automatically send speech the user never explicitly committed.\n\nA companion that returns a polished explanation demonstrates language generation over the context it received. It does not demonstrate that it heard the unsent part of a thought, understood why the user paused, or made a better engineering judgment than the user.\n\nThat distinction matters for developer confidence. Fast generated prose can make slower human reasoning feel obsolete, but the durable engineering work is elsewhere:\n\nThe practical next step is small: add a visible Hold control and measure accidental submissions before changing models or chasing lower latency. If interruptions fall but users dislike the extra action, offer both modes and remember the preference with clear consent.\n\nBefore shipping, verify that:\n\nTencent RTC also describes AI virtual companions and character dialogue within its [Social Entertainment solution](https://trtc.io/solutions/social-entertainment). The floor-control policy here is especially relevant to those longer, less command-like conversations: natural interaction does not mean removing control. Sometimes the most useful thing a voice companion can do is wait.\n\nWhere would you default to explicit handoff rather than automatic endpointing? More importantly, what evidence would persuade you to change that default?\n\n*Disclosure: I have a content relationship with Tencent RTC. Official Tencent RTC documentation was used as the implementation reference for this article.*", "url": "https://wpnews.pro/news/make-the-ai-wait-build-explicit-floor-control-for-a-tencent-rtc-voice-companion", "canonical_source": "https://dev.to/susiewang/make-the-ai-wait-build-explicit-floor-control-for-a-tencent-rtc-voice-companion-467d", "published_at": "2026-08-19 04:16:29+00:00", "updated_at": "2026-08-19 04:42:29.852778+00:00", "lang": "en", "topics": ["developer-tools", "ai-products", "natural-language-processing"], "entities": ["Tencent RTC", "OpenAI", "TypeScript"], "alternates": {"html": "https://wpnews.pro/news/make-the-ai-wait-build-explicit-floor-control-for-a-tencent-rtc-voice-companion", "markdown": "https://wpnews.pro/news/make-the-ai-wait-build-explicit-floor-control-for-a-tencent-rtc-voice-companion.md", "text": "https://wpnews.pro/news/make-the-ai-wait-build-explicit-floor-control-for-a-tencent-rtc-voice-companion.txt", "jsonld": "https://wpnews.pro/news/make-the-ai-wait-build-explicit-floor-control-for-a-tencent-rtc-voice-companion.jsonld"}}