A real-time voice companion can produce an answer quickly and still be exhausting to use.
The problem appears when a user s to think. Automatic endpoint detection interprets the silence as the end of a turn, sends an incomplete thought to the LLM, and starts speaking just as the user finds the next sentence.
For developers using AI as a rubber duck, tutor, or creative companion, this creates a surprisingly personal tension: the machine's fluency starts setting the pace. A feels like a mistake even though pausing is often where the useful reasoning happens.
The counterintuitive fix is not to make every stage faster. It is to give the user explicit control of the conversational floor.
In this tutorial, we'll build a TypeScript coordinator with two input modes:
The implementation will also reject stale LLM responses, stop output on interruption, preserve drafts across disconnects, and expose recoverable failures instead of silently resetting the conversation.
Tencent RTC provides the real-time voice layer for Conversational AI scenarios and can be connected to multiple LLM providers. Its
[Conversational AI overview]is the starting point for the media and AI integration. This tutorial keeps the application policy independent from provider-specific callbacks.
These acceptance cases are more useful than a vague requirement such as “support natural turn-taking”:
| Situation | Required behavior |
|---|---|
| User s in Automatic mode | A verified endpoint may submit the draft |
| User s in Hold mode | Keep collecting; do not call the LLM |
| User selects Ask AI | Commit exactly the visible draft |
| User speaks while the agent is answering | Stop playback and invalidate the old turn |
| An invalidated LLM response arrives later | Ignore it |
| The network disconnects | Cancel active work but retain the unsubmitted draft |
| The LLM fails | Show Retry and Edit controls |
| Speech synthesis fails | Keep the accepted text available for reading or replay |
The important distinction is that silence is an observation, not always consent to submit.
A production voice companion has several independently fallible parts:
microphone
│
▼
RTC/media transport
│
▼
speech recognition ──► application turn coordinator
│
▼
moderation/policy
│
▼
OpenAI or another LLM
│
▼
speech synthesis
│
▼
RTC playback
Do not represent this entire pipeline with one isTalking
boolean.
The RTC layer transports media. Speech recognition produces text and endpoint observations. The LLM generates a candidate answer. Speech synthesis produces output audio. Your application owns whether a draft may be submitted and whether a late result still belongs to the active turn.
Tencent RTC's Large Language Model configuration guide documents connecting OpenAI-compatible models and using request identification for routing and observability. Keep those identifiers aligned with your application turn IDs, but do not let provider configuration become your source of conversational state.
mkdir patient-voice-companion
cd patient-voice-companion
npm init -y
npm install openai
npm install --save-dev typescript tsx @types/node
npx tsc --init
mkdir src
npm pkg set scripts.test="tsx --test src/*.test.ts"
The coordinator below has no dependency on a specific RTC, recognition, or synthesis callback name. Integration adapters will translate provider events into this application-owned vocabulary.
Create src/turn.ts
:
export type InputMode = "automatic" | "hold";
export type Phase =
| "offline"
| "capturing"
| "thinking"
| "speaking"
| "recoverable";
export type PendingTurn = {
epoch: number;
prompt: string;
requestId: string;
attempt: number;
answer?: string;
};
export type TurnState = {
sessionId: string;
phase: Phase;
mode: InputMode;
epoch: number;
draft: string;
pending?: PendingTurn;
error?: "llm" | "synthesis";
};
export type Event =
| { type: "CONNECTED" }
| { type: "DISCONNECTED" }
| { type: "SET_MODE"; mode: InputMode }
| { type: "TRANSCRIPT_FINAL"; text: string }
| { type: "ENDPOINT_DETECTED" }
| { type: "ASK" }
| { type: "USER_SPEECH_STARTED" }
| { type: "LLM_OK"; epoch: number; answer: string }
| { type: "LLM_FAILED"; epoch: number }
| { type: "SPEECH_FINISHED"; epoch: number }
| { type: "SPEECH_FAILED"; epoch: number }
| { type: "RETRY" }
| { type: "DISCARD" };
export type Effect =
| {
kind: "requestLLM";
epoch: number;
prompt: string;
requestId: string;
}
| { kind: "cancelLLM"; epoch: number }
| { kind: "speak"; epoch: number; text: string }
| { kind: "stopPlayback" }
| { kind: "publishStatus"; message: string };
export type Transition = {
state: TurnState;
effects: Effect[];
};
export const initialState = (sessionId: string): TurnState => ({
sessionId,
phase: "offline",
mode: "automatic",
epoch: 0,
draft: ""
});
epoch
is the admission token for asynchronous work. An LLM or synthesis result is valid only when its epoch matches the current state.
A request ID is for routing and diagnostics; the epoch determines whether an answer may still be used. Those are related concerns, but they are not interchangeable.
Continue in src/turn.ts
:
function submit(state: TurnState): Transition {
const prompt = state.draft.trim();
if (!prompt) {
return {
state,
effects: [{ kind: "publishStatus", message: "Nothing to ask yet." }]
};
}
const epoch = state.epoch + 1;
const requestId = `${state.sessionId}:turn-${epoch}:attempt-1`;
return {
state: {
...state,
phase: "thinking",
epoch,
draft: "",
error: undefined,
pending: { epoch, prompt, requestId, attempt: 1 }
},
effects: [{ kind: "requestLLM", epoch, prompt, requestId }]
};
}
function interrupt(state: TurnState): Transition {
const effects: Effect[] = [];
if (state.phase === "thinking" && state.pending) {
effects.push({ kind: "cancelLLM", epoch: state.pending.epoch });
}
if (state.phase === "speaking") {
effects.push({ kind: "stopPlayback" });
}
return {
state: {
...state,
phase: "capturing",
epoch: state.epoch + 1,
pending: undefined,
error: undefined
},
effects
};
}
export function transition(state: TurnState, event: Event): Transition {
switch (event.type) {
case "CONNECTED":
return { state: { ...state, phase: "capturing" }, effects: [] };
case "DISCONNECTED": {
const stopped = interrupt(state);
return {
state: { ...stopped.state, phase: "offline" },
effects: stopped.effects
};
}
case "SET_MODE":
return {
state: { ...state, mode: event.mode },
effects: [
{
kind: "publishStatus",
message:
event.mode === "hold"
? "Holding the floor. Silence will not submit."
: "Automatic turn submission enabled."
}
]
};
case "TRANSCRIPT_FINAL":
return {
state: {
...state,
draft: [state.draft, event.text.trim()].filter(Boolean).join(" ")
},
effects: []
};
case "ENDPOINT_DETECTED":
return state.mode === "automatic"
? submit(state)
: { state, effects: [] };
case "ASK":
return submit(state);
case "USER_SPEECH_STARTED":
return state.phase === "thinking" || state.phase === "speaking"
? interrupt(state)
: { state, effects: [] };
case "LLM_OK":
if (event.epoch !== state.epoch || !state.pending) {
return { state, effects: [] };
}
return {
state: {
...state,
phase: "speaking",
pending: { ...state.pending, answer: event.answer }
},
effects: [{ kind: "speak", epoch: event.epoch, text: event.answer }]
};
case "LLM_FAILED":
if (event.epoch !== state.epoch || !state.pending) {
return { state, effects: [] };
}
return {
state: { ...state, phase: "recoverable", error: "llm" },
effects: [
{
kind: "publishStatus",
message: "The AI did not answer. Retry, edit, or discard this turn."
}
]
};
case "SPEECH_FINISHED":
if (event.epoch !== state.epoch) return { state, effects: [] };
return {
state: {
...state,
phase: "capturing",
pending: undefined,
error: undefined
},
effects: []
};
case "SPEECH_FAILED":
if (event.epoch !== state.epoch || !state.pending?.answer) {
return { state, effects: [] };
}
return {
state: { ...state, phase: "recoverable", error: "synthesis" },
effects: [
{
kind: "publishStatus",
message: "Audio playback failed. The text answer is still available."
}
]
};
case "RETRY": {
if (!state.pending || state.error !== "llm") {
return { state, effects: [] };
}
const attempt = state.pending.attempt + 1;
const requestId = `${state.sessionId}:turn-${state.epoch}:attempt-${attempt}`;
const pending = { ...state.pending, attempt, requestId };
return {
state: {
...state,
phase: "thinking",
pending,
error: undefined
},
effects: [
{
kind: "requestLLM",
epoch: state.epoch,
prompt: pending.prompt,
requestId
}
]
};
}
case "DISCARD":
return interrupt(state);
}
}
There are two deliberate asymmetries here:
That second rule matters because cancellation is not proof that remote computation stopped. The response may still arrive.
The effect runner connects the deterministic policy to real services:
export interface VoicePorts {
complete(input: {
prompt: string;
requestId: string;
}): Promise<string>;
cancelCompletion(epoch: number): Promise<void>;
speak(input: { text: string; epoch: number }): Promise<void>;
stopPlayback(): Promise<void>;
publishStatus(message: string): void;
}
export async function runEffect(
effect: Effect,
ports: VoicePorts,
dispatch: (event: Event) => void
): Promise<void> {
switch (effect.kind) {
case "requestLLM":
try {
const answer = await ports.complete({
prompt: effect.prompt,
requestId: effect.requestId
});
dispatch({ type: "LLM_OK", epoch: effect.epoch, answer });
} catch {
dispatch({ type: "LLM_FAILED", epoch: effect.epoch });
}
return;
case "cancelLLM":
await ports.cancelCompletion(effect.epoch).catch(() => undefined);
return;
case "speak":
try {
await ports.speak({ text: effect.text, epoch: effect.epoch });
dispatch({ type: "SPEECH_FINISHED", epoch: effect.epoch });
} catch {
dispatch({ type: "SPEECH_FAILED", epoch: effect.epoch });
}
return;
case "stopPlayback":
await ports.stopPlayback().catch(() => undefined);
return;
case "publishStatus":
ports.publishStatus(effect.message);
}
}
Notice that failed cancellation is tolerated. Safety comes from rejecting the stale result, not from assuming cancellation always wins the race.
If your selected OpenAI-compatible endpoint supports the Chat Completions shape, a server-side adapter can look like this:
import OpenAI from "openai";
import type { VoicePorts } from "./turn.js";
const client = new OpenAI({
apiKey: process.env.LLM_API_KEY,
baseURL: process.env.LLM_BASE_URL
});
export const llmPort: Pick<VoicePorts, "complete"> = {
async complete({ prompt, requestId }) {
console.info("llm.request", { requestId });
const result = await client.chat.completions.create({
model: process.env.LLM_MODEL!,
messages: [
{
role: "system",
content:
"You are a voice companion. Answer the submitted thought; do not pretend to have heard unsent audio."
},
{ role: "user", content: prompt }
]
});
const answer = result.choices[0]?.message?.content?.trim();
if (!answer) throw new Error("Empty model response");
console.info("llm.response", { requestId });
return answer;
}
};
Keep credentials on the server. Confirm the exact model and request format supported by the endpoint you configure; “OpenAI-compatible” should not be treated as a promise that every optional OpenAI feature behaves identically.
The requestId
above is application-owned correlation data. Carry the same identifier into the routing and observability configuration supported by your selected Tencent RTC Conversational AI setup rather than placing entire transcripts in logs.
Do not copy hypothetical callback names into your integration. Map the actual events from your Tencent RTC, recognition, and synthesis setup into the coordinator:
// Application-level mappings, not Tencent RTC API names.
recognitionAdapter.onFinalText(text =>
dispatch({ type: "TRANSCRIPT_FINAL", text })
);
recognitionAdapter.onEndpoint(() =>
dispatch({ type: "ENDPOINT_DETECTED" })
);
recognitionAdapter.onSpeechStarted(() =>
dispatch({ type: "USER_SPEECH_STARTED" })
);
ui.onHoldChanged(hold =>
dispatch({ type: "SET_MODE", mode: hold ? "hold" : "automatic" })
);
ui.onAsk(() => dispatch({ type: "ASK" }));
ui.onRetry(() => dispatch({ type: "RETRY" }));
ui.onDiscard(() => dispatch({ type: "DISCARD" }));
A visible control is preferable to relying only on phrases such as “let me think.” Recognition can mishear the phrase, and users should not have to remember a magic incantation to control whether their speech is submitted.
A voice command can be an additional convenience, but the current mode should remain visible and directly reversible.
Create src/turn.test.ts
:
import test from "node:test";
import assert from "node:assert/strict";
import { initialState, transition } from "./turn.js";
function connected() {
return transition(initialState("session-a"), { type: "CONNECTED" }).state;
}
test("silence does not submit while the user holds the floor", () => {
let state = connected();
state = transition(state, { type: "SET_MODE", mode: "hold" }).state;
state = transition(state, {
type: "TRANSCRIPT_FINAL",
text: "The race might be in the cache"
}).state;
const result = transition(state, { type: "ENDPOINT_DETECTED" });
assert.equal(result.state.phase, "capturing");
assert.equal(result.state.draft, "The race might be in the cache");
assert.deepEqual(result.effects, []);
});
test("Ask submits the exact accumulated draft", () => {
let state = connected();
state = transition(state, { type: "SET_MODE", mode: "hold" }).state;
state = transition(state, {
type: "TRANSCRIPT_FINAL",
text: "First part."
}).state;
state = transition(state, {
type: "TRANSCRIPT_FINAL",
text: "Second part."
}).state;
const result = transition(state, { type: "ASK" });
assert.equal(result.state.phase, "thinking");
assert.equal(result.state.pending?.prompt, "First part. Second part.");
assert.equal(result.effects[0]?.kind, "requestLLM");
});
test("a response from an interrupted turn is ignored", () => {
let state = connected();
state = transition(state, {
type: "TRANSCRIPT_FINAL",
text: "Explain this design"
}).state;
state = transition(state, { type: "ASK" }).state;
const oldEpoch = state.epoch;
state = transition(state, { type: "USER_SPEECH_STARTED" }).state;
const result = transition(state, {
type: "LLM_OK",
epoch: oldEpoch,
answer: "This answer is now stale"
});
assert.equal(result.state.phase, "capturing");
assert.equal(result.state.pending, undefined);
assert.deepEqual(result.effects, []);
});
test("disconnect retains an unsubmitted draft", () => {
let state = connected();
state = transition(state, {
type: "TRANSCRIPT_FINAL",
text: "Do not lose this thought"
}).state;
state = transition(state, { type: "DISCONNECTED" }).state;
assert.equal(state.phase, "offline");
assert.equal(state.draft, "Do not lose this thought");
});
Run the suite:
npm test
These tests do not prove microphone quality or network behavior. They verify the application invariant under callback orderings that are difficult to reproduce manually.
Explicit handoff is not universally better. It exchanges conversational speed for control.
| Experience | Better default | Reason |
|---|---|---|
| Short factual assistant commands | Automatic | The expected turn is brief and bounded |
| Coding rubber duck | Hold | Developers often inside one thought |
| Language pronunciation drill | Automatic | Fast repetition may be part of the exercise |
| Reflective or wellbeing companion | Hold | Silence may be intentional and sensitive |
| Hands-busy interaction | Automatic, with an accessible voice override | A screen control may be unavailable |
| Noisy social environment | Hold or push-to-talk | Endpoint observations may be unreliable |
The hidden cost of Hold mode is interaction overhead. The hidden cost of Automatic mode is accidental submission, interruption, and pressure to speak continuously.
Measure both rather than optimizing only model latency:
Avoid storing raw audio or complete transcripts merely to obtain these measurements. Use consent, retention limits, bounded identifiers, and stage-level events. Users should be told when audio is being processed by AI and should always have visible mute, stop, and exit controls.
They must be ignored for submission. Switching modes should change policy immediately; it should not depend on restarting recognition.
Both LLM_OK
and USER_SPEECH_STARTED
may be queued close together. Test both callback orders. Only the current epoch may reach audible playback.
The remote OpenAI-compatible request may continue. Keep the stale-result guard even if your provider exposes cancellation.
The simple accumulator in this tutorial will duplicate them. In production, normalize recognition results using stable segment identity supplied by the recognition layer, if available. Do not deduplicate only by text; a user can intentionally repeat a sentence.
The accepted prompt remains in pending
, so Retry can use the same text. The UI should also offer Edit, which copies that prompt back into a draft and invalidates the failed turn.
Do not ask the LLM again. Preserve and display pending.answer
, then offer text reading or a synthesis-only retry.
Keep the unsubmitted draft locally for a bounded period, mark the session offline, and require a successful reconnection before submission. Do not let reconnection automatically send speech the user never explicitly committed.
A companion that returns a polished explanation demonstrates language generation over the context it received. It does not demonstrate that it heard the unsent part of a thought, understood why the user d, or made a better engineering judgment than the user.
That distinction matters for developer confidence. Fast generated prose can make slower human reasoning feel obsolete, but the durable engineering work is elsewhere:
The practical next step is small: add a visible Hold control and measure accidental submissions before changing models or chasing lower latency. If interruptions fall but users dislike the extra action, offer both modes and remember the preference with clear consent.
Before shipping, verify that:
Tencent RTC also describes AI virtual companions and character dialogue within its Social Entertainment solution. The floor-control policy here is especially relevant to those longer, less command-like conversations: natural interaction does not mean removing control. Sometimes the most useful thing a voice companion can do is wait.
Where would you default to explicit handoff rather than automatic endpointing? More importantly, what evidence would persuade you to change that default?
Disclosure: I have a content relationship with Tencent RTC. Official Tencent RTC documentation was used as the implementation reference for this article.