Stop Snapshot-Testing Voice Conversations: Verify the Turn Protocol Instead

A developer proposes replacing snapshot-style assertions on voice-companion output with a deterministic test oracle for the conversation turn protocol, built in TypeScript with Vitest. The approach separates protocol verification—turn ownership, request IDs, and stale-event handling across speech recognition, LLM, and speech synthesis stages—from content evaluation, so that late model responses or delayed playback callbacks cannot be mistaken for product defects. The oracle is demonstrated against a Tencent RTC conversational AI pipeline.

A real-time voice companion can behave correctly while its test suite reports failure. The usual culprit is not that “the LLM knows better.” It is that the test encoded an assumption the product never promised: an exact sentence, a fixed callback order, or playback beginning within an arbitrary sleep interval. That creates an uncomfortable engineering tension. You do not want to dismiss failures as harmless AI variability, but you also do not want brittle tests blocking valid behavior. A better approach is to test the conversation protocol deterministically and evaluate the conversation content separately. For a Tencent RTC conversational AI application, that means verifying facts such as: We will build that test oracle in TypeScript, reproduce the dangerous races, and then map it onto the live voice pipeline. Consider this test: expect answer .toBe "Your meeting begins at 3 PM." ; The model returns: The meeting starts at 3 PM. The assertion failed, but that alone does not establish a product defect. The test mixed three different questions: Only the last question involves linguistic variability. The first two should remain deterministic. The human decision is therefore not “trust the model or trust the test.” It is: which product contract was this test supposed to enforce? A production voice companion is not one AI call. Keep these responsibilities separate: Microphone / RTC media ↓ Speech recognition ↓ Application turn coordinator ↓ LLM ↓ Speech synthesis ↓ RTC playback The application coordinator owns the current turn. Neither a late model response nor a delayed playback callback gets to decide that it is still relevant. Tencent RTC documents Conversational AI as a real-time voice interaction scenario that can connect users with multiple LLM providers. Its LLM configuration guidance also covers OpenAI-compatible models and request identifiers used for routing and observability: The event and effect names below belong to our application. They are not Tencent RTC SDK method names. mkdir voice-protocol-oracle cd voice-protocol-oracle npm init -y npm install --save-dev typescript vitest tsx @types/node npm pkg set type=module npm pkg set scripts.test="vitest run" mkdir src Create src/session.ts : export type Phase = | "idle" | "userSpeaking" | "transcribing" | "thinking" | "preparingAudio" | "speaking" | "recovery" | "offline"; export type State = { phase: Phase; turn: number; requestId?: string; }; export type Event = | { type: "USER STARTED" } | { type: "USER ENDED" } | { type: "TRANSCRIPT READY"; turn: number; requestId: string; text: string; } | { type: "TRANSCRIPT FAILED"; turn: number } | { type: "MODEL DONE"; turn: number; requestId: string; text: string; } | { type: "MODEL TIMEOUT"; turn: number; requestId: string } | { type: "PLAYBACK STARTED"; turn: number; requestId: string } | { type: "PLAYBACK ENDED"; turn: number; requestId: string } | { type: "PLAYBACK FAILED"; turn: number; requestId: string } | { type: "DISCONNECTED" }; export type Effect = | { type: "REQUEST TRANSCRIPT"; turn: number } | { type: "REQUEST MODEL"; turn: number; requestId: string; text: string; } | { type: "SPEAK"; turn: number; requestId: string; text: string; } | { type: "CANCEL AUDIO"; turn: number; requestId: string } | { type: "NOTICE"; message: string } | { type: "LOG STALE"; source: string; turn: number; requestId?: string }; export const initialState: State = { phase: "idle", turn: 0, }; function isCurrent state: State, turn: number, requestId?: string, : boolean { return state.turn === turn && requestId === undefined || state.requestId === requestId ; } function stale state: State, source: string, turn: number, requestId?: string, : State, Effect { return state, { type: "LOG STALE", source, turn, requestId } , ; } export function reduce state: State, event: Event, : State, Effect { switch event.type { case "USER STARTED": { if state.phase === "userSpeaking" return state, ; const effects: Effect = ; if state.phase === "preparingAudio" || state.phase === "speaking" && state.requestId { effects.push { type: "CANCEL AUDIO", turn: state.turn, requestId: state.requestId, } ; } return { phase: "userSpeaking", turn: state.turn + 1, }, effects, ; } case "USER ENDED": { if state.phase == "userSpeaking" return state, ; return { ...state, phase: "transcribing" }, { type: "REQUEST TRANSCRIPT", turn: state.turn } , ; } case "TRANSCRIPT READY": { if state.phase == "transcribing" || isCurrent state, event.turn { return stale state, "transcript", event.turn, event.requestId ; } if event.text.trim { return { ...state, phase: "recovery" }, { type: "NOTICE", message: "I did not catch that. Please try again." } , ; } return { ...state, phase: "thinking", requestId: event.requestId, }, { type: "REQUEST MODEL", turn: event.turn, requestId: event.requestId, text: event.text, }, , ; } case "TRANSCRIPT FAILED": { if state.phase == "transcribing" || isCurrent state, event.turn { return stale state, "transcript-error", event.turn ; } return { ...state, phase: "recovery" }, { type: "NOTICE", message: "Speech recognition is unavailable. Try again." } , ; } case "MODEL DONE": { if state.phase == "thinking" || isCurrent state, event.turn, event.requestId { return stale state, "model", event.turn, event.requestId ; } if event.text.trim { return { ...state, phase: "recovery" }, { type: "NOTICE", message: "No answer was produced. Please retry." } , ; } return { ...state, phase: "preparingAudio" }, { type: "SPEAK", turn: event.turn, requestId: event.requestId, text: event.text, }, , ; } case "MODEL TIMEOUT": { if state.phase == "thinking" || isCurrent state, event.turn, event.requestId { return stale state, "model-timeout", event.turn, event.requestId ; } return { ...state, phase: "recovery" }, { type: "NOTICE", message: "The response took too long. Please try again." } , ; } case "PLAYBACK STARTED": { if state.phase === "preparingAudio" && isCurrent state, event.turn, event.requestId { return { ...state, phase: "speaking" }, ; } // Playback may have crossed an interruption or cancellation boundary. // Stop it instead of accepting its late callback. return state, { type: "CANCEL AUDIO", turn: event.turn, requestId: event.requestId, }, { type: "LOG STALE", source: "playback-start", turn: event.turn, requestId: event.requestId, }, , ; } case "PLAYBACK ENDED": { if state.phase == "speaking" || isCurrent state, event.turn, event.requestId { return stale state, "playback-end", event.turn, event.requestId ; } return { phase: "idle", turn: state.turn }, ; } case "PLAYBACK FAILED": { if "preparingAudio", "speaking" .includes state.phase || isCurrent state, event.turn, event.requestId { return stale state, "playback-error", event.turn, event.requestId ; } return { ...state, phase: "recovery" }, { type: "NOTICE", message: "Audio playback failed. A text answer can be shown." } , ; } case "DISCONNECTED": { const effects: Effect = ; if state.phase === "preparingAudio" || state.phase === "speaking" && state.requestId { effects.push { type: "CANCEL AUDIO", turn: state.turn, requestId: state.requestId, } ; } effects.push { type: "NOTICE", message: "Connection lost. Reconnect to continue." } ; return { ...state, phase: "offline" }, effects ; } } } There are two important design choices here. First, turn changes as soon as new user speech begins. That immediately invalidates work from the previous turn. Second, a playback callback is not automatically trusted. If audio starts after its turn was interrupted, the coordinator issues another cancellation rather than moving back into speaking . Create src/session.test.ts : js import { describe, expect, it } from "vitest"; import { type Effect, type Event, initialState, reduce, } from "./session.js"; function scenario { let state = initialState; return { step event: Event : Effect { const next, effects = reduce state, event ; state = next; return effects; }, state { return state; }, }; } describe "voice turn protocol", = { it "rejects an LLM result from an interrupted turn", = { const s = scenario ; s.step { type: "USER STARTED" } ; s.step { type: "USER ENDED" } ; s.step { type: "TRANSCRIPT READY", turn: 1, requestId: "req-1", text: "Tell me a story", } ; // The user begins a replacement turn before req-1 completes. s.step { type: "USER STARTED" } ; s.step { type: "USER ENDED" } ; s.step { type: "TRANSCRIPT READY", turn: 2, requestId: "req-2", text: "Actually, stop", } ; const effects = s.step { type: "MODEL DONE", turn: 1, requestId: "req-1", text: "Once upon a time...", } ; expect effects.some effect = effect.type === "SPEAK" .toBe false ; expect s.state .toMatchObject { phase: "thinking", turn: 2, requestId: "req-2", } ; } ; it "cancels playback that starts after a barge-in", = { const s = scenario ; s.step { type: "USER STARTED" } ; s.step { type: "USER ENDED" } ; s.step { type: "TRANSCRIPT READY", turn: 1, requestId: "req-1", text: "Say hello", } ; s.step { type: "MODEL DONE", turn: 1, requestId: "req-1", text: "Hello ", } ; const interruption = s.step { type: "USER STARTED" } ; expect interruption .toContainEqual { type: "CANCEL AUDIO", turn: 1, requestId: "req-1", } ; const lateStart = s.step { type: "PLAYBACK STARTED", turn: 1, requestId: "req-1", } ; expect lateStart .toContainEqual { type: "CANCEL AUDIO", turn: 1, requestId: "req-1", } ; expect s.state .phase .toBe "userSpeaking" ; } ; it "does not speak a response that arrives after timeout", = { const s = scenario ; s.step { type: "USER STARTED" } ; s.step { type: "USER ENDED" } ; s.step { type: "TRANSCRIPT READY", turn: 1, requestId: "req-1", text: "What should we do today?", } ; s.step { type: "MODEL TIMEOUT", turn: 1, requestId: "req-1", } ; const lateResult = s.step { type: "MODEL DONE", turn: 1, requestId: "req-1", text: "Let us go for a walk.", } ; expect lateResult.some effect = effect.type === "SPEAK" .toBe false ; expect s.state .phase .toBe "recovery" ; } ; it.each "The meeting starts at 3 PM.", "Your meeting begins at 3 PM.", "accepts a protocol-valid wording: %s", answer = { const s = scenario ; s.step { type: "USER STARTED" } ; s.step { type: "USER ENDED" } ; s.step { type: "TRANSCRIPT READY", turn: 1, requestId: "req-1", text: "When is my meeting?", } ; const effects = s.step { type: "MODEL DONE", turn: 1, requestId: "req-1", text: answer, } ; expect effects .toContainEqual { type: "SPEAK", turn: 1, requestId: "req-1", text: answer, } ; } ; } ; Run the suite: npm test These tests do not claim that both meeting answers are factually correct. They establish that wording variation is not a turn-protocol failure. Factual correctness needs its own evidence and oracle. A practical test strategy separates concerns: | Layer | Suitable oracle | Example failure | |---|---|---| | Turn protocol | Deterministic state and effect assertions | Interrupted turn begins playback | | Safety and policy | Rules, schemas, allowlists, and human-approved cases | Companion gives disallowed advice | | Open-ended quality | Rubrics, reference facts, sampled human review | Answer is confusing despite being allowed | An LLM-based evaluator can assist the third layer, but it should not silently become the authority for the first two. If an evaluator says a response is acceptable while the response belongs to a stale turn, the application must still reject it. Likewise, when a test and an LLM disagree, inspect the contract before changing either one: This turns “the model was right” from a rhetorical conclusion into a reviewable engineering decision. Keep vendor and network operations behind ports: export interface VoicePorts { requestTranscript turn: number : Promise<void ; requestModel input: { turn: number; requestId: string; text: string; } : Promise<void ; speak input: { turn: number; requestId: string; text: string; } : Promise<void ; cancelAudio input: { turn: number; requestId: string; } : Promise<void ; showNotice message: string : void; } Your effect dispatcher translates application effects into the configured speech recognition, LLM, synthesis, and RTC operations. The corresponding callbacks are translated back into reducer events. For the LLM path, preserve the same application request identifier through the configured integration where supported. The Tencent RTC LLM configuration documentation https://trtc.io/document/68338 is the implementation reference for connecting an OpenAI-compatible model or an agent platform such as Dify or Coze. Do not let the adapter mutate session state directly. Its job is to report outcomes such as MODEL DONE , MODEL TIMEOUT , or PLAYBACK FAILED . The reducer decides whether those outcomes are still current. A single “response latency” number cannot tell you where a turn stalled. Record timestamps around boundaries such as: type TurnTiming = { turn: number; requestId: string; userEndedAt?: number; transcriptReadyAt?: number; modelRequestedAt?: number; modelCompletedAt?: number; playbackRequestedAt?: number; playbackStartedAt?: number; }; From those fields, derive separate measurements for: Choose timeout budgets from measurements in your own supported environments. Do not hide a slow phase behind automatic retries: a retry can produce a second, even later answer after the user has moved on. The user-visible fallback should also match the failed phase. “I didn’t catch that” is appropriate for transcription failure but misleading when the model timed out. Unit tests prove reducer behavior. Staging drills prove that adapters preserve it. Start one utterance, begin another, and release the first transcript afterward. Expected: The old transcript is logged as stale and never reaches the LLM. Delay model completion beyond the application deadline. Expected: Recovery remains visible. The late answer is not synthesized. Introduce a delay between requesting synthesis and receiving the playback-start callback. Interrupt during that gap. Expected: The late audio is cancelled immediately. State remains with the user’s new turn. Make speech output unavailable while retaining the model result. Expected: The interface exposes a clear fallback, such as displaying the text. It must not pretend the answer was spoken. Disconnect the RTC path while audio is active. Expected: Local playback is stopped, the session becomes visibly offline, and reconnection requires an explicit application policy. Deliver MODEL DONE or PLAYBACK ENDED twice. Expected: The duplicate cannot trigger a second spoken answer or advance a later turn. A social or companion experience should make its AI nature clear and provide visible controls for muting, interrupting, ending the session, and managing any retained transcript data. Moderation and privacy decisions belong to the application design; fluent output is not evidence that those decisions are safe. For character dialogue, AI companions, voice rooms, and related use cases, Tencent RTC also describes the broader Social Entertainment solution https://trtc.io/solutions/social-entertainment . Treat that page as scenario context, while using the product documentation for implementation details. Before shipping, verify that: The useful reframing is simple: a failed test is evidence of disagreement, not automatic proof that either the implementation or the model is correct. In a live voice application, the strongest tests are the ones that defend turn ownership and human control while leaving room for valid language variation. Relationship disclosure: This article was produced in connection with Tencent RTC. Official Tencent RTC documentation was used as the implementation reference. Which assertion in your current AI test suite is really a protocol guarantee—and which one is only an accidental preference for a particular wording?