{"slug": "stop-snapshot-testing-voice-conversations-verify-the-turn-protocol-instead", "title": "Stop Snapshot-Testing Voice Conversations: Verify the Turn Protocol Instead", "summary": "A developer proposes replacing snapshot-style assertions on voice-companion output with a deterministic test oracle for the conversation turn protocol, built in TypeScript with Vitest. The approach separates protocol verification—turn ownership, request IDs, and stale-event handling across speech recognition, LLM, and speech synthesis stages—from content evaluation, so that late model responses or delayed playback callbacks cannot be mistaken for product defects. The oracle is demonstrated against a Tencent RTC conversational AI pipeline.", "body_md": "A real-time voice companion can behave correctly while its test suite reports failure.\n\nThe usual culprit is not that “the LLM knows better.” It is that the test encoded an assumption the product never promised: an exact sentence, a fixed callback order, or playback beginning within an arbitrary sleep interval.\n\nThat creates an uncomfortable engineering tension. You do not want to dismiss failures as harmless AI variability, but you also do not want brittle tests blocking valid behavior.\n\nA better approach is to test the **conversation protocol** deterministically and evaluate the **conversation content** separately. For a Tencent RTC conversational AI application, that means verifying facts such as:\n\nWe will build that test oracle in TypeScript, reproduce the dangerous races, and then map it onto the live voice pipeline.\n\nConsider this test:\n\n```\nexpect(answer).toBe(\"Your meeting begins at 3 PM.\");\n```\n\nThe model returns:\n\n```\nThe meeting starts at 3 PM.\n```\n\nThe assertion failed, but that alone does not establish a product defect. The test mixed three different questions:\n\nOnly the last question involves linguistic variability. The first two should remain deterministic.\n\nThe human decision is therefore not “trust the model or trust the test.” It is: **which product contract was this test supposed to enforce?**\n\nA production voice companion is not one AI call. Keep these responsibilities separate:\n\n```\nMicrophone / RTC media\n        ↓\nSpeech recognition\n        ↓\nApplication turn coordinator\n        ↓\nLLM\n        ↓\nSpeech synthesis\n        ↓\nRTC playback\n```\n\nThe application coordinator owns the current turn. Neither a late model response nor a delayed playback callback gets to decide that it is still relevant.\n\nTencent RTC documents Conversational AI as a real-time voice interaction scenario that can connect users with multiple LLM providers. Its LLM configuration guidance also covers OpenAI-compatible models and request identifiers used for routing and observability:\n\nThe event and effect names below belong to our application. They are not Tencent RTC SDK method names.\n\n```\nmkdir voice-protocol-oracle\ncd voice-protocol-oracle\nnpm init -y\nnpm install --save-dev typescript vitest tsx @types/node\nnpm pkg set type=module\nnpm pkg set scripts.test=\"vitest run\"\nmkdir src\n```\n\nCreate `src/session.ts`:\n\n```\nexport type Phase =\n  | \"idle\"\n  | \"userSpeaking\"\n  | \"transcribing\"\n  | \"thinking\"\n  | \"preparingAudio\"\n  | \"speaking\"\n  | \"recovery\"\n  | \"offline\";\n\nexport type State = {\n  phase: Phase;\n  turn: number;\n  requestId?: string;\n};\n\nexport type Event =\n  | { type: \"USER_STARTED\" }\n  | { type: \"USER_ENDED\" }\n  | {\n      type: \"TRANSCRIPT_READY\";\n      turn: number;\n      requestId: string;\n      text: string;\n    }\n  | { type: \"TRANSCRIPT_FAILED\"; turn: number }\n  | {\n      type: \"MODEL_DONE\";\n      turn: number;\n      requestId: string;\n      text: string;\n    }\n  | { type: \"MODEL_TIMEOUT\"; turn: number; requestId: string }\n  | { type: \"PLAYBACK_STARTED\"; turn: number; requestId: string }\n  | { type: \"PLAYBACK_ENDED\"; turn: number; requestId: string }\n  | { type: \"PLAYBACK_FAILED\"; turn: number; requestId: string }\n  | { type: \"DISCONNECTED\" };\n\nexport type Effect =\n  | { type: \"REQUEST_TRANSCRIPT\"; turn: number }\n  | {\n      type: \"REQUEST_MODEL\";\n      turn: number;\n      requestId: string;\n      text: string;\n    }\n  | {\n      type: \"SPEAK\";\n      turn: number;\n      requestId: string;\n      text: string;\n    }\n  | { type: \"CANCEL_AUDIO\"; turn: number; requestId: string }\n  | { type: \"NOTICE\"; message: string }\n  | { type: \"LOG_STALE\"; source: string; turn: number; requestId?: string };\n\nexport const initialState: State = {\n  phase: \"idle\",\n  turn: 0,\n};\n\nfunction isCurrent(\n  state: State,\n  turn: number,\n  requestId?: string,\n): boolean {\n  return (\n    state.turn === turn &&\n    (requestId === undefined || state.requestId === requestId)\n  );\n}\n\nfunction stale(\n  state: State,\n  source: string,\n  turn: number,\n  requestId?: string,\n): [State, Effect[]] {\n  return [\n    state,\n    [{ type: \"LOG_STALE\", source, turn, requestId }],\n  ];\n}\n\nexport function reduce(\n  state: State,\n  event: Event,\n): [State, Effect[]] {\n  switch (event.type) {\n    case \"USER_STARTED\": {\n      if (state.phase === \"userSpeaking\") return [state, []];\n\n      const effects: Effect[] = [];\n\n      if (\n        (state.phase === \"preparingAudio\" || state.phase === \"speaking\") &&\n        state.requestId\n      ) {\n        effects.push({\n          type: \"CANCEL_AUDIO\",\n          turn: state.turn,\n          requestId: state.requestId,\n        });\n      }\n\n      return [\n        {\n          phase: \"userSpeaking\",\n          turn: state.turn + 1,\n        },\n        effects,\n      ];\n    }\n\n    case \"USER_ENDED\": {\n      if (state.phase !== \"userSpeaking\") return [state, []];\n\n      return [\n        { ...state, phase: \"transcribing\" },\n        [{ type: \"REQUEST_TRANSCRIPT\", turn: state.turn }],\n      ];\n    }\n\n    case \"TRANSCRIPT_READY\": {\n      if (\n        state.phase !== \"transcribing\" ||\n        !isCurrent(state, event.turn)\n      ) {\n        return stale(state, \"transcript\", event.turn, event.requestId);\n      }\n\n      if (!event.text.trim()) {\n        return [\n          { ...state, phase: \"recovery\" },\n          [{ type: \"NOTICE\", message: \"I did not catch that. Please try again.\" }],\n        ];\n      }\n\n      return [\n        {\n          ...state,\n          phase: \"thinking\",\n          requestId: event.requestId,\n        },\n        [\n          {\n            type: \"REQUEST_MODEL\",\n            turn: event.turn,\n            requestId: event.requestId,\n            text: event.text,\n          },\n        ],\n      ];\n    }\n\n    case \"TRANSCRIPT_FAILED\": {\n      if (state.phase !== \"transcribing\" || !isCurrent(state, event.turn)) {\n        return stale(state, \"transcript-error\", event.turn);\n      }\n\n      return [\n        { ...state, phase: \"recovery\" },\n        [{ type: \"NOTICE\", message: \"Speech recognition is unavailable. Try again.\" }],\n      ];\n    }\n\n    case \"MODEL_DONE\": {\n      if (\n        state.phase !== \"thinking\" ||\n        !isCurrent(state, event.turn, event.requestId)\n      ) {\n        return stale(state, \"model\", event.turn, event.requestId);\n      }\n\n      if (!event.text.trim()) {\n        return [\n          { ...state, phase: \"recovery\" },\n          [{ type: \"NOTICE\", message: \"No answer was produced. Please retry.\" }],\n        ];\n      }\n\n      return [\n        { ...state, phase: \"preparingAudio\" },\n        [\n          {\n            type: \"SPEAK\",\n            turn: event.turn,\n            requestId: event.requestId,\n            text: event.text,\n          },\n        ],\n      ];\n    }\n\n    case \"MODEL_TIMEOUT\": {\n      if (\n        state.phase !== \"thinking\" ||\n        !isCurrent(state, event.turn, event.requestId)\n      ) {\n        return stale(state, \"model-timeout\", event.turn, event.requestId);\n      }\n\n      return [\n        { ...state, phase: \"recovery\" },\n        [{ type: \"NOTICE\", message: \"The response took too long. Please try again.\" }],\n      ];\n    }\n\n    case \"PLAYBACK_STARTED\": {\n      if (\n        state.phase === \"preparingAudio\" &&\n        isCurrent(state, event.turn, event.requestId)\n      ) {\n        return [{ ...state, phase: \"speaking\" }, []];\n      }\n\n      // Playback may have crossed an interruption or cancellation boundary.\n      // Stop it instead of accepting its late callback.\n      return [\n        state,\n        [\n          {\n            type: \"CANCEL_AUDIO\",\n            turn: event.turn,\n            requestId: event.requestId,\n          },\n          {\n            type: \"LOG_STALE\",\n            source: \"playback-start\",\n            turn: event.turn,\n            requestId: event.requestId,\n          },\n        ],\n      ];\n    }\n\n    case \"PLAYBACK_ENDED\": {\n      if (\n        state.phase !== \"speaking\" ||\n        !isCurrent(state, event.turn, event.requestId)\n      ) {\n        return stale(state, \"playback-end\", event.turn, event.requestId);\n      }\n\n      return [{ phase: \"idle\", turn: state.turn }, []];\n    }\n\n    case \"PLAYBACK_FAILED\": {\n      if (\n        ![\"preparingAudio\", \"speaking\"].includes(state.phase) ||\n        !isCurrent(state, event.turn, event.requestId)\n      ) {\n        return stale(state, \"playback-error\", event.turn, event.requestId);\n      }\n\n      return [\n        { ...state, phase: \"recovery\" },\n        [{ type: \"NOTICE\", message: \"Audio playback failed. A text answer can be shown.\" }],\n      ];\n    }\n\n    case \"DISCONNECTED\": {\n      const effects: Effect[] = [];\n\n      if (\n        (state.phase === \"preparingAudio\" || state.phase === \"speaking\") &&\n        state.requestId\n      ) {\n        effects.push({\n          type: \"CANCEL_AUDIO\",\n          turn: state.turn,\n          requestId: state.requestId,\n        });\n      }\n\n      effects.push({ type: \"NOTICE\", message: \"Connection lost. Reconnect to continue.\" });\n      return [{ ...state, phase: \"offline\" }, effects];\n    }\n  }\n}\n```\n\nThere are two important design choices here.\n\nFirst, `turn` changes as soon as new user speech begins. That immediately invalidates work from the previous turn.\n\nSecond, a playback callback is not automatically trusted. If audio starts after its turn was interrupted, the coordinator issues another cancellation rather than moving back into `speaking`.\n\nCreate `src/session.test.ts`:\n\n``` js\nimport { describe, expect, it } from \"vitest\";\nimport {\n  type Effect,\n  type Event,\n  initialState,\n  reduce,\n} from \"./session.js\";\n\nfunction scenario() {\n  let state = initialState;\n\n  return {\n    step(event: Event): Effect[] {\n      const [next, effects] = reduce(state, event);\n      state = next;\n      return effects;\n    },\n    state() {\n      return state;\n    },\n  };\n}\n\ndescribe(\"voice turn protocol\", () => {\n  it(\"rejects an LLM result from an interrupted turn\", () => {\n    const s = scenario();\n\n    s.step({ type: \"USER_STARTED\" });\n    s.step({ type: \"USER_ENDED\" });\n    s.step({\n      type: \"TRANSCRIPT_READY\",\n      turn: 1,\n      requestId: \"req-1\",\n      text: \"Tell me a story\",\n    });\n\n    // The user begins a replacement turn before req-1 completes.\n    s.step({ type: \"USER_STARTED\" });\n    s.step({ type: \"USER_ENDED\" });\n    s.step({\n      type: \"TRANSCRIPT_READY\",\n      turn: 2,\n      requestId: \"req-2\",\n      text: \"Actually, stop\",\n    });\n\n    const effects = s.step({\n      type: \"MODEL_DONE\",\n      turn: 1,\n      requestId: \"req-1\",\n      text: \"Once upon a time...\",\n    });\n\n    expect(effects.some((effect) => effect.type === \"SPEAK\")).toBe(false);\n    expect(s.state()).toMatchObject({\n      phase: \"thinking\",\n      turn: 2,\n      requestId: \"req-2\",\n    });\n  });\n\n  it(\"cancels playback that starts after a barge-in\", () => {\n    const s = scenario();\n\n    s.step({ type: \"USER_STARTED\" });\n    s.step({ type: \"USER_ENDED\" });\n    s.step({\n      type: \"TRANSCRIPT_READY\",\n      turn: 1,\n      requestId: \"req-1\",\n      text: \"Say hello\",\n    });\n    s.step({\n      type: \"MODEL_DONE\",\n      turn: 1,\n      requestId: \"req-1\",\n      text: \"Hello!\",\n    });\n\n    const interruption = s.step({ type: \"USER_STARTED\" });\n    expect(interruption).toContainEqual({\n      type: \"CANCEL_AUDIO\",\n      turn: 1,\n      requestId: \"req-1\",\n    });\n\n    const lateStart = s.step({\n      type: \"PLAYBACK_STARTED\",\n      turn: 1,\n      requestId: \"req-1\",\n    });\n\n    expect(lateStart).toContainEqual({\n      type: \"CANCEL_AUDIO\",\n      turn: 1,\n      requestId: \"req-1\",\n    });\n    expect(s.state().phase).toBe(\"userSpeaking\");\n  });\n\n  it(\"does not speak a response that arrives after timeout\", () => {\n    const s = scenario();\n\n    s.step({ type: \"USER_STARTED\" });\n    s.step({ type: \"USER_ENDED\" });\n    s.step({\n      type: \"TRANSCRIPT_READY\",\n      turn: 1,\n      requestId: \"req-1\",\n      text: \"What should we do today?\",\n    });\n    s.step({\n      type: \"MODEL_TIMEOUT\",\n      turn: 1,\n      requestId: \"req-1\",\n    });\n\n    const lateResult = s.step({\n      type: \"MODEL_DONE\",\n      turn: 1,\n      requestId: \"req-1\",\n      text: \"Let us go for a walk.\",\n    });\n\n    expect(lateResult.some((effect) => effect.type === \"SPEAK\")).toBe(false);\n    expect(s.state().phase).toBe(\"recovery\");\n  });\n\n  it.each([\n    \"The meeting starts at 3 PM.\",\n    \"Your meeting begins at 3 PM.\",\n  ])(\"accepts a protocol-valid wording: %s\", (answer) => {\n    const s = scenario();\n\n    s.step({ type: \"USER_STARTED\" });\n    s.step({ type: \"USER_ENDED\" });\n    s.step({\n      type: \"TRANSCRIPT_READY\",\n      turn: 1,\n      requestId: \"req-1\",\n      text: \"When is my meeting?\",\n    });\n\n    const effects = s.step({\n      type: \"MODEL_DONE\",\n      turn: 1,\n      requestId: \"req-1\",\n      text: answer,\n    });\n\n    expect(effects).toContainEqual({\n      type: \"SPEAK\",\n      turn: 1,\n      requestId: \"req-1\",\n      text: answer,\n    });\n  });\n});\n```\n\nRun the suite:\n\n```\nnpm test\n```\n\nThese tests do not claim that both meeting answers are factually correct. They establish that wording variation is not a turn-protocol failure. Factual correctness needs its own evidence and oracle.\n\nA practical test strategy separates concerns:\n\n| Layer | Suitable oracle | Example failure | \n|---|---|---|\n| Turn protocol | Deterministic state and effect assertions | Interrupted turn begins playback | \n| Safety and policy | Rules, schemas, allowlists, and human-approved cases | Companion gives disallowed advice | \n| Open-ended quality | Rubrics, reference facts, sampled human review | Answer is confusing despite being allowed | \n\nAn LLM-based evaluator can assist the third layer, but it should not silently become the authority for the first two. If an evaluator says a response is acceptable while the response belongs to a stale turn, the application must still reject it.\n\nLikewise, when a test and an LLM disagree, inspect the contract before changing either one:\n\nThis turns “the model was right” from a rhetorical conclusion into a reviewable engineering decision.\n\nKeep vendor and network operations behind ports:\n\n```\nexport interface VoicePorts {\n  requestTranscript(turn: number): Promise<void>;\n\n  requestModel(input: {\n    turn: number;\n    requestId: string;\n    text: string;\n  }): Promise<void>;\n\n  speak(input: {\n    turn: number;\n    requestId: string;\n    text: string;\n  }): Promise<void>;\n\n  cancelAudio(input: {\n    turn: number;\n    requestId: string;\n  }): Promise<void>;\n\n  showNotice(message: string): void;\n}\n```\n\nYour effect dispatcher translates application effects into the configured speech recognition, LLM, synthesis, and RTC operations. The corresponding callbacks are translated back into reducer events.\n\nFor the LLM path, preserve the same application request identifier through the configured integration where supported. The Tencent RTC [LLM configuration documentation](https://trtc.io/document/68338) is the implementation reference for connecting an OpenAI-compatible model or an agent platform such as Dify or Coze.\n\nDo not let the adapter mutate session state directly. Its job is to report outcomes such as `MODEL_DONE`, `MODEL_TIMEOUT`, or `PLAYBACK_FAILED`. The reducer decides whether those outcomes are still current.\n\nA single “response latency” number cannot tell you where a turn stalled. Record timestamps around boundaries such as:\n\n```\ntype TurnTiming = {\n  turn: number;\n  requestId: string;\n  userEndedAt?: number;\n  transcriptReadyAt?: number;\n  modelRequestedAt?: number;\n  modelCompletedAt?: number;\n  playbackRequestedAt?: number;\n  playbackStartedAt?: number;\n};\n```\n\nFrom those fields, derive separate measurements for:\n\nChoose timeout budgets from measurements in your own supported environments. Do not hide a slow phase behind automatic retries: a retry can produce a second, even later answer after the user has moved on.\n\nThe user-visible fallback should also match the failed phase. “I didn’t catch that” is appropriate for transcription failure but misleading when the model timed out.\n\nUnit tests prove reducer behavior. Staging drills prove that adapters preserve it.\n\nStart one utterance, begin another, and release the first transcript afterward.\n\n**Expected:** The old transcript is logged as stale and never reaches the LLM.\n\nDelay model completion beyond the application deadline.\n\n**Expected:** Recovery remains visible. The late answer is not synthesized.\n\nIntroduce a delay between requesting synthesis and receiving the playback-start callback. Interrupt during that gap.\n\n**Expected:** The late audio is cancelled immediately. State remains with the user’s new turn.\n\nMake speech output unavailable while retaining the model result.\n\n**Expected:** The interface exposes a clear fallback, such as displaying the text. It must not pretend the answer was spoken.\n\nDisconnect the RTC path while audio is active.\n\n**Expected:** Local playback is stopped, the session becomes visibly offline, and reconnection requires an explicit application policy.\n\nDeliver `MODEL_DONE` or `PLAYBACK_ENDED` twice.\n\n**Expected:** The duplicate cannot trigger a second spoken answer or advance a later turn.\n\nA social or companion experience should make its AI nature clear and provide visible controls for muting, interrupting, ending the session, and managing any retained transcript data. Moderation and privacy decisions belong to the application design; fluent output is not evidence that those decisions are safe.\n\nFor character dialogue, AI companions, voice rooms, and related use cases, Tencent RTC also describes the broader [Social Entertainment solution](https://trtc.io/solutions/social-entertainment). Treat that page as scenario context, while using the product documentation for implementation details.\n\nBefore shipping, verify that:\n\nThe useful reframing is simple: a failed test is evidence of disagreement, not automatic proof that either the implementation or the model is correct. In a live voice application, the strongest tests are the ones that defend turn ownership and human control while leaving room for valid language variation.\n\n*Relationship disclosure: This article was produced in connection with Tencent RTC. Official Tencent RTC documentation was used as the implementation reference.*\n\nWhich assertion in your current AI test suite is really a protocol guarantee—and which one is only an accidental preference for a particular wording?", "url": "https://wpnews.pro/news/stop-snapshot-testing-voice-conversations-verify-the-turn-protocol-instead", "canonical_source": "https://dev.to/susiewang/stop-snapshot-testing-voice-conversations-verify-the-turn-protocol-instead-109j", "published_at": "2026-10-25 17:08:59+00:00", "updated_at": "2026-10-25 17:13:31.247403+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "large-language-models", "ai-products"], "entities": ["Tencent RTC", "TypeScript", "Vitest", "OpenAI"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/stop-snapshot-testing-voice-conversations-verify-the-turn-protocol-instead", "markdown": "https://wpnews.pro/news/stop-snapshot-testing-voice-conversations-verify-the-turn-protocol-instead.md", "text": "https://wpnews.pro/news/stop-snapshot-testing-voice-conversations-verify-the-turn-protocol-instead.txt", "jsonld": "https://wpnews.pro/news/stop-snapshot-testing-voice-conversations-verify-the-turn-protocol-instead.jsonld"}}