{"slug": "build-a-voice-assistant-with-the-openai-realtime-api-and-webrtc", "title": "Build a Voice Assistant with the OpenAI Realtime API and WebRTC", "summary": "A tutorial by Rachel Goldstein shows how to build a browser-based voice assistant using the OpenAI Realtime API and WebRTC, with a Node.js server that mints ephemeral tokens and a static page that streams audio both ways. The assistant uses the gpt-realtime-2.1 model, supports interruptible voice chat, and includes a function call to fetch the user's local time.", "body_md": "# Build a Voice Assistant with the OpenAI Realtime API and WebRTC\n\nWire a browser to gpt-realtime-2.1 over WebRTC for interruptible voice chat with function calling.\n\n[Rachel Goldstein](https://sourcefeed.dev/u/rachel_goldstein)\n\n## What you'll build\n\nA browser-based voice assistant you can talk to — and interrupt mid-sentence — built on the [OpenAI Realtime API](https://developers.openai.com/api/docs/guides/realtime) over [WebRTC](https://webrtc.org), with a function call the model invokes to fetch your actual local time. You'll end with a zero-dependency [Node.js](https://nodejs.org) server that mints ephemeral tokens and a static page that streams audio both ways.\n\n## Prerequisites\n\n- Node.js 20 or later (you need the built-in global\n`fetch`\n\n; tested with Node 22 LTS) - An OpenAI account with billing enabled and an API key exported as\n`OPENAI_API_KEY`\n\n— Realtime sessions bill per audio token, so expect a few cents for this tutorial - A current desktop browser (Chrome, Edge, Firefox, or Safari) and a working microphone\n- Verified July 2026 against the GA Realtime API and the\n`gpt-realtime-2.1`\n\nmodel\n\nNo npm packages required — the server uses only Node built-ins.\n\n## 1. Scaffold the project\n\n```\nmkdir voice-assistant && cd voice-assistant\nmkdir public\nprintf '{ \"name\": \"voice-assistant\", \"type\": \"module\" }\\n' > package.json\nexport OPENAI_API_KEY=\"sk-...\"   # your real key\n```\n\nThe `package.json`\n\nexists only to enable ES modules; there's nothing to install.\n\n## 2. Mint ephemeral tokens on the server\n\nYour API key must never reach the browser. Instead, the server calls `POST /v1/realtime/client_secrets`\n\nto create a short-lived client secret (an `ek_...`\n\nstring, 10-minute default TTL) that the browser uses to connect directly to OpenAI:\n\n```\nsequenceDiagram\n    participant B as Browser\n    participant S as Your Node server\n    participant O as OpenAI Realtime API\n    B->>S: GET /token\n    S->>O: POST /v1/realtime/client_secrets (API key)\n    O-->>S: ephemeral key (ek_...)\n    S-->>B: ephemeral key\n    B->>O: POST /v1/realtime/calls (SDP offer + ek_...)\n    O-->>B: SDP answer\n    B<<->>O: WebRTC: mic audio ⇄ model audio + \"oai-events\" data channel\n```\n\nCreate `server.js`\n\n. It serves the static files and exposes `/token`\n\n. The session config — model, voice, instructions, and the tool definition — lives here, baked into the client secret at creation time, so the browser can't tamper with it:\n\n``` js\nimport { createServer } from \"node:http\";\nimport { readFile } from \"node:fs/promises\";\n\nconst SESSION_CONFIG = {\n  session: {\n    type: \"realtime\",\n    model: \"gpt-realtime-2.1\",\n    instructions:\n      \"You are a concise, friendly voice assistant. \" +\n      \"When the user asks about the time, call get_current_time and \" +\n      \"answer using its result.\",\n    audio: { output: { voice: \"marin\" } },\n    tools: [\n      {\n        type: \"function\",\n        name: \"get_current_time\",\n        description: \"Get the user's current local time and timezone.\",\n        parameters: { type: \"object\", properties: {}, required: [] },\n      },\n    ],\n    tool_choice: \"auto\",\n  },\n};\n\nconst server = createServer(async (req, res) => {\n  if (req.url === \"/token\") {\n    const r = await fetch(\"https://api.openai.com/v1/realtime/client_secrets\", {\n      method: \"POST\",\n      headers: {\n        Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,\n        \"Content-Type\": \"application/json\",\n      },\n      body: JSON.stringify(SESSION_CONFIG),\n    });\n    res.writeHead(r.status, { \"Content-Type\": \"application/json\" });\n    res.end(JSON.stringify(await r.json()));\n    return;\n  }\n  const path = req.url === \"/\" ? \"/index.html\" : req.url;\n  try {\n    const file = await readFile(`./public${path}`);\n    const type = path.endsWith(\".js\") ? \"text/javascript\" : \"text/html\";\n    res.writeHead(200, { \"Content-Type\": type });\n    res.end(file);\n  } catch {\n    res.writeHead(404);\n    res.end(\"Not found\");\n  }\n});\n\nserver.listen(3000, () =>\n  console.log(\"Voice assistant running at http://localhost:3000\")\n);\n```\n\n## 3. Build the page\n\nCreate `public/index.html`\n\n. The Connect button matters beyond UX: starting from a click satisfies browser autoplay policy, so the model's audio is allowed to play.\n\n```\n<!DOCTYPE html>\n<html lang=\"en\">\n  <head>\n    <meta charset=\"utf-8\" />\n    <title>Voice Assistant</title>\n  </head>\n  <body>\n    <button id=\"connect\">Connect</button>\n    <p id=\"status\">Disconnected</p>\n    <pre id=\"transcript\"></pre>\n    <script type=\"module\" src=\"/app.js\"></script>\n  </body>\n</html>\n```\n\n## 4. Connect the browser over WebRTC\n\nCreate `public/app.js`\n\n. The flow: fetch an ephemeral key, build an `RTCPeerConnection`\n\nwith your mic as the outgoing track and an `<audio>`\n\nelement for the model's voice, open the `oai-events`\n\ndata channel for JSON events, then exchange SDP with `POST https://api.openai.com/v1/realtime/calls`\n\n.\n\n``` js\nconst connectBtn = document.getElementById(\"connect\");\nconst statusEl = document.getElementById(\"status\");\nconst transcriptEl = document.getElementById(\"transcript\");\n\nlet pc, dc;\n\nconnectBtn.addEventListener(\"click\", async () => {\n  connectBtn.disabled = true;\n  statusEl.textContent = \"Connecting…\";\n\n  const tokenResponse = await fetch(\"/token\");\n  const { value: EPHEMERAL_KEY } = await tokenResponse.json();\n\n  pc = new RTCPeerConnection();\n\n  const audioEl = document.createElement(\"audio\");\n  audioEl.autoplay = true;\n  pc.ontrack = (e) => (audioEl.srcObject = e.streams[0]);\n\n  const mic = await navigator.mediaDevices.getUserMedia({ audio: true });\n  pc.addTrack(mic.getTracks()[0]);\n\n  dc = pc.createDataChannel(\"oai-events\");\n  dc.addEventListener(\"open\", () => {\n    statusEl.textContent = \"Connected — start talking\";\n  });\n  dc.addEventListener(\"message\", (e) => handleServerEvent(JSON.parse(e.data)));\n\n  const offer = await pc.createOffer();\n  await pc.setLocalDescription(offer);\n\n  const sdpResponse = await fetch(\"https://api.openai.com/v1/realtime/calls\", {\n    method: \"POST\",\n    body: offer.sdp,\n    headers: {\n      Authorization: `Bearer ${EPHEMERAL_KEY}`,\n      \"Content-Type\": \"application/sdp\",\n    },\n  });\n  await pc.setRemoteDescription({ type: \"answer\", sdp: await sdpResponse.text() });\n});\n```\n\nThat's the entire transport. Turn-taking and barge-in need no extra code — the API's server-side voice activity detection is on by default, and over WebRTC it tracks playback for you, so speaking over the assistant cuts it off cleanly.\n\n## 5. Handle events and function calls\n\nAppend this to `public/app.js`\n\n. Every server event arrives as JSON on the data channel. Two matter here: transcript deltas (to render what the assistant is saying) and `response.done`\n\n, whose `response.output`\n\narray contains any `function_call`\n\nitems. For each call you run the function locally, send the result back as a `function_call_output`\n\nconversation item — echoing the `call_id`\n\n— then send `response.create`\n\nso the model speaks the answer:\n\n```\nfunction handleServerEvent(event) {\n  console.log(event.type);\n  switch (event.type) {\n    case \"response.output_audio_transcript.delta\":\n      transcriptEl.textContent += event.delta;\n      break;\n    case \"response.output_audio_transcript.done\":\n      transcriptEl.textContent += \"\\n\\n\";\n      break;\n    case \"response.done\":\n      for (const item of event.response.output ?? []) {\n        if (item.type === \"function_call\") runTool(item);\n      }\n      break;\n    case \"error\":\n      console.error(event.error);\n      break;\n  }\n}\n\nfunction runTool(item) {\n  let output = { error: `Unknown tool: ${item.name}` };\n  if (item.name === \"get_current_time\") {\n    output = {\n      time: new Date().toLocaleTimeString(),\n      timezone: Intl.DateTimeFormat().resolvedOptions().timeZone,\n    };\n  }\n  dc.send(\n    JSON.stringify({\n      type: \"conversation.item.create\",\n      item: {\n        type: \"function_call_output\",\n        call_id: item.call_id,\n        output: JSON.stringify(output),\n      },\n    })\n  );\n  dc.send(JSON.stringify({ type: \"response.create\" }));\n}\n```\n\n## Verify it works\n\nStart the server from the shell where you exported the key:\n\n```\nnode server.js\n```\n\nExpected terminal output:\n\n```\nVoice assistant running at http://localhost:3000\n```\n\nOpen `http://localhost:3000`\n\n, click **Connect**, and allow microphone access. The status line flips to \"Connected — start talking.\" Say: *\"Hey — what time is it right now?\"* You should hear a spoken reply quoting your actual local time and timezone, and the transcript appears in the `<pre>`\n\nelement. The browser console logs the event flow:\n\n```\nsession.created\ninput_audio_buffer.speech_started\ninput_audio_buffer.speech_stopped\nresponse.created\nresponse.output_audio_transcript.delta   (repeated)\nresponse.done                            (contains the function_call item)\nresponse.done                            (the spoken answer, after the tool result)\n```\n\nNow interrupt it: ask a question, and while it's answering, start talking. Playback stops immediately and it responds to your new input.\n\n## Troubleshooting\n\n** Incorrect API key provided: undefined** (401 from\n\n`/token`\n\n) — the server process can't see `OPENAI_API_KEY`\n\n, so it sent `Bearer undefined`\n\n. Export the variable in the same shell that runs `node server.js`\n\n, or prefix the command: `OPENAI_API_KEY=\"sk-...\" node server.js`\n\n.** TypeError: Cannot read properties of undefined (reading 'getUserMedia')** — you opened the page over an insecure origin (e.g.\n\n`http://192.168.1.20:3000`\n\nfrom another machine). `navigator.mediaDevices`\n\nonly exists in secure contexts; `http://localhost`\n\ncounts, LAN IPs don't. Use localhost or put the server behind HTTPS.** NotAllowedError: Permission denied** from\n\n`getUserMedia`\n\n— microphone access was blocked, now or previously. Click the camera/mic icon in the address bar, reset the permission for the site, and reload.**401 on the POST https://api.openai.com/v1/realtime/calls request** — the ephemeral key expired before the SDP exchange. Client secrets default to a 10-minute TTL, so this happens when a tab sits open between minting and connecting. The code above fetches a fresh token inside the click handler; keep that ordering if you refactor.\n\n## Next steps\n\n- Add real tools with arguments — the\n`parameters`\n\nfield is standard JSON Schema, so a`get_weather(city)`\n\nfunction is a few lines plus a fetch in`runTool`\n\n- Enable input transcription in the session's\n`audio.input`\n\nconfig to render the user's side of the conversation too - Read the\n[Realtime function calling guide](https://developers.openai.com/api/docs/guides/realtime-function-calling)for multi-call turns, and the[WebRTC guide](https://developers.openai.com/api/docs/guides/realtime-webrtc)for the unified SDP interface that keeps your server in the connection path - For production, look at the\n[Agents SDK voice agents quickstart](https://openai.github.io/openai-agents-js/guides/voice-agents/quickstart/), which wraps this transport with sessions, handoffs, and guardrails\n\n## Sources & further reading\n\n-\n[Realtime API with WebRTC](https://developers.openai.com/api/docs/guides/realtime-webrtc)— developers.openai.com -\n[Realtime API function calling](https://developers.openai.com/api/docs/guides/realtime-function-calling)— developers.openai.com -\n[Create client secret - API Reference](https://developers.openai.com/api/reference/resources/realtime/subresources/client_secrets/methods/create)— developers.openai.com -\n[Realtime conversations](https://developers.openai.com/api/docs/guides/realtime-conversations)— developers.openai.com -\n[Models - OpenAI API](https://developers.openai.com/api/docs/models)— developers.openai.com\n\n[Rachel Goldstein](https://sourcefeed.dev/u/rachel_goldstein)· Dev Tools Editor\n\nRachel has been embedded in the developer tooling ecosystem for nearly eight years, covering everything from IDE wars and package-manager drama to the quiet rise of AI-assisted coding. She has a soft spot for open-source maintainers and an unhealthy number of terminal emulators installed on a single laptop.\n\n## Discussion 0\n\nNo comments yet\n\nBe the first to weigh in.", "url": "https://wpnews.pro/news/build-a-voice-assistant-with-the-openai-realtime-api-and-webrtc", "canonical_source": "https://sourcefeed.dev/a/build-a-voice-assistant-with-the-openai-realtime-api-and-webrtc", "published_at": "2026-07-28 17:40:34+00:00", "updated_at": "2026-07-28 17:56:24.770749+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-tools", "developer-tools"], "entities": ["OpenAI", "OpenAI Realtime API", "WebRTC", "Node.js", "gpt-realtime-2.1", "Rachel Goldstein"], "alternates": {"html": "https://wpnews.pro/news/build-a-voice-assistant-with-the-openai-realtime-api-and-webrtc", "markdown": "https://wpnews.pro/news/build-a-voice-assistant-with-the-openai-realtime-api-and-webrtc.md", "text": "https://wpnews.pro/news/build-a-voice-assistant-with-the-openai-realtime-api-and-webrtc.txt", "jsonld": "https://wpnews.pro/news/build-a-voice-assistant-with-the-openai-realtime-api-and-webrtc.jsonld"}}