{"slug": "build-voice-agents-openai-realtime-api-guide", "title": "Build Voice Agents: OpenAI Realtime API Guide", "summary": "A developer guide explains how to build low-latency voice agents using OpenAI's Realtime API, which processes audio natively over a WebSocket connection, reducing network latency below 300 milliseconds. The guide covers connection setup, session configuration, and security best practices, such as proxying API keys through edge middleware.", "body_md": "Building low-latency audio pipelines with the OpenAI Realtime API lets developers launch human-like conversational voice agents in production. Traditionally, building a voice interface meant chaining three separate model layers: automatic speech recognition (ASR), a text-based LLM logic layer, and text-to-speech (TTS) synthesis. That multi-step pipeline introduced significant round-trip network delays, making natural conversation impossible. Native audio processing over a persistent WebSocket connection changes this, reducing network latency below 300 milliseconds. This guide explains how to establish connection states, stream raw audio buffers, and optimise session configurations.\n\n**API Security Warning:** Never expose your OpenAI API key directly inside client-side browser scripts. Always proxy the WebSocket connection through a secure edge middleware (such as a Cloudflare Worker) that appends authorisation headers before forwarding packets to OpenAI.\n\n**Key Takeaways:**\n\n**WebSocket Connection:** Connect directly to OpenAI's realtime WebSocket gateway using edge proxies.\n**Native Modalities:** Specify both `text` and `audio` in your initial session update config payload.\n**Audio Format:** Stream user speech as base64-encoded mono PCM16 chunks at 24kHz.\n**Speech Interruption:** Monitor server speech-started signals to halt client playback instantly.\n\nTraditional voice stacks treat both speech recognition and synthesis models as external wrappers around a central text model. Native audio models remove that overhead by handling speech directly.\n\nWith OpenAI's realtime models, the network processes audio inputs and outputs natively. The model receives raw audio waveforms, parses tone, inflection, and content, and generates natural voice output directly. As a result, the pipeline eliminates ASR transcription errors and TTS synthesis bottlenecks.\n\nManaging this persistent connection relies on WebSockets. The connection remains open throughout the call, allowing the agent to interrupt its output if it detects user speech, so the experience matches a real telephone call.\n\n{{< cta-button url=\"/en/ai-integration-services/\" text=\"Get AI Integration Services\" >}}\n\nBefore writing a line of code, make sure your environment is ready. This build assumes you are comfortable with asynchronous JavaScript and have the following in place:\n\n`npm create cloudflare@latest`.` ws` package for a Node.js gateway, or the native `WebSocket` global available inside Workers.`getUserMedia` plus an `AudioWorklet` for resampling).\nBudget a little time for account setup too: Realtime access and billing must be enabled on your organisation before the gateway will accept a session.\n\nTo start, you open a connection to the OpenAI Realtime gateway, specifying the realtime model in your headers.\n\nThe JavaScript code below demonstrates how to initialise the connection, configure session modalities, and handle streaming input and output buffers:\n\n``` python\nimport WebSocket from \"ws\";\n\nexport async function startVoiceAgent(env) {\n  // Connect to the OpenAI Realtime WebSocket gateway\n  const url = \"wss://api.openai.com/v1/realtime?model=gpt-realtime\";\n  const ws = new WebSocket(url, {\n    headers: {\n      \"Authorization\": `Bearer ${env.OPENAI_API_KEY}`,\n      \"OpenAI-Beta\": \"realtime=v1\"\n    }\n  });\n\n  ws.on(\"open\", () => {\n    console.log(\"WebSocket connection established with OpenAI Realtime API\");\n\n    // Configure session modalities and voice parameters\n    const sessionConfig = {\n      type: \"session.update\",\n      session: {\n        modalities: [\"text\", \"audio\"],\n        instructions: \"You are a helpful customer service assistant for Mecanik.\",\n        voice: \"alloy\",\n        input_audio_format: \"pcm16\",\n        output_audio_format: \"pcm16\",\n        temperature: 0.7\n      }\n    };\n    ws.send(JSON.stringify(sessionConfig));\n  });\n\n  ws.on(\"message\", (data) => {\n    const event = JSON.parse(data);\n\n    // Handle incoming audio content from the server\n    if (event.type === \"response.audio.delta\") {\n      const audioBuffer = Buffer.from(event.delta, \"base64\");\n      // Output buffer to client audio player\n      playAudioChunk(audioBuffer);\n    }\n  });\n}\n```\n\nWhen building this handler, ensure your API key remains hidden from the client browser. Specifically, establish an edge middleware on your server to handle authorisation before proxying the WebSocket connection. To learn about edge API configurations, read our guide on [building a serverless API with Cloudflare Workers](https://mecanik.dev/en/posts/building-a-serverless-api-with-cloudflare-workers/).\n\nThe snippet above connects from a trusted server, but it never showed the piece that keeps your key safe: the proxy itself. On Cloudflare Workers you cannot attach custom headers to the `new WebSocket()` constructor, so you open the upstream connection with `fetch` and an `Upgrade` header instead. The Worker accepts the browser's socket, dials OpenAI with your secret key attached, then pipes frames between the two.\n\n```\nexport default {\n  async fetch(request, env) {\n    if (request.headers.get(\"Upgrade\") !== \"websocket\") {\n      return new Response(\"Expected a WebSocket upgrade\", { status: 426 });\n    }\n\n    // 1. Accept the browser <-> Worker socket\n    const [client, server] = Object.values(new WebSocketPair());\n    server.accept();\n\n    // 2. Open the Worker <-> OpenAI socket with the secret key attached\n    const upstreamResponse = await fetch(\n      \"https://api.openai.com/v1/realtime?model=gpt-realtime\",\n      {\n        headers: {\n          Upgrade: \"websocket\",\n          Authorization: `Bearer ${env.OPENAI_API_KEY}`,\n          \"OpenAI-Beta\": \"realtime=v1\"\n        }\n      }\n    );\n\n    const upstream = upstreamResponse.webSocket;\n    if (!upstream) {\n      return new Response(\"Upstream refused the upgrade\", { status: 502 });\n    }\n    upstream.accept();\n\n    // 3. Pipe frames in both directions\n    server.addEventListener(\"message\", (e) => upstream.send(e.data));\n    upstream.addEventListener(\"message\", (e) => server.send(e.data));\n\n    const close = () => { try { server.close(); upstream.close(); } catch {} };\n    server.addEventListener(\"close\", close);\n    upstream.addEventListener(\"close\", close);\n\n    return new Response(null, { status: 101, webSocket: client });\n  }\n};\n```\n\nStore the key as an encrypted secret with `wrangler secret put OPENAI_API_KEY` rather than in `wrangler.toml`, so it never lands in your repository. The browser now connects to `wss://your-worker.workers.dev` and never sees the credential. Cloudflare documents this bidirectional pattern in its [Workers WebSockets reference](https://developers.cloudflare.com/workers/runtime-apis/websockets/).\n\nOnce the session update is accepted, your client must capture microphone input, compress it to 24kHz mono PCM16 data, and stream it as base64 segments. Because the connection remains persistent, handling network dropouts is critical. A local chunk buffer ensures that brief cellular connection dropouts do not result in audio packet loss or jittery agent responses: the client retains the buffer and replays it immediately upon reconnect.\n\n```\n// Example of streaming user microphone data\nfunction streamMicrophoneChunk(ws, base64AudioChunk) {\n  const audioEvent = {\n    type: \"input_audio_buffer.append\",\n    audio: base64AudioChunk\n  };\n  ws.send(JSON.stringify(audioEvent));\n}\n```\n\nWhenever the user stops speaking, the server processes the accumulated audio buffer automatically and triggers a model response. For a complete API event list, review the [OpenAI Realtime Guide](https://platform.openai.com/docs/guides/realtime).\n\nAdditionally, implement echo cancellation in your frontend player. If the microphone picks up the speaker output, the agent will interpret its own voice as a user interruption, causing the session loop to fail. To learn more about frontend optimisation, check our guide on [WordPress vs custom web development](https://mecanik.dev/en/posts/wordpress-vs-custom-web-development-what-uk-businesses-need-to-know/).\n\nBy default you have to tell the model when a turn ends. Enabling server-side voice activity detection (VAD) hands that job to OpenAI: the gateway watches the incoming buffer, decides when the user has stopped speaking, and triggers a response automatically. Add a `turn_detection` block to the session update you send during the handshake.\n\n``` js\nconst sessionConfig = {\n  type: \"session.update\",\n  session: {\n    modalities: [\"text\", \"audio\"],\n    voice: \"alloy\",\n    input_audio_format: \"pcm16\",\n    output_audio_format: \"pcm16\",\n    turn_detection: {\n      type: \"server_vad\",\n      threshold: 0.5,\n      prefix_padding_ms: 300,\n      silence_duration_ms: 500\n    }\n  }\n};\n```\n\nWith VAD active, the server emits `input_audio_buffer.speech_started` the instant it hears the user talk over the agent. Treat that event as a hard stop: flush every queued audio chunk in your player, otherwise the previous response keeps playing while the new one arrives.\n\n``` js\nlet playbackQueue = [];\n\nws.on(\"message\", (raw) => {\n  const event = JSON.parse(raw);\n  if (event.type === \"input_audio_buffer.speech_started\") {\n    // User is interrupting: drop everything still buffered\n    playbackQueue = [];\n    stopSpeaker();\n  }\n});\n```\n\nRaising `silence_duration_ms` makes the agent wait longer before replying, which suits slow or hesitant speakers; lowering it makes the exchange feel snappier but risks cutting people off mid-sentence.\n\nTo deploy your voice agent middleware, start by configuring a secure WebSocket routing gateway on serverless edge nodes to protect your OpenAI credentials. This prevents scrapers from discovering your keys.\n\nNext, draft your system prompts and rules carefully. Set the tone, vocabulary, and response instructions inside the initial payload update session event. Consequently, this establishes a clear scope for conversational flows.\n\nThen implement robust speaker interruption logic. You should listen for the `input_audio_buffer.speech_started` event and halt audio playback in your frontend player immediately. Finally, optimise global latency by deploying your Workers in close regional proximity to your users to reduce routing hops. To explore edge hosting options, read our [Cloudflare Workers AI tutorial](https://mecanik.dev/en/posts/cloudflare-workers-ai-tutorial/).\n\nMost first-run failures come from a handful of predictable mistakes. The table below maps the symptom you will see to its usual cause and fix.\n\n| Symptom | Likely cause | Fix | \n|---|---|---|\n| Connection closes with a `401` immediately | Missing or malformed `Authorization` header | Confirm the proxy attaches `Bearer <key>` and that the key has Realtime access | \n| Agent hears only silence or garbled speech | Audio sent at the wrong sample rate or bit depth | Resample to 24kHz mono PCM16 before base64 encoding | \n| Model never responds after the user speaks | `turn_detection` is disabled and no manual commit is sent | Enable `server_vad` , or send`input_audio_buffer.commit` then`response.create` | \n| Agent talks over the user | The `speech_started` handler does not clear the queue | Empty the playback buffer and stop the speaker on that event | \n| Responses cut off mid-sentence | `max_response_output_tokens` set too low | Raise the limit or leave it as `inf` | \n\nTwo subtler issues deserve attention. A `rate_limits.updated` event arriving with a low remaining balance is your early warning that concurrent sessions are about to be throttled; log it and back off rather than hammering reconnects. And if the agent occasionally answers its own last sentence, the microphone is capturing speaker output, so tighten echo cancellation or move testers onto headsets.\n\nLocal testing is awkward because you need real audio flowing in both directions. The fastest loop is to run the Worker with `wrangler dev`, point a small browser page at it, and watch the event stream in the console before you worry about audio quality. Log every inbound event type during development; once the sequence of `session.updated`, `speech_started`, `response.audio.delta`, and `response.done` looks correct, you know the plumbing is sound.\n\nBefore you ship, weigh a few production realities:\n\nA short pilot with real callers will surface accent, noise, and interruption edge cases that no scripted test covers, so run one before any wide launch.\n\n**Related reading:** [Building AI Agents with Cloudflare Workers and LangChain](https://mecanik.dev/en/posts/cloudflare-workers-ai-agent/), [AI Integration Cost: 2026 Enterprise Budgeting Guide](https://mecanik.dev/en/posts/ai-integration-cost-enterprise-budgeting-guide/), [Claude Fable 5 Hybrid Reasoning: Thinking vs. Speed Modes](https://mecanik.dev/en/posts/claude-fable-5-hybrid-reasoning-api/) and [Claude Opus 4.8 vs. OpenAI GPT-5: Which API is Best?](https://mecanik.dev/en/posts/claude-opus-4-8-vs-gpt-5-api/).\n\n**What is the OpenAI Realtime API?**\n\nThe OpenAI Realtime API is a WebSocket interface that allows developers to stream raw audio in and out of the model, bypassing separate ASR/TTS modules. By processing audio natively, the model preserves emotional tone, accent inflections, and speech nuances, achieving latency times below 300 milliseconds.\n\n**How do I handle voice interruptions?**\n\nListen for the `input_audio_buffer.speech_started` server event. When received, clear your frontend audio buffers and halt speaker playback immediately. Consequently, this creates a natural conversational flow, allowing the AI agent to stop talking instantly when the user begins speaking.\n\n**What audio formats does the API support?**\n\nThe API natively supports 24kHz mono PCM16 (raw 16-bit signed integer) and G.711 (u-law and a-law) audio formats. Developers must capture microphone data, convert it to these specific formats on the client-side, and stream it as base64-encoded strings inside JSON WebSocket frames.\n\n**Do I need a separate server to coordinate WebSocket traffic?**\n\nYes. Running a serverless edge coordinator (such as Cloudflare Workers or a lightweight Node.js gateway) is highly recommended. The proxy receives microphone inputs from client browsers, attaches secure authorization headers, and redirects the binary stream to OpenAI's gateway.\n\n**How do I prevent echo loops in real-time voice agents?**\n\nEcho loops occur when the speaker audio leaks back into the client microphone, triggering false speaker interruption events. Developers must implement echo cancellation algorithms on the client-side or use headsets during testing to prevent feedback loops from breaking sessions.", "url": "https://wpnews.pro/news/build-voice-agents-openai-realtime-api-guide", "canonical_source": "https://dev.to/mecanik-dev/build-voice-agents-openai-realtime-api-guide-3d5h", "published_at": "2026-09-08 18:00:00+00:00", "updated_at": "2026-09-08 18:17:38.476466+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-products", "developer-tools"], "entities": ["OpenAI", "Cloudflare", "Mecanik"], "alternates": {"html": "https://wpnews.pro/news/build-voice-agents-openai-realtime-api-guide", "markdown": "https://wpnews.pro/news/build-voice-agents-openai-realtime-api-guide.md", "text": "https://wpnews.pro/news/build-voice-agents-openai-realtime-api-guide.txt", "jsonld": "https://wpnews.pro/news/build-voice-agents-openai-realtime-api-guide.jsonld"}}