Build Voice Agents: OpenAI Realtime API Guide A developer guide explains how to build low-latency voice agents using OpenAI's Realtime API, which processes audio natively over a WebSocket connection, reducing network latency below 300 milliseconds. The guide covers connection setup, session configuration, and security best practices, such as proxying API keys through edge middleware. Building low-latency audio pipelines with the OpenAI Realtime API lets developers launch human-like conversational voice agents in production. Traditionally, building a voice interface meant chaining three separate model layers: automatic speech recognition ASR , a text-based LLM logic layer, and text-to-speech TTS synthesis. That multi-step pipeline introduced significant round-trip network delays, making natural conversation impossible. Native audio processing over a persistent WebSocket connection changes this, reducing network latency below 300 milliseconds. This guide explains how to establish connection states, stream raw audio buffers, and optimise session configurations. API Security Warning: Never expose your OpenAI API key directly inside client-side browser scripts. Always proxy the WebSocket connection through a secure edge middleware such as a Cloudflare Worker that appends authorisation headers before forwarding packets to OpenAI. Key Takeaways: WebSocket Connection: Connect directly to OpenAI's realtime WebSocket gateway using edge proxies. Native Modalities: Specify both text and audio in your initial session update config payload. Audio Format: Stream user speech as base64-encoded mono PCM16 chunks at 24kHz. Speech Interruption: Monitor server speech-started signals to halt client playback instantly. Traditional voice stacks treat both speech recognition and synthesis models as external wrappers around a central text model. Native audio models remove that overhead by handling speech directly. With OpenAI's realtime models, the network processes audio inputs and outputs natively. The model receives raw audio waveforms, parses tone, inflection, and content, and generates natural voice output directly. As a result, the pipeline eliminates ASR transcription errors and TTS synthesis bottlenecks. Managing this persistent connection relies on WebSockets. The connection remains open throughout the call, allowing the agent to interrupt its output if it detects user speech, so the experience matches a real telephone call. {{< cta-button url="/en/ai-integration-services/" text="Get AI Integration Services" }} Before writing a line of code, make sure your environment is ready. This build assumes you are comfortable with asynchronous JavaScript and have the following in place: npm create cloudflare@latest . ws package for a Node.js gateway, or the native WebSocket global available inside Workers. getUserMedia plus an AudioWorklet for resampling . Budget a little time for account setup too: Realtime access and billing must be enabled on your organisation before the gateway will accept a session. To start, you open a connection to the OpenAI Realtime gateway, specifying the realtime model in your headers. The JavaScript code below demonstrates how to initialise the connection, configure session modalities, and handle streaming input and output buffers: python import WebSocket from "ws"; export async function startVoiceAgent env { // Connect to the OpenAI Realtime WebSocket gateway const url = "wss://api.openai.com/v1/realtime?model=gpt-realtime"; const ws = new WebSocket url, { headers: { "Authorization": Bearer ${env.OPENAI API KEY} , "OpenAI-Beta": "realtime=v1" } } ; ws.on "open", = { console.log "WebSocket connection established with OpenAI Realtime API" ; // Configure session modalities and voice parameters const sessionConfig = { type: "session.update", session: { modalities: "text", "audio" , instructions: "You are a helpful customer service assistant for Mecanik.", voice: "alloy", input audio format: "pcm16", output audio format: "pcm16", temperature: 0.7 } }; ws.send JSON.stringify sessionConfig ; } ; ws.on "message", data = { const event = JSON.parse data ; // Handle incoming audio content from the server if event.type === "response.audio.delta" { const audioBuffer = Buffer.from event.delta, "base64" ; // Output buffer to client audio player playAudioChunk audioBuffer ; } } ; } When building this handler, ensure your API key remains hidden from the client browser. Specifically, establish an edge middleware on your server to handle authorisation before proxying the WebSocket connection. To learn about edge API configurations, read our guide on building a serverless API with Cloudflare Workers https://mecanik.dev/en/posts/building-a-serverless-api-with-cloudflare-workers/ . The snippet above connects from a trusted server, but it never showed the piece that keeps your key safe: the proxy itself. On Cloudflare Workers you cannot attach custom headers to the new WebSocket constructor, so you open the upstream connection with fetch and an Upgrade header instead. The Worker accepts the browser's socket, dials OpenAI with your secret key attached, then pipes frames between the two. export default { async fetch request, env { if request.headers.get "Upgrade" == "websocket" { return new Response "Expected a WebSocket upgrade", { status: 426 } ; } // 1. Accept the browser <- Worker socket const client, server = Object.values new WebSocketPair ; server.accept ; // 2. Open the Worker <- OpenAI socket with the secret key attached const upstreamResponse = await fetch "https://api.openai.com/v1/realtime?model=gpt-realtime", { headers: { Upgrade: "websocket", Authorization: Bearer ${env.OPENAI API KEY} , "OpenAI-Beta": "realtime=v1" } } ; const upstream = upstreamResponse.webSocket; if upstream { return new Response "Upstream refused the upgrade", { status: 502 } ; } upstream.accept ; // 3. Pipe frames in both directions server.addEventListener "message", e = upstream.send e.data ; upstream.addEventListener "message", e = server.send e.data ; const close = = { try { server.close ; upstream.close ; } catch {} }; server.addEventListener "close", close ; upstream.addEventListener "close", close ; return new Response null, { status: 101, webSocket: client } ; } }; Store the key as an encrypted secret with wrangler secret put OPENAI API KEY rather than in wrangler.toml , so it never lands in your repository. The browser now connects to wss://your-worker.workers.dev and never sees the credential. Cloudflare documents this bidirectional pattern in its Workers WebSockets reference https://developers.cloudflare.com/workers/runtime-apis/websockets/ . Once the session update is accepted, your client must capture microphone input, compress it to 24kHz mono PCM16 data, and stream it as base64 segments. Because the connection remains persistent, handling network dropouts is critical. A local chunk buffer ensures that brief cellular connection dropouts do not result in audio packet loss or jittery agent responses: the client retains the buffer and replays it immediately upon reconnect. // Example of streaming user microphone data function streamMicrophoneChunk ws, base64AudioChunk { const audioEvent = { type: "input audio buffer.append", audio: base64AudioChunk }; ws.send JSON.stringify audioEvent ; } Whenever the user stops speaking, the server processes the accumulated audio buffer automatically and triggers a model response. For a complete API event list, review the OpenAI Realtime Guide https://platform.openai.com/docs/guides/realtime . Additionally, implement echo cancellation in your frontend player. If the microphone picks up the speaker output, the agent will interpret its own voice as a user interruption, causing the session loop to fail. To learn more about frontend optimisation, check our guide on WordPress vs custom web development https://mecanik.dev/en/posts/wordpress-vs-custom-web-development-what-uk-businesses-need-to-know/ . By default you have to tell the model when a turn ends. Enabling server-side voice activity detection VAD hands that job to OpenAI: the gateway watches the incoming buffer, decides when the user has stopped speaking, and triggers a response automatically. Add a turn detection block to the session update you send during the handshake. js const sessionConfig = { type: "session.update", session: { modalities: "text", "audio" , voice: "alloy", input audio format: "pcm16", output audio format: "pcm16", turn detection: { type: "server vad", threshold: 0.5, prefix padding ms: 300, silence duration ms: 500 } } }; With VAD active, the server emits input audio buffer.speech started the instant it hears the user talk over the agent. Treat that event as a hard stop: flush every queued audio chunk in your player, otherwise the previous response keeps playing while the new one arrives. js let playbackQueue = ; ws.on "message", raw = { const event = JSON.parse raw ; if event.type === "input audio buffer.speech started" { // User is interrupting: drop everything still buffered playbackQueue = ; stopSpeaker ; } } ; Raising silence duration ms makes the agent wait longer before replying, which suits slow or hesitant speakers; lowering it makes the exchange feel snappier but risks cutting people off mid-sentence. To deploy your voice agent middleware, start by configuring a secure WebSocket routing gateway on serverless edge nodes to protect your OpenAI credentials. This prevents scrapers from discovering your keys. Next, draft your system prompts and rules carefully. Set the tone, vocabulary, and response instructions inside the initial payload update session event. Consequently, this establishes a clear scope for conversational flows. Then implement robust speaker interruption logic. You should listen for the input audio buffer.speech started event and halt audio playback in your frontend player immediately. Finally, optimise global latency by deploying your Workers in close regional proximity to your users to reduce routing hops. To explore edge hosting options, read our Cloudflare Workers AI tutorial https://mecanik.dev/en/posts/cloudflare-workers-ai-tutorial/ . Most first-run failures come from a handful of predictable mistakes. The table below maps the symptom you will see to its usual cause and fix. | Symptom | Likely cause | Fix | |---|---|---| | Connection closes with a 401 immediately | Missing or malformed Authorization header | Confirm the proxy attaches Bearer