Building Low-Latency Voice Agents with OpenAI’s GPT-Realtime-2.1 and GPT-Realtime-2.1-mini

OpenAI released GPT-Realtime-2.1 and GPT-Realtime-2.1-mini, low-latency voice models that handle audio generation and understanding through a single model over a live connection. The mini model adds reasoning to the realtime voice stack at the same price as the earlier mini, with p95 latency improved by at least 25% due to better caching. The release enables developers to build voice agents with lower end-to-end latency and better conversational feel without chaining separate speech-to-text and text-to-speech services.

OpenAI’s latest Realtime API update is interesting for builders for one main reason: it makes voice agents easier to ship without forcing you into the classic speech-to-text plus text-to-speech pipeline. The release includes two models: gpt-realtime-2.1 gpt-realtime-2.1-mini Both are aimed at low-latency voice and multimodal experiences. The mini model is the notable one if you care about cost and throughput, because it adds reasoning to the realtime voice stack while keeping the same price as the earlier gpt-realtime-mini . OpenAI also says p95 latency improved by at least 25% across Realtime voice models thanks to better caching. For teams building support bots, scheduling assistants, or in-app voice UX, those details matter more than benchmark bragging rights. The core architectural change is that the Realtime API handles audio generation and understanding through a single model over a live connection. That means you are not chaining separate STT and TTS services together and paying the latency penalty between them. That single-model approach has two practical benefits: Lower end-to-end latency Less handoff overhead means the user hears a response sooner. Better conversational feel Speech nuance is easier to preserve when the system is not breaking the interaction into multiple disconnected steps. For voice agents, that matters because delays are not just a performance issue. They change user behavior. If a model goes silent during a tool call, people assume the call dropped and start talking over it. gpt-realtime-2.1-mini adds gpt-realtime-2.1-mini is positioned as the faster, more cost-efficient option. The important addition is that it is now a mini reasoning model for realtime voice . That means the model can internally think before speaking, even in a live audio session. It also supports tool use through the Realtime API, so the flow can look like this: That combination is useful because it gives you a way to keep the conversation moving while still letting the model coordinate multi-step actions. A lot of voice assistant failures are not about intelligence. They are about pacing. Without reasoning and a clear spoken preamble, a model may jump straight into a function call and then sit quietly. In a live conversation, that silence feels broken. With reasoning, the assistant can say something like: Those short bridge phrases keep the user oriented while the backend function runs. OpenAI also exposes reasoning effort controls: minimal , low , medium , high , and xhigh . The default is low , and that is the recommended starting point for most production voice agents. Higher effort means more latency and more output tokens, so you should only raise it when the task actually needs deeper planning. The release gives you a clear tradeoff. gpt-realtime-2.1 when: The full model also supports speech-to-speech with configurable reasoning effort and tool use. gpt-realtime-2.1-mini when: This is the practical builder’s decision: if the assistant is doing high-frequency, everyday tasks, the mini tier is likely the better default. If the interaction is high-stakes or the agent has to be especially robust, the full model is the safer choice. OpenAI’s pricing is split by text, audio, and image tokens. The mini model keeps the earlier mini pricing while adding reasoning. A few published numbers worth noting: gpt-realtime-2.1-mini audio output: gpt-realtime-2.1 audio output: gpt-realtime-2.1-mini audio input: The more important operational detail is caching. OpenAI says p95 latency dropped at least 25% across Realtime voice models because of improved caching, and cached input tokens are billed at a steep discount. For long-lived voice sessions, that can make a big difference because the system prompt is reused after the first turn. In practice, that means your setup cost gets amortized across the conversation. Caching helps, but it does not make realtime usage “cheap” in an abstract sense. Audio token costs are still harder to reason about than plain text calls, especially when: If you are planning production rollout, you should estimate cost per session with your actual traffic patterns, not just token rates. The documented connection pattern is straightforward: The security model is also important. You should keep your standard API key on the server and mint a short-lived client secret for the browser. Your server creates a realtime session with the target model, instructions, reasoning effort, and tools. The response returns an ephemeral key that the browser can use safely for the live connection. This is the right pattern because it avoids exposing your long-lived API key client-side. On the client, the flow is: RTCPeerConnection That setup is enough to start a realtime session where the model can listen, respond, and send structured events over the data channel. The basic pattern is useful if you are building: A few examples from the release show where the model mix makes sense: Customer support triage The model can look up an account, inspect an invoice, and narrate the process instead of going silent during tool calls. Appointment scheduling Exact values matter here. A voice agent can confirm dates and then call a reschedule function once the details are precise. In-app voice assistant Lower cost makes it more realistic to keep the feature available at scale. Field data capture Improved alphanumeric recognition helps when users speak codes, part numbers, or serial values that need to be repeated back accurately. If I were integrating this into a product, I would start with...