Your Pydantic AI agents just gained a voice

Pydantic AI has launched realtime voice support for its AI agents, enabling live speech-to-speech conversations through a provider-agnostic API that works with OpenAI Realtime, Azure OpenAI, Gemini Live, and xAI Grok Voice. The feature, available via the pydantic-ai[realtime] extra, keeps tools, history, and API keys server-side, with optional browser WebRTC for direct audio streaming, and includes built-in observability through Logfire.

A Pydantic AI agent https://ai.pydantic.dev/agent/ is a plain Python object with no interface baked in. The same agent already runs as an object you call run on https://ai.pydantic.dev/agent/ running-agents , in the terminal https://ai.pydantic.dev/cli/ , behind a built-in web chat https://ai.pydantic.dev/web/ , and streamed to your own frontend https://ai.pydantic.dev/ui/overview/ . Now it can hold a live, spoken conversation too. Voice is just another interface https://ai.pydantic.dev/interfaces/ on the agent you already have. Under the hood it's realtime speech-to-speech: OpenAI Realtime https://ai.pydantic.dev/realtime/openai/ , Azure OpenAI https://ai.pydantic.dev/realtime/azure/ , Gemini Live https://ai.pydantic.dev/realtime/gemini/ , and xAI Grok Voice https://ai.pydantic.dev/realtime/xai/ , behind one provider-agnostic API. The model hears and speaks directly, with no transcribe-then-generate-then-synthesize pipeline in between, so latency is low and interruptions feel natural. Your backend always runs the agent, so tools, history, and your API key stay server-side. The audio travels however suits your app: captured on the server, or, for a browser, directly between the browser and the provider over WebRTC https://ai.pydantic.dev/realtime/deployment/ browser-webrtc-server-sideband , with your backend attached to the same call as a sideband so nothing moves client-side but the audio. Browser WebRTC works on OpenAI and Azure OpenAI, with a runnable FastAPI example https://ai.pydantic.dev/examples/realtime-webrtc/ to start from. A voice agent in one file Realtime lives behind an extra. Install Logfire https://pydantic.dev/logfire next to it and the session traces itself: uv add "pydantic-ai realtime " logfire With the audio captured on the server, a voice agent is one session and three small loops microphone in, speaker out, and a live transcript : python import asyncio import contextlib from collections.abc import AsyncIterator import logfire from pydantic ai import Agent from pydantic ai.realtime import RealtimeSession logfire.configure logfire.instrument pydantic ai agent = Agent instructions='You are a helpful voice assistant.' @agent.tool plain async def get weather city: str - str: return f'Sunny in {city}' async def stream microphone session: RealtimeSession - None: ... capture or resample 16-bit mono PCM at session.audio input sample rate and await session.send audio chunk async def play audio chunks: AsyncIterator bytes - None: async for chunk in chunks: ... write the PCM chunk to your speaker async def main : async with agent.realtime 'openai:gpt-realtime-2.1' .session as session: mic = asyncio.create task stream microphone session speaker = asyncio.create task play audio session.stream audio async for part in session.stream transcripts : live captions print f'{part.speaker}: {part.transcript}' user: What's the weather like in Paris? assistant: It's sunny in Paris right now. if part.speaker == 'assistant': break one exchange; a real call keeps listening mic.cancel with contextlib.suppress asyncio.CancelledError : await mic await speaker asyncio.run main send audio streams the caller's microphone in, stream audio streams the spoken reply back, and stream transcripts gives you a live transcript of both sides. The voice assistant example https://ai.pydantic.dev/examples/realtime-voice/ fills the two audio placeholders in with sounddevice . That agent is not a new VoiceAgent type: it's the same Agent , and get weather is your ordinary server-side tool, called mid-call. The two logfire lines are the whole of the observability setup. What carries over Opening a realtime socket to a provider is a few lines in any SDK. The work is everything around the model: tools that actually run, history you can trust, costs you can see, and a security model that keeps your API key off the browser. That's the part Pydantic AI does, and it works the way it does for text agents: Your typed tools run server-side. The same tools you registered with @agent.tool , with their dependencies, validation, and retries. Each call runs in the background so it never blocks the session; whether the model keeps speaking while it waits is provider-specific. Your The same opt-in behaviors you compose onto a text agent, including third-party ones, resolve once at connect time. capabilities https://ai.pydantic.dev/capabilities/overview/ carry over. Some run-graph features https://ai.pydantic.dev/realtime/capabilities/ like output validators don't apply. The session builds real message history. Spoken turns become the same ModelRequest / ModelResponse messages a text run produces, including tool calls, and transcripts when the provider supplies them. Usage and limits work. session.usage accumulates tokens with audio and cached breakdowns; usage limits caps a runaway session the same way it caps a run. It's A realtime session emits OpenTelemetry spans the way an agent run does, so instrumented https://ai.pydantic.dev/realtime/observability/ . Logfire /logfire or any other OTel backend reads it without special handling. One agent, many modalities Because a voice session records canonical message history, voice and text compose. Hand a finished call to a text agent for structured extraction: python from typing import Literal from pydantic import BaseModel from pydantic ai import Agent from pydantic ai.realtime import RealtimeSession logfire.configure logfire.instrument pydantic ai class SupportTicket BaseModel : summary: str severity: Literal 'low', 'medium', 'high' next action: str support agent = Agent instructions='You are a friendly support line. Keep replies short.' notetaker = Agent 'openai:gpt-5.6-sol', output type=SupportTicket async def take call session: RealtimeSession - None: ... stream the caller's mic in and play replies back until they hang up async def main : async with support agent.realtime 'openai:gpt-realtime-2.1' .session as session: await take call session ticket = await notetaker.run 'Summarize this call as a support ticket.', message history=session.all messages , print ticket.output The voice call and the text summary are two agents sharing one message history. It goes the other way too: seed a voice session with message history= from an earlier text conversation and the caller picks up where they left off. The phone bot and the assistant in your app can be the same Agent with the same audit trail. Portable across providers, and through the gateway Every provider implements the same RealtimeModel interface and normalizes into one typed event vocabulary, so the core of your event loop stays the same when your provider changes. Each model declares its capabilities on its profile : manual turn-taking, barge-in truncation, non-blocking tool calls, and the provider-native tools it supports, like Gemini's search grounding. You branch on what a model can do instead of discovering it mid-call. Portability includes the Pydantic AI Gateway https://pydantic.dev/ai-gateway . Route a session through it by naming the upstream provider, and nothing else about your code changes: python from pydantic ai import Agent agent = Agent instructions='You are a helpful voice assistant.' agent.realtime 'gateway/openai:gpt-realtime-2.1' agent.realtime 'gateway/google:gemini-3.1-flash-live-preview' I 'mYour voice traffic then gets the same single key, spend limits, and routing as your text traffic. Apart from obvious reasons, this matters for audio because realtime usage is billed by each provider's audio-token or per-minute pricing, so an unattended call can run up cost very fast. Try it Realtime support ships in pydantic-ai today: uv add "pydantic-ai realtime " That covers OpenAI, Azure OpenAI, and Gemini Live, because the pydantic-ai package already bundles those SDKs. Add xai-realtime for xAI Grok Voice. On the lean pydantic-ai-slim package, name the provider yourself: pydantic-ai-slim openai-realtime also covers Azure , google-realtime , or xai-realtime . Start with the realtime docs https://ai.pydantic.dev/realtime/overview/ , then pick an example: a terminal voice assistant https://ai.pydantic.dev/examples/realtime-voice/ , the browser WebRTC app https://ai.pydantic.dev/examples/realtime-webrtc/ , a camera agent that watches and narrates https://ai.pydantic.dev/examples/realtime-camera/ , or handing a call off to a text agent https://ai.pydantic.dev/examples/realtime-handoff/ . See what the call did Voice arrives in the trace as text. Spoken turns land under pydantic ai.all messages , the attribute a text run already writes, with transcripts standing in for the audio. One session span holds the whole call, with a user speech span for each stretch the caller talked and a chat span for each reply. A tool call mid-conversation gets its own span, with the arguments the model chose and what your function returned. pydantic ai.audio chunks dropped and transcript items dropped count what the session could not keep up with. Below is the agent trace from this post on a real call: four questions, three execute tool get weather spans, 61.87s costing $0.08, or about $4.50 for an hour of talking at that rate. Pydantic Logfire https://pydantic.dev/logfire files each session alongside your other agent runs, since the session span carries the same agent name its Runs view groups on. If you build something with it, or hit something that doesn't feel right, we want to hear about it, on GitHub https://github.com/pydantic/pydantic-ai or in Slack https://logfire.pydantic.dev/docs/join-slack/ .