{"slug": "your-pydantic-ai-agents-just-gained-a-voice", "title": "Your Pydantic AI agents just gained a voice", "summary": "Pydantic AI has launched realtime voice support for its AI agents, enabling live speech-to-speech conversations through a provider-agnostic API that works with OpenAI Realtime, Azure OpenAI, Gemini Live, and xAI Grok Voice. The feature, available via the pydantic-ai[realtime] extra, keeps tools, history, and API keys server-side, with optional browser WebRTC for direct audio streaming, and includes built-in observability through Logfire.", "body_md": "A Pydantic AI [agent](https://ai.pydantic.dev/agent/) is a plain Python object with no interface baked in. The same agent already runs [as an object you call run() on](https://ai.pydantic.dev/agent/#running-agents),\n\n[in the terminal](https://ai.pydantic.dev/cli/),\n\n[behind a built-in web chat](https://ai.pydantic.dev/web/), and\n\n[streamed to your own frontend](https://ai.pydantic.dev/ui/overview/). Now it can hold a live, spoken conversation too.\n\n[Voice is just another interface](https://ai.pydantic.dev/interfaces/)on the agent you already have.\n\nUnder the hood it's realtime speech-to-speech: [OpenAI Realtime](https://ai.pydantic.dev/realtime/openai/), [Azure OpenAI](https://ai.pydantic.dev/realtime/azure/), [Gemini Live](https://ai.pydantic.dev/realtime/gemini/), and [xAI Grok Voice](https://ai.pydantic.dev/realtime/xai/), behind one provider-agnostic API. The model hears and speaks directly, with no transcribe-then-generate-then-synthesize pipeline in between, so latency is low and interruptions feel natural.\n\nYour backend always runs the agent, so tools, history, and your API key stay server-side. The audio travels however suits your app: captured on the server, or, for a browser, [directly between the browser and the provider over WebRTC](https://ai.pydantic.dev/realtime/deployment/#browser-webrtc-server-sideband), with your backend attached to the same call as a sideband so nothing moves client-side but the audio. Browser WebRTC works on OpenAI and Azure OpenAI, with a [runnable FastAPI example](https://ai.pydantic.dev/examples/realtime-webrtc/) to start from.\n\nA voice agent in one file\n\nRealtime lives behind an extra. Install [Logfire](https://pydantic.dev/logfire) next to it and the session traces itself:\n\n```\nuv add \"pydantic-ai[realtime]\" logfire\n```\n\nWith the audio captured on the server, a voice agent is one session and three small loops (microphone in, speaker out, and a live transcript):\n\n``` python\nimport asyncio\nimport contextlib\nfrom collections.abc import AsyncIterator\n\nimport logfire\nfrom pydantic_ai import Agent\nfrom pydantic_ai.realtime import RealtimeSession\n\nlogfire.configure()\nlogfire.instrument_pydantic_ai()\n\nagent = Agent(instructions='You are a helpful voice assistant.')\n\n@agent.tool_plain\nasync def get_weather(city: str) -> str:\n    return f'Sunny in {city}'\n\nasync def stream_microphone(session: RealtimeSession) -> None:\n    ...  # capture or resample 16-bit mono PCM at `session.audio_input_sample_rate` and `await session.send_audio(chunk)`\n\nasync def play_audio(chunks: AsyncIterator[bytes]) -> None:\n    async for chunk in chunks:\n        ...  # write the PCM chunk to your speaker\n\nasync def main():\n    async with agent.realtime('openai:gpt-realtime-2.1').session() as session:\n        mic = asyncio.create_task(stream_microphone(session))\n        speaker = asyncio.create_task(play_audio(session.stream_audio()))\n\n        async for part in session.stream_transcripts():  # live captions\n            print(f'{part.speaker}: {part.transcript}')\n            #> user: What's the weather like in Paris?\n            #> assistant: It's sunny in Paris right now.\n            if part.speaker == 'assistant':\n                break  # one exchange; a real call keeps listening\n\n    mic.cancel()\n    with contextlib.suppress(asyncio.CancelledError):\n        await mic\n    await speaker\n\nasyncio.run(main())\n```\n\n`send_audio()`\n\nstreams the caller's microphone in, `stream_audio()`\n\nstreams the spoken reply back, and `stream_transcripts()`\n\ngives you a live transcript of both sides. The [voice assistant example](https://ai.pydantic.dev/examples/realtime-voice/) fills the two audio placeholders in with `sounddevice`\n\n. That `agent`\n\nis not a new `VoiceAgent`\n\ntype: it's the same `Agent`\n\n, and `get_weather`\n\nis your ordinary server-side tool, called mid-call. The two `logfire`\n\nlines are the whole of the observability setup.\n\nWhat carries over\n\nOpening a realtime socket to a provider is a few lines in any SDK. The work is everything around the model: tools that actually run, history you can trust, costs you can see, and a security model that keeps your API key off the browser. That's the part Pydantic AI does, and it works the way it does for text agents:\n\n**Your typed tools run server-side.** The same tools you registered with`@agent.tool`\n\n, with their dependencies, validation, and retries. Each call runs in the background so it never blocks the session; whether the model keeps speaking while it waits is provider-specific.**Your** The same opt-in behaviors you compose onto a text agent, including third-party ones, resolve once at connect time. ([capabilities](https://ai.pydantic.dev/capabilities/overview/)carry over.[Some run-graph features](https://ai.pydantic.dev/realtime/capabilities/)like output validators don't apply.)**The session builds real message history.** Spoken turns become the same`ModelRequest`\n\n/`ModelResponse`\n\nmessages a text run produces, including tool calls, and transcripts when the provider supplies them.**Usage and limits work.**`session.usage`\n\naccumulates tokens with audio and cached breakdowns;`usage_limits`\n\ncaps a runaway session the same way it caps a run.**It's** A realtime session emits OpenTelemetry spans the way an agent run does, so[instrumented](https://ai.pydantic.dev/realtime/observability/).[Logfire](/logfire)or any other OTel backend reads it without special handling.\n\nOne agent, many modalities\n\nBecause a voice session records canonical message history, voice and text compose. Hand a finished call to a text agent for structured extraction:\n\n``` python\nfrom typing import Literal\n\nfrom pydantic import BaseModel\n\nfrom pydantic_ai import Agent\nfrom pydantic_ai.realtime import RealtimeSession\n\nlogfire.configure()\nlogfire.instrument_pydantic_ai()\n\nclass SupportTicket(BaseModel):\n    summary: str\n    severity: Literal['low', 'medium', 'high']\n    next_action: str\n\nsupport_agent = Agent(instructions='You are a friendly support line. Keep replies short.')\nnotetaker = Agent('openai:gpt-5.6-sol', output_type=SupportTicket)\n\nasync def take_call(session: RealtimeSession) -> None:\n    ...  # stream the caller's mic in and play replies back until they hang up\n\nasync def main():\n    async with support_agent.realtime('openai:gpt-realtime-2.1').session() as session:\n        await take_call(session)\n\n    ticket = await notetaker.run(\n        'Summarize this call as a support ticket.',\n        message_history=session.all_messages(),\n    )\n    print(ticket.output)\n```\n\nThe voice call and the text summary are two agents sharing one message history. It goes the other way too: seed a voice session with `message_history=`\n\nfrom an earlier text conversation and the caller picks up where they left off. The phone bot and the assistant in your app can be the same `Agent`\n\nwith the same audit trail.\n\nPortable across providers, and through the gateway\n\nEvery provider implements the same `RealtimeModel`\n\ninterface and normalizes into one typed event vocabulary, so the core of your event loop stays the same when your provider changes. Each model declares its capabilities on its `profile`\n\n: manual turn-taking, barge-in truncation, non-blocking tool calls, and the provider-native tools it supports, like Gemini's search grounding. You branch on what a model can do instead of discovering it mid-call.\n\nPortability includes the [Pydantic AI Gateway](https://pydantic.dev/ai-gateway). Route a session through it by naming the upstream provider, and nothing else about your code changes:\n\n``` python\nfrom pydantic_ai import Agent\n\nagent = Agent(instructions='You are a helpful voice assistant.')\n\nagent.realtime('gateway/openai:gpt-realtime-2.1')\nagent.realtime('gateway/google:gemini-3.1-flash-live-preview')\n```\n\nI['mYour voice traffic then gets the same single key, spend limits, and routing as your text traffic. Apart from obvious reasons, this matters for audio because realtime usage is billed by each provider's audio-token or per-minute pricing, so an unattended call can run up cost *very* fast.\n\nTry it\n\nRealtime support ships in `pydantic-ai`\n\ntoday:\n\n```\nuv add \"pydantic-ai[realtime]\"\n```\n\nThat covers OpenAI, Azure OpenAI, and Gemini Live, because the `pydantic-ai`\n\npackage already bundles those SDKs. Add `[xai-realtime]`\n\nfor xAI Grok Voice. On the lean `pydantic-ai-slim`\n\npackage, name the provider yourself: `pydantic-ai-slim[openai-realtime]`\n\n(also covers Azure), `[google-realtime]`\n\n, or `[xai-realtime]`\n\n.\n\nStart with the [realtime docs](https://ai.pydantic.dev/realtime/overview/), then pick an example: a [terminal voice assistant](https://ai.pydantic.dev/examples/realtime-voice/), the [browser WebRTC app](https://ai.pydantic.dev/examples/realtime-webrtc/), a [camera agent that watches and narrates](https://ai.pydantic.dev/examples/realtime-camera/), or [handing a call off to a text agent](https://ai.pydantic.dev/examples/realtime-handoff/).\n\nSee what the call did\n\nVoice arrives in the trace as text. Spoken turns land under `pydantic_ai.all_messages`\n\n, the attribute a text run already writes, with transcripts standing in for the audio.\n\nOne session span holds the whole call, with a `user speech`\n\nspan for each stretch the caller talked and a `chat`\n\nspan for each reply. A tool call mid-conversation gets its own span, with the arguments the model chose and what your function returned. `pydantic_ai.audio_chunks_dropped`\n\nand `transcript_items_dropped`\n\ncount what the session could not keep up with.\n\nBelow is the agent trace from this post on a real call: four questions, three `execute_tool get_weather`\n\nspans, 61.87s costing $0.08, or about $4.50 for an hour of talking at that rate.\n\n[Pydantic Logfire](https://pydantic.dev/logfire) files each session alongside your other agent runs, since the session span carries the same `agent_name`\n\nits Runs view groups on.\n\nIf you build something with it, or hit something that doesn't feel right, we want to hear about it, on [GitHub](https://github.com/pydantic/pydantic-ai) or in [Slack](https://logfire.pydantic.dev/docs/join-slack/).", "url": "https://wpnews.pro/news/your-pydantic-ai-agents-just-gained-a-voice", "canonical_source": "https://pydantic.dev/articles/pydantic-ai-voice-agent", "published_at": "2026-08-26 09:00:00+00:00", "updated_at": "2026-08-27 15:50:23.570420+00:00", "lang": "en", "topics": ["ai-tools", "ai-products", "ai-infrastructure"], "entities": ["Pydantic AI", "OpenAI", "Azure OpenAI", "Gemini Live", "xAI Grok Voice", "Logfire", "FastAPI"], "alternates": {"html": "https://wpnews.pro/news/your-pydantic-ai-agents-just-gained-a-voice", "markdown": "https://wpnews.pro/news/your-pydantic-ai-agents-just-gained-a-voice.md", "text": "https://wpnews.pro/news/your-pydantic-ai-agents-just-gained-a-voice.txt", "jsonld": "https://wpnews.pro/news/your-pydantic-ai-agents-just-gained-a-voice.jsonld"}}