How to Build a Voice AI Agent with OpenAI's GPT-Live, Twilio Agent Connect, and TypeScript

Twilio released a GPTLiveProvider in its open-source TypeScript SDK Twilio Agent Connect (v2.4.0+) that bridges Twilio Programmable Voice Media Streams directly to OpenAI's GPT-Live-1 speech-to-speech model, letting developers build inbound and outbound phone voice agents without custom WebSocket plumbing. The tutorial requires Node.js 22.13+, a Twilio phone number with Voice capability, and an OpenAI API key with GPT-Live access, and uses a declarative config interface where defaultSessionConfig is sent as GPT-Live's session.start payload with the model, instructions, and audio.format set to TWILIO_AUDIO_FORMAT_FOR_GPT_LIVE.

How to Build a Voice AI Agent with OpenAI's GPT-Live, Twilio Agent Connect, and TypeScript Time to read: Written by Reviewed by "It works great in my testing script, but the second we put it on a real phone call, it falls apart." That's a common complaint once a voice AI demo meets production traffic. OpenAI's GPT-Live https://openai.com/index/introducing-gpt-live/ is a family of full-duplex speech-to-speech, or S2S, models. They separate the conversation and any necessary background work, allowing you to delegate to your own backend or an OpenAI hosted model. Twilio Agent Connect https://www.twilio.com/docs/conversations/agent-connect is Twilio's open-source TypeScript SDK for building voice and messaging AI agents, and we now ship a GPTLiveProvider that bridges Twilio Programmable Voice https://www.twilio.com/docs/voice Media Streams https://www.twilio.com/docs/voice/guides/media-streams-guides directly to GPT-Live-1 in the OpenAI API https://openai.com/index/introducing-gpt-live-1-in-the-api/ . It all works through a flexible, declarative config interface – you don’t need any custom WebSocket plumbing. In this tutorial, you'll use TypeScript and build a phone number that connects inbound callers to a GPT-Live-powered voice agent. It’ll greet the caller and the caller can ask it a question that triggers a tool call. Then, I’ll show you how to place an outbound call with Twilio and GPT-Live. While we build together, I’ll also show you other features like per-call customization, and you can explore from there. Let’s get started Prerequisites and common pitfalls to getting started prerequisites-and-common-pitfalls-to-getting-started - A free Twilio account - Sign up for an account here https://www.twilio.com/try-twilio . - A Twilio phone number with Voice capability - buy one from your Twilio Console https://1console.twilio.com/ . - An OpenAI API key from a project with GPT-Live access. - Node.js 22.13+ and the Twilio Agent Connect https://twilio.github.io/twilio-agent-connect-typescript/ SDK v2.4.0+ installed, along with dotenv npm install twilio-agent-connect dotenv . - A public HTTPS/WSS endpoint for local development — ngrok https://ngrok.com/ works well. Build the app build-the-app The full working example lives at getting started/examples/gpt-live https://github.com/twilio/twilio-agent-connect-typescript/tree/main/getting started/examples/gpt-live . Follow along below, or clone the repo and run that example directly. Step 1: Set up TAC and the Fastify server toc-heading-407bf5aa-5f27-4e47-ba99-3012cf2c08f6 Every TAC voice app starts with a Twilio Agent Connect instance and a VoiceChannel . VoiceChannel is provider-agnostic: the same class hosts Conversation Relay https://www.twilio.com/docs/voice/twiml/connect/conversationrelay , GPT Realtime, and GPT-Live, depending on which config you pass it. TACConfig.fromEnv reads its settings from environment variables. Create a .env file in your project root with TWILIO ACCOUNT SID and TWILIO AUTH TOKEN found on your Console dashboard https://console.twilio.com/ , TWILIO API KEY and TWILIO API SECRET from API keys & tokens , TWILIO PHONE NUMBER from Phone Numbers Manage Active Numbers , TWILIO VOICE PUBLIC DOMAIN your ngrok domain from the prerequisites step , and OPENAI API KEY , and the dotenv/config import above picks them up automatically. Step 2: Configure the GPT-Live session and greeting toc-heading-3456827a-1417-43a6-a86a-28446c5f72e8 defaultSessionConfig is sent as GPT-Live's session.start payload the moment the model connects. At minimum it needs the model, instructions the system prompt , and audio.format set to TWILIO AUDIO FORMAT FOR GPT LIVE . Setting welcomeInstruction makes the agent speak first instead of waiting for the caller, which is useful for any call where the caller doesn't know they've reached an AI agent yet. Step 3: Wire up a tool call toc-heading-5835064a-a695-4871-a2e3-43cb803f9257 GPT-Live doesn't call functions directly – instead, it delegates tool calls https://developers.openai.com/api/docs/guides/live-delegation either through Client Delegation where you need to build the delegation integration or through a Responses API model. In this tutorial, we’ll use Responses delegation https://developers.openai.com/api/docs/guides/live-delegation?delegation-mode=responses configure-responses-delegation . Any tool created with defineTool can be exposed this way: register it in tools: ... so Twilio Agent Connect can execute it, and separately list its schema under delegation.responses.tools so the model knows it exists. Keeping tool execution outside the real-time voice loop means your tool logic can be as slow or complex as it needs to be. Consider all the things you might include in your logic: a database lookup, an API call, or a business rule. The 3-second setTimeout in the code above makes the strength of this model visible: run the demo and the call keeps going while the tool runs. GPT-Live delegates the decision to call a tool to a Responses API model, then Twilio Agent Connect runs the actual TypeScript function and speaks back the result. And hopefully, that shows how straightforward OpenAI’s Responses delegation is. Of course,when you build your own agent, swap getWeather for a function that hits your own backend. For example, you might look up an order status, check appointment availability, or pull account details. All the same wiring applies without changing anything else here. Step 4: Wire up the server toc-heading-f5750e4c-7127-4285-b240-4a4378e70b4c TACServer mounts the TwiML endpoint at /twiml by default. Step 5: Point your Twilio number at the webhook toc-heading-2871d186-5f1d-456b-ac80-13c97d431480 With the server running behind a public tunnel like ngrok , tell Twilio where to send incoming calls: - In the Twilio Console, go to Phone Numbers Manage Active Numbers and select your number. - Under Voice Configuration, set "A call comes in" to Webhook, and enter https://<your-domain /twiml . - Set the HTTP method to POST , then save. Now, you're ready to call in ☎️ And there you have it: you now have a phone number that greets callers, answers questions, and calls real functions, with no WebSocket code required. Run, test, troubleshoot toc-heading-301badde-36ba-435a-b107-e9e51c98eb60 Call your Twilio number… if everything is set up correctly, you should hear the agent's welcomeInstruction immediately, without saying anything first. After the greeting, ask "what's the weather in Los Angeles?" — the model should invoke get weather via Responses delegation and read back the result in the same turn. Have fun talking to your agent – and when you hang up, the transcript accumulated in ConversationSession.metadata.transcript prints to your console if you register an onConversationEnded callback that reads it off the session when the call ends. Here’s an example: If you don't hear a greeting, double check that welcomeInstruction is set. Going further: placing outbound calls toc-heading-9f46f0ff-cbbc-4997-b4dc-24713eaeb861 Now that you have an agent callers can reach, let's build one that reaches out to them instead. Place an outbound call place-an-outbound-call The same GPTLiveProvider that answers inbound calls can also place them. An outbound agent is useful for proactive outreach like appointment reminders, order updates, or callback flows where the agent initiates contact instead of waiting for a caller to dial in. To complete the outbound agent demo, continue in the same file and reuse the voiceChannel you already built in Steps 1-4. to must be in E.164 format https://www.twilio.com/docs/glossary/what-e164 e.g., +15551234567 , which is what Twilio's Voice API requires. In the full example, this runs behind a --to flag. Start the server with npm run dev -- --to +15551234567 and it places the call as soon as the server starts, while still answering inbound calls as usual. That's the entire outbound flow. TAC handles placing the call, connecting the Media Stream, and bridging it to GPT-Live the same way it does for an inbound call. From here, initiateOutboundConversation is the one API you need whether you're calling one number or looping over a list to run a whole outbound campaign. Other features to explore This tutorial covers the basics, but Twilio Agent Connect and GPT-Live support more than what's shown here: - One SDK, every channel. TAC also ships providers for SMS, RCS, WhatsApp, and Chat https://www.twilio.com/docs/conversations/agent-connect/channels , if your agent needs to talk to customers outside of voice. - Swappable voice backends. The same VoiceChannel API works with Conversation Relay and the OpenAI Realtime API https://openai.com/index/introducing-gpt-realtime/ . - Production-ready call handling. Status, answering-machine detection, and recording callbacks, plus programmatic hang-up, for turning a demo into something that survives real phone traffic. - Per-call session customization. Override the session config for a specific call without touching your channel-wide defaults. Conclusion Congratulations You now have a phone number that connects callers to a GPT-Live-1-powered voice agent, whether by picking up when they place a call or by dialing outbound. You agent greets people proactively, and can answer more advanced questions by calling a tool. The same pattern I laid out in this post works for any scenario where a caller needs a tool-using voice agent. So whatever you’re building – from appointment booking to order status lookups – this example extends to your use case, and any per-caller prompts and other features when you need them. From here, check out the Twilio Agent Connect API reference https://twilio.github.io/twilio-agent-connect-typescript/ for the full VoiceProvider interface, or the OpenAI Realtime API provider if you need an alternative speech-to-speech backend. Additional Resources Wen Zhu is a Software Engineer at Twilio. Away from the keyboard, Wen heads for the trails whenever the weather cooperates, is usually planning the next trip before the current one ends, and dials in shots on a home espresso machine with the kind of patience normally saved for chasing down flaky tests. Wen can be reached on LinkedIn https://www.linkedin.com/in/wen-zhu/ . Xinghao Huang is a Software Engineer at Twilio. Off the clock, he cooks his way through both Chinese and Western cuisines, keeps a spice cabinet of a few dozen jars — all matching, naturally — better organized than most of his codebases, and maintains a personal recipe website. He can be reached at jahuang at twilio.com. Related Posts Related Resources Twilio Docs From APIs to SDKs to sample apps API reference documentation, SDKs, helper libraries, quickstarts, and tutorials for your language and platform. Resource Center The latest ebooks, industry reports, and webinars Learn from customer engagement experts to improve your own communication. Ahoy Twilio's developer community hub Best practices, code samples, and inspiration to build communications and digital engagement experiences.