# How to Build a Voice AI Agent with OpenAI's GPT-Live, Twilio Agent Connect, and TypeScript > Source: > Published: 2026-10-15 00:00:00+00:00 # How to Build a Voice AI Agent with OpenAI's GPT-Live, Twilio Agent Connect, and TypeScript Time to read: **Written by** **Reviewed by** *"It works great in my testing script, but the second we put it on a real phone call, it falls apart."* That's a common complaint once a voice AI demo meets production traffic. [OpenAI's GPT-Live](https://openai.com/index/introducing-gpt-live/) is a family of full-duplex speech-to-speech, or S2S, models. They separate the conversation and any necessary background work, allowing you to delegate to your own backend or an OpenAI hosted model. [Twilio Agent Connect](https://www.twilio.com/docs/conversations/agent-connect) is Twilio's open-source TypeScript SDK for building voice and messaging AI agents, and we now ship a `GPTLiveProvider` that bridges [Twilio Programmable Voice](https://www.twilio.com/docs/voice) [Media Streams](https://www.twilio.com/docs/voice/guides/media-streams-guides) directly to [GPT-Live-1 in the OpenAI API](https://openai.com/index/introducing-gpt-live-1-in-the-api/). It all works through a flexible, declarative config interface – you don’t need any custom WebSocket plumbing. In this tutorial, you'll use TypeScript and build a phone number that connects inbound callers to a GPT-Live-powered voice agent. It’ll greet the caller and the caller can ask it a question that triggers a tool call. Then, I’ll show you how to place an outbound call with Twilio and GPT-Live. While we build together, I’ll also show you other features like per-call customization, and you can explore from there. Let’s get started! ## [Prerequisites and common pitfalls to getting started](#prerequisites-and-common-pitfalls-to-getting-started) - A free Twilio account - [Sign up for an account here](https://www.twilio.com/try-twilio) . - A Twilio phone number with Voice capability - [buy one from your Twilio Console](https://1console.twilio.com/) . - An OpenAI API key from a project with GPT-Live access. - Node.js 22.13+ and the [Twilio Agent Connect](https://twilio.github.io/twilio-agent-connect-typescript/) SDK v2.4.0+ installed, along with`dotenv` (`npm install twilio-agent-connect dotenv` ). - A public HTTPS/WSS endpoint for local development — [ngrok](https://ngrok.com/) works well. ## [Build the app](#build-the-app) The full working example lives at [getting_started/examples/gpt-live](https://github.com/twilio/twilio-agent-connect-typescript/tree/main/getting_started/examples/gpt-live). Follow along below, or clone the repo and run that example directly. ### [Step 1: Set up TAC and the Fastify server](#toc-heading-407bf5aa-5f27-4e47-ba99-3012cf2c08f6) Every TAC voice app starts with a Twilio Agent Connect instance and a `VoiceChannel`. `VoiceChannel` is provider-agnostic: the same class hosts [Conversation Relay](https://www.twilio.com/docs/voice/twiml/connect/conversationrelay), GPT Realtime, and GPT-Live, depending on which config you pass it. `TACConfig.fromEnv()` reads its settings from environment variables. Create a `.env` file in your project root with `TWILIO_ACCOUNT_SID` and `TWILIO_AUTH_TOKEN` (found on your [Console dashboard](https://console.twilio.com/)), `TWILIO_API_KEY` and `TWILIO_API_SECRET` (from API keys & tokens), `TWILIO_PHONE_NUMBER` (from Phone Numbers > Manage > Active Numbers), `TWILIO_VOICE_PUBLIC_DOMAIN` (your ngrok domain from the prerequisites step), and `OPENAI_API_KEY`, and the `dotenv/config` import above picks them up automatically. ### [Step 2: Configure the GPT-Live session and greeting](#toc-heading-3456827a-1417-43a6-a86a-28446c5f72e8) `defaultSessionConfig` is sent as GPT-Live's `session.start` payload the moment the model connects. At minimum it needs the model, instructions (the system prompt), and `audio.format` set to `TWILIO_AUDIO_FORMAT_FOR_GPT_LIVE`. Setting `welcomeInstruction` makes the agent speak first instead of waiting for the caller, which is useful for any call where the caller doesn't know they've reached an AI agent yet. ### [Step 3: Wire up a tool call](#toc-heading-5835064a-a695-4871-a2e3-43cb803f9257) GPT-Live doesn't call functions directly – instead, [it delegates tool calls](https://developers.openai.com/api/docs/guides/live-delegation) either through Client Delegation (where you need to build the delegation integration) or through a Responses API model. In this tutorial, we’ll use [Responses delegation](https://developers.openai.com/api/docs/guides/live-delegation?delegation-mode=responses#configure-responses-delegation). Any tool created with `defineTool()` can be exposed this way: register it in `tools: [...]` so Twilio Agent Connect can execute it, and separately list its schema under `delegation.responses.tools` so the model knows it exists. Keeping tool execution *outside* the real-time voice loop means your tool logic can be as slow (or complex) as it needs to be. Consider all the things you might include in your logic: a database lookup, an API call, or a business rule. The 3-second `setTimeout` in the code above makes the strength of this model visible: run the demo and the call keeps going while the tool runs. GPT-Live delegates the decision to call a tool to a Responses API model, then Twilio Agent Connect runs the actual TypeScript function and speaks back the result. And hopefully, that shows how straightforward OpenAI’s Responses delegation is. Of course,when you build your own agent, swap `getWeather` for a function that hits your own backend. For example, you might look up an order status, check appointment availability, or pull account details. All the same wiring applies without changing anything else here. ### [Step 4: Wire up the server](#toc-heading-f5750e4c-7127-4285-b240-4a4378e70b4c) `TACServer` mounts the TwiML endpoint at `/twiml` by default. ### [Step 5: Point your Twilio number at the webhook](#toc-heading-2871d186-5f1d-456b-ac80-13c97d431480) With the server running behind a public tunnel (like ngrok), tell Twilio where to send incoming calls: - In the Twilio Console, go to **Phone Numbers > Manage > Active Numbers** and select your number. - Under Voice Configuration, set "A call comes in" to Webhook, and enter `https:///twiml` . - Set the HTTP method to **POST** , then save. Now, you're ready to call in! ☎️ *And there you have it:* you now have a phone number that greets callers, answers questions, and calls real functions, with no WebSocket code required. ## [Run, test, troubleshoot](#toc-heading-301badde-36ba-435a-b107-e9e51c98eb60) Call your Twilio number… if everything is set up correctly, you should hear the agent's `welcomeInstruction` immediately, without saying anything first. After the greeting, ask "what's the weather in Los Angeles?" — the model should invoke `get_weather` via Responses delegation and read back the result in the same turn. Have fun talking to your agent – and when you hang up, the transcript accumulated in `ConversationSession.metadata.transcript` prints to your console if you register an `onConversationEnded` callback that reads it off the session when the call ends. Here’s an example: If you don't hear a greeting, double check that `welcomeInstruction` is set. ## [Going further: placing outbound calls](#toc-heading-9f46f0ff-cbbc-4997-b4dc-24713eaeb861) Now that you have an agent callers can reach, let's build one that reaches out to them instead. ### [Place an outbound call](#place-an-outbound-call) The same `GPTLiveProvider` that answers inbound calls can also *place* them. An outbound agent is useful for proactive outreach like appointment reminders, order updates, or callback flows where the agent initiates contact instead of waiting for a caller to dial in. To complete the outbound agent demo, continue in the same file and reuse the `voiceChannel` you already built in Steps 1-4. to must be in [E.164 format](https://www.twilio.com/docs/glossary/what-e164) (e.g., +15551234567), which is what Twilio's Voice API requires. In the full example, this runs behind a --to flag. Start the server with npm run dev -- --to +15551234567 and it places the call as soon as the server starts, while still answering inbound calls as usual. That's the entire outbound flow. TAC handles placing the call, connecting the Media Stream, and bridging it to GPT-Live the same way it does for an inbound call. From here, initiateOutboundConversation is the one API you need whether you're calling one number or looping over a list to run a whole outbound campaign. ### Other features to explore This tutorial covers the basics, but Twilio Agent Connect and GPT-Live support more than what's shown here: - One SDK, every channel. [TAC also ships providers for SMS, RCS, WhatsApp, and Chat](https://www.twilio.com/docs/conversations/agent-connect/channels) , if your agent needs to talk to customers outside of voice. - Swappable voice backends. The same VoiceChannel API works with Conversation Relay and the [OpenAI Realtime API](https://openai.com/index/introducing-gpt-realtime/) . - Production-ready call handling. Status, answering-machine detection, and recording callbacks, plus programmatic hang-up, for turning a demo into something that survives real phone traffic. - Per-call session customization. Override the session config for a specific call without touching your channel-wide defaults. ## Conclusion Congratulations! You now have a phone number that connects callers to a GPT-Live-1-powered voice agent, whether by picking up when they place a call or by dialing outbound. You agent greets people proactively, and can answer more advanced questions by calling a tool. The same pattern I laid out in this post works for any scenario where a caller needs a tool-using voice agent. So whatever you’re building – from appointment booking to order status lookups – this example extends to your use case, and any per-caller prompts and other features when you need them. From here, check out the [Twilio Agent Connect API reference](https://twilio.github.io/twilio-agent-connect-typescript/) for the full VoiceProvider interface, or the OpenAI Realtime API provider if you need an alternative speech-to-speech backend. ### Additional Resources *Wen Zhu is a Software Engineer at Twilio. Away from the keyboard, Wen heads for the trails whenever the weather cooperates, is usually planning the next trip before the current one ends, and dials in shots on a home espresso machine with the kind of patience normally saved for chasing down flaky tests. Wen can be reached on [LinkedIn](https://www.linkedin.com/in/wen-zhu/).* *Xinghao Huang is a Software Engineer at Twilio. Off the clock, he cooks his way through both Chinese and Western cuisines, keeps a spice cabinet of a few dozen jars — all matching, naturally — better organized than most of his codebases, and maintains a personal recipe website. He can be reached at jahuang [at] twilio.com.* ## Related Posts ## Related Resources Twilio Docs From APIs to SDKs to sample apps API reference documentation, SDKs, helper libraries, quickstarts, and tutorials for your language and platform. Resource Center The latest ebooks, industry reports, and webinars Learn from customer engagement experts to improve your own communication. Ahoy Twilio's developer community hub Best practices, code samples, and inspiration to build communications and digital engagement experiences.