ElevenLabs + Twilio: Building Voice Call Applications

A developer has published a step-by-step guide for building an AI-powered voice call application that combines Twilio's telephony webhooks with ElevenLabs' text-to-speech API and OpenAI's GPT-4o-mini for conversational responses. The Flask-based example routes inbound calls through Twilio's speech recognition, generates a reply via ChatGPT, and plays back ElevenLabs-generated audio, with ngrok used to expose the local server to Twilio's webhook.

If you’ve ever wanted to turn a simple phone call into an interactive, AI‑powered experience, you’re in the right place. Twilio gives you the plumbing to make and receive voice calls, while ElevenLabs provides state‑of‑the‑art text‑to‑speech TTS and voice cloning. Put them together and you can build everything from personalized voicemail assistants to real‑time language translation over the phone. In this post we’ll walk through a minimal but fully functional example: <Gather By the end you’ll have a reusable Flask endpoint that you can deploy on any cloud provider. Pro tip: If you need a high‑quality voice for your brand, ElevenLabs’ voice cloning lets you upload a few minutes of audio and get a custom voice that sounds like a real person. Check it out here: https://try.elevenlabs.io/kr07zfuqn1bp https://try.elevenlabs.io/kr07zfuqn1bp | What you need | Why | |---|---| | Twilio account | To get a phone number and expose a webhook | | ElevenLabs API key | To call the TTS endpoint | | OpenAI API key optional | For speech‑to‑text and conversational AI | | Python 3.9+ | The example uses Flask | | ngrok or any public URL | To expose your local server to Twilio | Install the required Python packages: pip install flask twilio requests openai Create a new phone number in the Twilio console and point its Voice & Fax → A CALL COMES IN webhook to https://<your‑public‑url /voice . When Twilio receives a call it will POST an XML document Twiml that tells it what to do next. python app.py from flask import Flask, request, Response from twilio.twiml.voice response import VoiceResponse, Gather app = Flask name @app.route "/voice", methods= "POST" def voice : resp = VoiceResponse Ask the caller to speak after the beep gather = Gather input="speech", action="/process", method="POST", timeout=5 gather.say "Hi Tell me what you need help with." resp.append gather If no speech was captured, fallback resp.say "Sorry, I didn't catch that. Goodbye " resp.hangup return Response str resp , mimetype="application/xml" Run the app locally and expose it with ngrok: python app.py ngrok http 5000 Take the HTTPS ngrok URL e.g., https://abcd1234.ngrok.io and paste it into the Twilio webhook field. When the caller finishes speaking, Twilio sends a POST to /process with a SpeechResult field that already contains a transcription thanks to Twilio’s built‑in speech recognition . If you want higher accuracy or multiple languages, you can pipe the raw audio to Whisper instead, but for this demo we’ll use Twilio’s result directly. python import os import openai import requests ELEVEN API KEY = os.getenv "ELEVEN API KEY" OPENAI API KEY = os.getenv "OPENAI API KEY" ELEVEN TTS URL = "https://api.elevenlabs.io/v1/text-to-speech" openai.api key = OPENAI API KEY @app.route "/process", methods= "POST" def process : Grab what Twilio recognized user text = request.form.get "SpeechResult", "" if not user text: return fallback response "I didn't hear anything." Generate a ChatGPT response reply = chatgpt reply user text Convert reply to audio with ElevenLabs audio url = eleven tts reply Build TwiML to play the audio resp = VoiceResponse resp.play audio url resp.hangup return Response str resp , mimetype="application/xml" php def chatgpt reply prompt: str - str: completion = openai.ChatCompletion.create model="gpt-4o-mini", messages= {"role": "user", "content": prompt} , temperature=0.7, return completion.choices 0 .message.content.strip ElevenLabs’ API accepts plain text and returns a short‑lived URL to an MP3 file. The endpoint we’ll use is /v1/text-to-speech/{voice id} . For most developers the “default” voice EXAVITQu4vr4xnSDxMaL works great, but you can replace it with a cloned voice ID if you’ve uploaded your own samples. php def eleven tts text: str - str: voice id = "EXAVITQu4vr4xnSDxMaL" default voice headers = { "xi-api-key": ELEVEN API KEY, "Content-Type": "application/json" } payload = { "text": text, "model id": "eleven monolingual v1", "voice settings": { "stability": 0.75, "similarity boost": 0.85 } } response = requests.post f"{ELEVEN TTS URL}/{voice id}", json=payload, headers=headers, timeout=15, response.raise for status The API returns raw audio bytes; we upload to a temporary storage service For simplicity, we’ll use ElevenLabs’ own temporary URL feature: return response.json "audio url" Note: If you prefer not to host the MP3 yourself, ElevenLabs can give you a temporary URL that Twilio can stream directly, as shown above. Your final app.py should look roughly like this: python import os from flask import Flask, request, Response from twilio.twiml.voice response import VoiceResponse, Gather import openai import requests app = Flask name ELEVEN API KEY = os.getenv "ELEVEN API KEY" OPENAI API KEY = os.getenv "OPENAI API KEY" ELEVEN TTS URL = "https://api.elevenlabs.io/v1/text-to-speech" openai.api key = OPENAI API KEY def chatgpt reply prompt: str - str: completion = openai.ChatCompletion.create model="gpt-4o-mini", messages= {"role": "user", "content": prompt} , temperature=0.7, return completion.choices 0 .message.content.strip def eleven tts text: str - str: voice id = "EXAVITQu4vr4xnSDxMaL" headers = { "xi-api-key": ELEVEN API KEY, "Content-Type": "application/json" } payload = { "text": text, "model id": "eleven monolingual v1", "voice settings": {"stability": 0.75, "similarity boost": 0.85} } resp = requests.post f"{ELEVEN TTS URL}/{voice id}", json=payload, headers=headers resp.raise for status return resp.json "audio url" def fallback response message: str : resp = VoiceResponse resp.say message resp.hangup return Response str resp , mimetype="application/xml" @app.route "/voice", methods= "POST" def voice : resp = VoiceResponse gather = Gather input="speech", action="/process", method="POST", timeout=5 gather.say "Hey there What can I help you with today?" resp.append gather resp.say "Sorry, I didn't hear anything. Bye " resp.hangup return Response str resp , mimetype="application/xml" @app.route "/process", methods= "POST" def process : user text = request.form.get "SpeechResult", "" if not user text: return fallback response "I didn't catch that." reply = chatgpt reply user text audio url = eleven tts reply resp = VoiceResponse resp.play audio url resp.hangup return Response str resp , mimetype="application/xml" if name == " main ": app.run debug=True, port=5000 Deploy the script to your favorite host Heroku, Fly.io, Render, etc. , point the Twilio webhook at the live URL, and you’re ready to make calls that sound like a real person—powered by ElevenLabs. | Feature | How to add it | |---|---| | Voice cloning | Upload a few minutes of your own voice to ElevenLabs, grab the returned voice id , and replace the default voice id in eleven tts . | | Multi‑language support | Use Whisper or Azure Speech‑to‑Text for transcription, then set model id to eleven multilingual v2 when calling ElevenLabs. | | Persisted conversation | Store the conversation id from OpenAI and pass it back on each turn to maintain context. | | Call recording | Add <Record in the TwiML to capture the whole interaction for compliance or analytics. | | Interactive menus | Use multiple <Gather blocks and DTMF input="dtmf" to build IVR trees. | Building a voice‑first AI app used to require a deep dive into DSP libraries and self‑hosted TTS engines. With Twilio handling the telephony plumbing and ElevenLabs delivering crystal‑clear, human‑like speech, the barrier to entry is now a few lines of code. Ready to give your callers a voice that sounds truly human? Grab an API key from ElevenLabs and start experimenting today: 👉 Try ElevenLabs now: https://try.elevenlabs.io/kr07zfuqn1bp https://try.elevenlabs.io/kr07zfuqn1bp Happy coding, and may your next voice app sound as good as it feels