Many people on WhatsApp don't type. They hold the mic button and talk. If your AI agent only understands text, those users get nothing back.
In this tutorial you'll connect an AI agent to WhatsApp so that:
The agent can be anything that answers an HTTP POST: a LangGraph graph, a CrewAI flow, an n8n workflow or a ten-line FastAPI app. We'll use the last one so you can see every moving part, then swap in your real agent at the end.
This is about voice notes, the recorded messages WhatsApp users send with the mic button, not live WhatsApp calls.
The glue is agent-voice-bridge, a small open-source FastAPI gateway (Apache 2.0) we wrote for this. We looked for an existing project that combines all three: any agent over HTTP, both Meta and Twilio, and voice note in → voice note out. We didn't find one, so we built it.
WhatsApp user ──► Twilio ──► agent-voice-bridge ──► your agent
│ ▲ │
speech-to-text text-to-speech │
│ │ │
WhatsApp user ◄── Twilio ◄──────┴────┴──── reply_text ◄─┘
{"reply_text": "..."}.
Your agent never sees a Twilio or Meta payload. The same agent also works on Meta's WhatsApp Cloud API without changes.
cloudflared
This is the whole agent. It just echoes what it heard, which is perfect for checking the pipes.
from fastapi import FastAPI
app = FastAPI()
@app.post("/agent")
async def agent(msg: dict) -> dict:
return {"reply_text": f"You said ({msg['message_type']}): {msg['content']}"}
pip install fastapi uvicorn
uvicorn echo_agent:app --port 9000
The bridge will send it a body like this:
{
"sender_id": "whatsapp:+905551112233",
"message_type": "voice",
"content": "What time does the store open tomorrow?",
"channel": "twilio"
}
sender_id is stable per person, so use it as your conversation or session key.
In a second terminal:
ngrok http 8000
Copy the https://…ngrok-free.app URL. That's your public URL.
join two-words. Send that phrase from your WhatsApp to the sandbox number.
git clone https://github.com/denegra-ai/agent-voice-bridge.git
cd agent-voice-bridge
python3 scripts/configure.py
The setup wizard asks only what your choices need:
http://host.docker.internal:9000/agent (the bridge runs in Docker, your echo agent on your laptop)
Keys are typed hidden and checked against the real service before anything is saved. The wizard writes .env and prints the webhook URL you need next.
docker compose up -d --build
curl http://localhost:8000/health
On the same Twilio page, open Sandbox settings and set When a message comes in to:
https://<your-ngrok-url>/webhook/twilio
Method POST, then Save.
Send a text to the sandbox number. You get:
You said (text): hello
Now hold the mic button and say something. A few seconds later you get a voice note back that reads out "You said (voice): …" with your transcript.
Point AGENT_WEBHOOK_URL at anything that accepts the JSON above and returns reply_text. For example, a LangGraph graph with a checkpointer:
from fastapi import FastAPI
from my_agent import graph # your compiled LangGraph graph
app = FastAPI()
@app.post("/agent")
async def agent(msg: dict) -> dict:
result = await graph.ainvoke(
{"messages": [("user", msg["content"])]},
config={"configurable": {"thread_id": msg["sender_id"]}},
)
return {"reply_text": result["messages"][-1].content}
In n8n it's a Webhook node (respond using "Respond to Webhook") and a Respond to Webhook node that returns {"reply_text": "..."} at the end of your workflow.
If your agent needs a token, set AGENT_AUTH_TOKEN and the bridge sends it as Authorization: Bearer ….
X-Twilio-Signature and Meta's X-Hub-Signature-256 are checked. Message content is never logged. Generated audio sits at an unguessable URL and is deleted after an hour.STT_PROVIDER=deepgram or TTS_PROVIDER=elevenlabs in .env.
A note on what's been tested: we've run the full voice round trip on a real phone through the Twilio sandbox, once with Azure OpenAI for speech and once with Deepgram (speech-to-text) and ElevenLabs (text-to-speech). The plain OpenAI option uses the same code as Azure OpenAI but hasn't had its own phone test yet. Meta's Cloud API path is verified up to the signed webhook and the agent call; we haven't yet done a full round trip on a phone through Meta. If you try it, we'd love to hear how it goes in the issues.
The repo is here: https://github.com/denegra-ai/agent-voice-bridge. Stars, issues and PRs are welcome.
agent-voice-bridge is open source and backed by Denegra.