Let any AI agent answer WhatsApp voice notes in 10 minutes

A developer at denegra-ai built agent-voice-bridge, an open-source Apache 2.0 FastAPI gateway that connects any HTTP-based AI agent to WhatsApp voice notes, handling speech-to-text and text-to-speech so agents receive plain JSON and reply with text. The bridge supports both Twilio and Meta's WhatsApp Cloud API, and the writeup walks through a Docker-based setup with ngrok, a configure wizard, and a ten-line echo agent for testing.

Many people on WhatsApp don't type. They hold the mic button and talk. If your AI agent only understands text, those users get nothing back. In this tutorial you'll connect an AI agent to WhatsApp so that: The agent can be anything that answers an HTTP POST: a LangGraph graph, a CrewAI flow, an n8n workflow or a ten-line FastAPI app. We'll use the last one so you can see every moving part, then swap in your real agent at the end. This is about voice notes, the recorded messages WhatsApp users send with the mic button, not live WhatsApp calls. The glue is agent-voice-bridge https://github.com/denegra-ai/agent-voice-bridge , a small open-source FastAPI gateway Apache 2.0 we wrote for this. We looked for an existing project that combines all three: any agent over HTTP, both Meta and Twilio, and voice note in → voice note out. We didn't find one, so we built it. WhatsApp user ──► Twilio ──► agent-voice-bridge ──► your agent │ ▲ │ speech-to-text text-to-speech │ │ │ │ WhatsApp user ◄── Twilio ◄──────┴────┴──── reply text ◄─┘ {"reply text": "..."} . Your agent never sees a Twilio or Meta payload. The same agent also works on Meta's WhatsApp Cloud API without changes. cloudflared This is the whole agent. It just echoes what it heard, which is perfect for checking the pipes. python echo agent.py from fastapi import FastAPI app = FastAPI @app.post "/agent" async def agent msg: dict - dict: return {"reply text": f"You said {msg 'message type' } : {msg 'content' }"} pip install fastapi uvicorn uvicorn echo agent:app --port 9000 The bridge will send it a body like this: { "sender id": "whatsapp:+905551112233", "message type": "voice", "content": "What time does the store open tomorrow?", "channel": "twilio" } sender id is stable per person, so use it as your conversation or session key. In a second terminal: ngrok http 8000 Copy the https://…ngrok-free.app URL. That's your public URL. join two-words . Send that phrase from your WhatsApp to the sandbox number. git clone https://github.com/denegra-ai/agent-voice-bridge.git cd agent-voice-bridge python3 scripts/configure.py The setup wizard asks only what your choices need: http://host.docker.internal:9000/agent the bridge runs in Docker, your echo agent on your laptop Keys are typed hidden and checked against the real service before anything is saved. The wizard writes .env and prints the webhook URL you need next. docker compose up -d --build curl http://localhost:8000/health {"status":"ok","version":"0.1.1","channels": "twilio" } On the same Twilio page, open Sandbox settings and set When a message comes in to: https://<your-ngrok-url /webhook/twilio Method POST , then Save . Send a text to the sandbox number. You get: You said text : hello Now hold the mic button and say something. A few seconds later you get a voice note back that reads out "You said voice : …" with your transcript. Point AGENT WEBHOOK URL at anything that accepts the JSON above and returns reply text . For example, a LangGraph graph with a checkpointer: python from fastapi import FastAPI from my agent import graph your compiled LangGraph graph app = FastAPI @app.post "/agent" async def agent msg: dict - dict: result = await graph.ainvoke {"messages": "user", msg "content" }, config={"configurable": {"thread id": msg "sender id" }}, return {"reply text": result "messages" -1 .content} In n8n it's a Webhook node respond using "Respond to Webhook" and a Respond to Webhook node that returns {"reply text": "..."} at the end of your workflow. If your agent needs a token, set AGENT AUTH TOKEN and the bridge sends it as Authorization: Bearer … . X-Twilio-Signature and Meta's X-Hub-Signature-256 are checked. Message content is never logged. Generated audio sits at an unguessable URL and is deleted after an hour. STT PROVIDER=deepgram or TTS PROVIDER=elevenlabs in .env . A note on what's been tested: we've run the full voice round trip on a real phone through the Twilio sandbox, once with Azure OpenAI for speech and once with Deepgram speech-to-text and ElevenLabs text-to-speech . The plain OpenAI option uses the same code as Azure OpenAI but hasn't had its own phone test yet. Meta's Cloud API path is verified up to the signed webhook and the agent call; we haven't yet done a full round trip on a phone through Meta. If you try it, we'd love to hear how it goes in the issues https://github.com/denegra-ai/agent-voice-bridge/issues . The repo is here: https://github.com/denegra-ai/agent-voice-bridge https://github.com/denegra-ai/agent-voice-bridge . Stars, issues and PRs are welcome. agent-voice-bridge is open source and backed by Denegra https://denegra.ai .