Anyone who has learned a foreign language knows the gap between a passed exam and real fluency with native speakers. Closing it takes practice with someone at native level, and not everyone has a native-speaking partner or the budget and time for endless lessons.
So I asked myself: what if an AI voice agent could be that partner? Someone to have casual conversations with, like a buddy who speaks at native level, never loses patience, and points out your mistakes when they matter.
I built it with ElevenLabs agents and Cloudflare Workers.
An ElevenLabs agent handles the voice, and a Cloudflare Worker (Hono) does everything else: it serves a small Vite + TypeScript page, stores data in D1, and exposes the endpoints the agent calls.
recall, which returns what happened in recent sessions, and store, which saves items from the conversation.
Memory. When I built this, ElevenLabs agents had no built-in memory across sessions, so I added it myself: the webhook stores each session, and the agent calls recall when a call starts. I load only the last two or three sessions. Pulling in more makes the agent sound robotic. Nobody remembers a hundred earlier conversations with a friend.
No secrets in the browser. The Worker puts the user's identity into the page before it reaches the browser, as an HMAC-signed ID. The signing key only exists in the Worker.
const rewritten = new HTMLRewriter()
.on('elevenlabs-convai', {
element(el) {
el.setAttribute(
'dynamic-variables',
JSON.stringify({ user_id: token, name: displayName }),
);
},
})
.transform(response);
rewritten.headers.set('Cache-Control', 'no-store');
The no-store header matters, because this page is personalized and must never be served from a shared cache.
The agent's endpoints are not public. The agent never decides whose data it touches. The widget passes the signed user ID to the agent as a dynamic variable, and every tool call sends it back. The tool routes verify the signature before reading or writing anything, so an invented or altered ID is rejected. The post-call webhook has its own check. Route wiring, simplified:
app.get('/tools/recall', recallHandler); // verifies the signed user_id first
app.post('/tools/store', storeHandler); // same check before any write
app.post('/api/webhook', saveSession); // webhook signature check
The browser sees the signed ID but can't create a new one.
Routing. Static files come straight from the asset bundle. Only /, /api/* and /tools/* invoke the Worker.
If you're building voice agents and want to know more about my agent - drop a comment, I'm happy to share details.