Building Ava: A Real-Time AI Voice English Learning Assistant with Murf Falcon & LiveKit
10 Days of Voice Agents โ VoiceForBharat Edition
Over the past 10 days, I built Ava, an AI-powered voice English learning assistant designed to help learners practice English through natural, real-time conversations.
The project started as a simple voice agent and gradually evolved into a complete voice AI system with memory, live tools, outbound calling, human escalation, call analytics, and specialist-agent handoffs.
The biggest goal was not just to make an AI that can talk, but to build an agent that can listen, understand, remember, take actions, and know when it needs help from a human or another specialist agent.
What is Ava?
Ava is a conversational AI voice assistant designed primarily for English-learning practice.
A learner can talk to Ava naturally instead of typing messages. Ava listens to the learner's voice, understands the conversation, generates a response, and speaks it back in real time.
The core voice pipeline is:
๐ค User Voice โ Deepgram STT โ Gemini LLM โ Murf Falcon TTS โ ๐ User
The real-time communication layer is handled using LiveKit.
๐ก Why Voice?
Learning English is not only about reading and writing. Speaking and listening are equally important.
A voice-based assistant makes it possible for learners to practice:
๐ฃ๏ธ Speaking English
๐ Listening comprehension
๐ฌ Natural conversations
๐ Vocabulary
โ๏ธ Common mistakes
๐ฏ Confidence while speaking
Instead of interacting with a chatbot through text, learners can have a more natural conversation with Ava.
Day 1 โ Voice Agent Foundation ๐ค
I started by setting up the basic voice agent using LiveKit.
The initial goal was simple:
Listen to the user โ process the speech โ generate an answer โ speak the answer.
This created the foundation for the rest of the project.
Day 2 โ Personality & Guardrails ๐ง
I gave Ava a clear personality and behavior.
Ava was designed to be:
Friendly
Short and conversational
Respectful
Helpful
Focused on English learning
I also added safety guardrails so Ava doesn't assist with things like hacking, fraud, password theft, OTPs, or other unsafe requests.
Day 3 โ Frontend Experience ๐ป
I customized the frontend to make the voice-agent experience easier to understand.
The interface provides feedback about the current state of the voice conversation and connects the user with the AI agent.
This helped turn the project from a backend experiment into an actual user-facing application.
Day 4 โ Memory ๐ง ๐พ
One of the most interesting features was adding persistent memory.
Ava can remember useful information about returning users, such as:
Name
Language preference
Current learning level
Topics covered
Common mistakes
But there is an important privacy rule:
Ava asks for permission before saving useful personal information.
This made the memory system more user-controlled instead of automatically storing everything.
Day 5 โ Live Tools ๐ฆ๏ธ
I added a live weather tool.
For example, a user can ask:
"What's the weather in Bhopal?"
Instead of guessing, Ava calls a weather API, retrieves current information, and explains the result naturally.
This was an important step because the agent could now take an action instead of only generating text.
Day 6 โ Outbound Voice Calls ๐
Next, I extended the project to support outbound calling.
The agent could initiate a call and interact with a learner through a phone/SIP-based workflow.
I also added rules for outbound conversations:
Identify itself as an AI
Explain why it is calling
Ask whether the learner has time
Respect "stop" requests
End the call if the learner is busy
Never pressure the learner
This introduced another important lesson:
Voice agents need to respect the user's time and consent.
Day 7 โ Human Escalation ๐จ๐ซ
Ava doesn't try to solve everything.
If a learner becomes confused, frustrated, or explicitly asks for a human teacher, Ava can offer to escalate the issue.
Before creating an escalation request, Ava asks for permission.
The request can contain information such as:
User name
Problem
What Ava already tried
Urgency
Language
Preferred follow-up method
A reference ID is then generated for the request.
Day 8 โ Call Analytics ๐
I built a call analytics system to track conversations.
The dashboard can provide information such as:
Total calls
Successful calls
Failed calls
Pending calls
Success rate
Call duration
This helped me understand that building a voice agent isn't only about the conversation itself.
Observability and analytics are also important parts of a real voice AI application.
Day 9 โ Specialist Agent Handoff ๐
I then added a specialist-agent architecture.
Ava acts as the main assistant.
When a user asks something related to call analytics, Ava can transfer the conversation to a Call Analytics Specialist.
For example:
User: "How many calls were successful?"
Ava can respond:
"I'll connect you to our call analytics specialist."
The specialist then takes over the conversation.
This introduced the idea of multi-agent collaboration.
Day 10 โ Share & Reflect ๐
The final day was about documenting the project, sharing what I learned, and making the project useful for other developers.
The biggest lesson from these 10 days was:
A voice agent is much more than an LLM that can speak.
It needs speech recognition, reasoning, voice synthesis, real-time communication, memory, tools, safety, observability, and reliable handoffs.
๐๏ธ Architecture
The overall system looks like this:
โโโโโโโโโโโโโโโโโโโโ
โ User โ
โ Voice / Browser โ
โโโโโโโโโโฌโโโโโโโโโโ
โ
โผ
โโโโโโโโโโโโโโโโโโโโ
โ LiveKit โ
โ Real-time Audio โ
โโโโโโโโโโฌโโโโโโโโโโ
โ
โผ
โโโโโโโโโโโโโโโโโโโโ
โ Deepgram โ
โ STT โ
โโโโโโโโโโฌโโโโโโโโโโ
โ
โผ
โโโโโโโโโโโโโโโโโโโโ
โ Gemini LLM โ
โ Reasoning / Chat โ
โโโโโโโโโโฌโโโโโโโโโโ
โ
โโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโ
โผ โผ โผ
โโโโโโโโโโโโ โโโโโโโโโโโโ โโโโโโโโโโโโโโ
โ Memory โ โ Tools โ โ Specialist โ
โ Database โ โ Weather โ โ Agent โ
โโโโโโโโโโโโ โโโโโโโโโโโโ โโโโโโโโโโโโโโ
โ
โผ
โโโโโโโโโโโโโโโโโโโโ
โ Murf Falcon โ
โ TTS โ
โโโโโโโโโโฌโโโโโโโโโโ
โ
โผ
๐ Voice Response
โ๏ธ Tech Stack
AI & Voice
Murf Falcon โ Text-to-Speech
Deepgram โ Speech-to-Text
Google Gemini โ LLM
Silero VAD โ Voice Activity Detection
Real-Time
LiveKit
LiveKit Agents
Backend
Python
SQLite
HTTPX
Flask
Frontend
Next.js
React
TypeScript
๐ Challenges I Faced
The project definitely did not work perfectly on the first attempt.
I encountered several real-world problems during development:
๐ GitHub Secret Scanning
GitHub blocked one of my pushes because an API key was detected inside a commit.
I learned that simply changing the current file isn't enough if the secret still exists in Git history.
The solution was to remove the secret from the affected commits/history and keep only safe placeholders such as:
GOOGLE_API_KEY=your_google_api_key
๐ LiveKit Connection Problems
I also encountered LiveKit connection problems such as:
RegionError("region fetch timed out")
and:
signal connection timed out
At one point, DNS resolution itself was failing.
Testing with commands such as:
nslookup murf-voice-agent-g91x9izl.livekit.cloud
curl -I https://murf-voice-agent-g91x9izl.livekit.cloud
helped identify whether the problem was in my application or the network connection.
๐๏ธ Deepgram Connection Timeout
Another issue was:
Deepgram did not receive audio data or a text message
within the timeout window
This taught me that real-time voice systems are highly dependent on stable network connections and continuous audio flow.
๐ Voice Interruption & Latency
I also worked through issues where Ava's voice would break or conversations would feel less smooth.
This required looking at the complete pipeline:
Microphone โ LiveKit โ STT โ LLM โ TTS โ LiveKit โ Speaker
rather than assuming the problem was only in the TTS.
๐ Security
One of the most important lessons was never exposing API keys.
Keys should stay inside environment variables:
.env.local
and should never be committed to GitHub.
The repository should contain only safe examples:
GOOGLE_API_KEY=your_google_api_key
MURF_API_KEY=your_murf_api_key
DEEPGRAM_API_KEY=your_deepgram_api_key
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret
๐ GitHub Repository
๐ Source Code:
[Add your GitHub repository link here]
Make sure the public repository contains no API keys, phone numbers, caller information, database files, or other private data.
๐ฎ What I Want to Build Next
There are several things I'd like to improve:
Better multilingual support
More natural conversation handling
Improved interruption handling
Better voice latency
More learning-focused tools
Advanced learner analytics
Better specialist-agent routing
Production-grade monitoring
More robust phone-call infrastructure
๐ฏ Final Takeaway
The biggest thing I learned during these 10 days is that building a voice agent is not just about connecting an LLM to a TTS model.
A useful voice agent needs to:
Listen โ Understand โ Reason โ Act โ Speak โ Remember โ Escalate โ Improve
Building Ava gave me hands-on experience with the complete voice AI pipeline and taught me a lot about real-time systems, APIs, databases, AI tools, agent architecture, debugging, and production challenges.
I'm proud to have completed:
๐ 10 Days of Voice Agents โ VoiceForBharat Edition
Built with Murf Falcon + LiveKit + Gemini + Deepgram.
A huge learning experience, and definitely just the beginning! ๐๏ธ๐ค