I Built a Voice-First AI Tutor for Bharat in 10 Days 🇮🇳 — Here’s My Complete Journey
Over the past 10 days, I participated in the 10 Days of Voice Agents challenge hosted by Murf AI. I built Vidya Vani, an intelligent, low-latency, multi-agent voice tutor that helps users practice spoken English and Mathematics. It features dynamic LLM question generation, memory retention across sessions, live analytics, and seamless agent handoffs—all powered by the blazing-fast Murf Falcon TTS and LiveKit WebRTC.
This is the full story of why I built it, the architecture that powers it, the intense roadblocks I hit, and how you can build one too!
The Problem: The Education Gap in Bharat
India is a country of incredible diversity, but when it comes to foundational education—specifically English literacy and Mathematics—there is a massive accessibility gap. Quality education is often concentrated in urban hubs, leaving learners in rural and semi-urban areas without access to dedicated, patient tutors for 1-on-1 practice.
While there are plenty of ed-tech apps and text-based AI chatbots available, they all suffer from the same fundamental flaw for foundational learners: friction.
Practicing spoken English with a text-based chatbot is intimidating. It requires spelling proficiency, typing speed, and it does absolutely nothing to help with conversational confidence or pronunciation.
The Solution: We needed a voice-first approach. By leveraging voice, we entirely remove the friction of typing and screen-staring. Users simply speak to their phone or computer, making the interaction as natural, accessible, and human as talking to a real teacher.
Meet Vidya Vani & Aryabhata
I set out to build a 24/7 educational voice tutor for the Learning & Literacy track of the challenge. But as the days progressed, I realized a single AI prompt trying to act as a master of all subjects was prone to hallucinations and confusion. So, I split the persona into two distinct experts.
English Practice: She helps users practice English vocabulary and grammar by dynamically generating unique exercises on the fly.
Persistent Memory: She remembers the user's name and past progress across sessions using a local SQLite database.
Human Escalation: If a user becomes frustrated, confused, or asks for a real human, she autonomously logs an escalation ticketing request for a human teacher to follow up.
Under the Hood: The Architecture & Tech Stack
Building a real-time conversational AI is completely different from building a standard web app. You aren't just sending a REST request and waiting for a JSON response; you are managing real-time, bi-directional audio streams.
Here is the tech stack that made it possible to achieve ultra-low latency:
Transport Layer (LiveKit WebRTC): WebRTC is the backbone of real-time audio. LiveKit's open-source infrastructure handles the complex networking, ensuring that the audio streaming between the user's browser and my Python backend server is seamless, even on less-than-ideal Indian networks.
Speech-to-Text (Deepgram): Before the AI can think, it needs to hear. Deepgram is lightning fast at transcribing the user's spoken audio into text, and it handles various Indian accents and background noises brilliantly.
The Brain (OpenAI/Anthropic LLMs): The LLM acts as the orchestrator. It evaluates the user's answers, generates dynamic questions based on context, and decides when to trigger specific Python tools.
Text-to-Speech (Murf Falcon): This was the absolute star of the show! Murf Falcon TTS generates expressive, ultra-fast Indian voices in real-time. By utilizing voices like "Pooja" (for Vidya Vani) and "Samir" (for Aryabhata), the interaction felt incredibly native and comfortable for an Indian audience.
Deep Dive: The Coolest Features
Multi-Agent Orchestration
As mentioned, having one AI do everything is a recipe for disaster. By utilizing LiveKit's experimental multi-agent capabilities, I was able to build a system where the AI hands off the context to another agent. This keeps system prompts clean, concise, and highly specialized. Aryabhata doesn't need to know how to teach English grammar, and Vidya Vani doesn't need to know how to grade arithmetic!
Dynamic Content Generation vs. Static Banks
Initially, I thought about a JSON file with 100 math problems. But that gets boring quickly. Instead, I prompted the LLMs to act as dynamic content generators. Instead of a hardcoded list of questions, the agent dynamically generates custom word problems based on what the user needs to practice right at that moment. No two study sessions are ever the same!
Live Analytics & Real-Time Dashboarding
What good is an educational tool if teachers can't track progress? I built a Next.js frontend with Recharts that displays live call outcomes. Every time an agent grades an exercise (right or wrong), the LLM triggers a hidden Python function tool. This tool writes the score to the database, and the frontend dashboard updates in real-time so administrators can track success vs. failure rates across the platform.
Persistent Memory state
Voice agents usually suffer from "Goldfish Syndrome"—they forget who you are the second you hang up. By utilizing a local SQLite database and instructing the agent to trigger a lookup_caller_memory tool at the start of a call, Vidya Vani greets returning learners by their first name and asks if they want to continue where they left off.
The "Pulling My Hair Out" Moments (Challenges)
Building real-time voice agents forces you to unlearn everything you know about text-based chatbots. You have to actively manage "turn-taking", speech interruptions, Voice Activity Detection (VAD), and latency. Here are the biggest hurdles I faced:
Challenge 1: The Silent Handoff Bug
During the multi-agent handoff, I wrote a prompt instructing the main agent to say "I will transfer you" and then call the transfer function tool in the same turn. What happened: The LLM would often just generate the spoken text and "forget" to execute the hidden tool call. The user would hear "I will transfer you..." and then sit in absolute silence forever! The Fix: I updated the tool description to explicitly enforce the agent to do both simultaneously. Furthermore, when the new agent (Aryabhata) took over, he would wait silently for the user to speak first. I had to write a script in the backend to auto-trigger Aryabhata's greeting 1.5 seconds after the handoff was complete!
**Challenge 2: The Overly Polite Specialist **
After Aryabhata graded a math problem and logged the score via a tool call, he would often go completely silent, assuming his turn was over and waiting for the user to prompt him again. The Fix: I had to explicitly write CRITICAL rules in the System Prompt: “After evaluating an answer and calling a tool, you MUST explicitly ask the user if they want to try another problem and wait for their confirmation! Do not stay silent.” In Voice AI, the agent must explicitly manage the conversational flow.
Challenge 3: Balancing Latency and Intelligence
The faster the AI responds, the more natural the conversation feels. However, complex system prompts and heavy tool-calling slow down the LLM's Time-To-First-Token (TTFT). The Fix: I moved as much logic as possible out of the LLM prompt and into the Python backend. By letting the LLM focus solely on the conversation and using lightweight tools for database lookups and routing, combined with Murf Falcon's insane TTS speed, the latency dropped to a level where interruptions felt natural.
Show Me The Code! (How to Build Your Own)
Want to build your own real-time voice AI? You don't need a supercomputer—just a decent internet connection and some API keys. Here is how you can set up my project locally!
1. Clone the Repository & Set Up the Backend: Ensure you have Python installed. We use uv for lightning-fast dependency management (if you haven't used uv yet, you are missing out!).
git clone [Your GitHub Repo URL]
cd backend
uv sync
2. Environment Variables: Never hardcode your API keys! Create a .env file in the backend and frontend directories based on the provided .env.example files. You will need API keys for:
LiveKit Cloud (For WebRTC Transport)
OpenAI (For the LLM Brain)
Deepgram (For STT)
Murf AI (For Falcon TTS)
3. Run the Project: Start the backend agent server first:
uv run python src/agent.py dev
Then, in a new terminal, spin up the Next.js frontend to see the UI and analytics dashboard:
cd frontend
npm run dev
Code Highlight: The Multi-Agent Handoff
Here is a simplified snippet of how the main agent transfers the live call to the Math Specialist using LiveKit's function tools. Notice how we use asyncio.create_task to force the new agent to speak immediately!
**@function_tool(description="Call this tool to hand off the conversation to the Maths Practice Specialist when the user asks for Math.")
async def transfer_to_maths_specialist(self, context: RunContext) -> Agent:
maths_agent = MathsSpecialist(
chat_ctx=self.chat_ctx.copy(exclude_instructions=True),
parent_agent=self
)
async def trigger_intro():
await asyncio.sleep(1.5)
if hasattr(self, "session") and self.session:
await self.session.generate_reply(
instructions="Introduce yourself as Aryabhata the Maths Specialist and IMMEDIATELY ask the first beginner math problem."
)
import asyncio
asyncio.create_task(trigger_intro())
return maths_agent**
What's Next for Vidya Vani?
Ten days is a short amount of time, and there is so much more I want to build. If I had another month, I would love to add:
True Multilingual Support: Allow users to seamlessly switch between Hindi, Marathi, Bengali, and English mid-sentence, with the agent dynamically responding in the same language.
Gamification: Add a persistent points, streaks, and badges system to the frontend analytics dashboard to keep young learners motivated to return every day!
SMS Integration: Send a text message summary of the day's lesson to the parents after the call ends.
🔗 Links
GitHub Repository: https://github.com/satwik146/voice-for-Bharat