{"slug": "10-days-to-build-a-voice-ai-tutor-the-good-the-bad-and-the-why-is-it-silent", "title": "10 Days to Build a Voice AI Tutor: The Good, The Bad, and The \"Why Is It Silent?!\"", "summary": "A developer built Vidya Vani, a voice-first AI tutor for India, in 10 days as part of Murf AI's 10 Days of Voice Agents challenge. The multi-agent system uses LiveKit WebRTC, Deepgram, OpenAI/Anthropic LLMs, and Murf Falcon TTS to provide low-latency spoken English and math practice, with features like dynamic question generation and persistent memory.", "body_md": "**I Built a Voice-First AI Tutor for Bharat in 10 Days 🇮🇳 — Here’s My Complete Journey**\n\nOver the past 10 days, I participated in the 10 Days of Voice Agents challenge hosted by Murf AI. I built Vidya Vani, an intelligent, low-latency, multi-agent voice tutor that helps users practice spoken English and Mathematics. It features dynamic LLM question generation, memory retention across sessions, live analytics, and seamless agent handoffs—all powered by the blazing-fast Murf Falcon TTS and LiveKit WebRTC.\n\nThis is the full story of why I built it, the architecture that powers it, the intense roadblocks I hit, and how you can build one too!\n\nThe Problem: The Education Gap in Bharat\n\nIndia is a country of incredible diversity, but when it comes to foundational education—specifically English literacy and Mathematics—there is a massive accessibility gap. Quality education is often concentrated in urban hubs, leaving learners in rural and semi-urban areas without access to dedicated, patient tutors for 1-on-1 practice.\n\nWhile there are plenty of ed-tech apps and text-based AI chatbots available, they all suffer from the same fundamental flaw for foundational learners: friction.\n\nPracticing spoken English with a text-based chatbot is intimidating. It requires spelling proficiency, typing speed, and it does absolutely nothing to help with conversational confidence or pronunciation.\n\nThe Solution: We needed a voice-first approach. By leveraging voice, we entirely remove the friction of typing and screen-staring. Users simply speak to their phone or computer, making the interaction as natural, accessible, and human as talking to a real teacher.\n\nMeet Vidya Vani & Aryabhata\n\nI set out to build a 24/7 educational voice tutor for the Learning & Literacy track of the challenge. But as the days progressed, I realized a single AI prompt trying to act as a master of all subjects was prone to hallucinations and confusion. So, I split the persona into two distinct experts.\n\nEnglish Practice: She helps users practice English vocabulary and grammar by dynamically generating unique exercises on the fly.\n\nPersistent Memory: She remembers the user's name and past progress across sessions using a local SQLite database.\n\nHuman Escalation: If a user becomes frustrated, confused, or asks for a real human, she autonomously logs an escalation ticketing request for a human teacher to follow up.\n\n**Under the Hood: The Architecture & Tech Stack**\n\nBuilding a real-time conversational AI is completely different from building a standard web app. You aren't just sending a REST request and waiting for a JSON response; you are managing real-time, bi-directional audio streams.\n\nHere is the tech stack that made it possible to achieve ultra-low latency:\n\nTransport Layer (LiveKit WebRTC): WebRTC is the backbone of real-time audio. LiveKit's open-source infrastructure handles the complex networking, ensuring that the audio streaming between the user's browser and my Python backend server is seamless, even on less-than-ideal Indian networks.\n\nSpeech-to-Text (Deepgram): Before the AI can think, it needs to hear. Deepgram is lightning fast at transcribing the user's spoken audio into text, and it handles various Indian accents and background noises brilliantly.\n\nThe Brain (OpenAI/Anthropic LLMs): The LLM acts as the orchestrator. It evaluates the user's answers, generates dynamic questions based on context, and decides when to trigger specific Python tools.\n\nText-to-Speech (Murf Falcon): This was the absolute star of the show! Murf Falcon TTS generates expressive, ultra-fast Indian voices in real-time. By utilizing voices like \"Pooja\" (for Vidya Vani) and \"Samir\" (for Aryabhata), the interaction felt incredibly native and comfortable for an Indian audience.\n\n**Deep Dive: The Coolest Features**\n\nMulti-Agent Orchestration\n\nAs mentioned, having one AI do everything is a recipe for disaster. By utilizing LiveKit's experimental multi-agent capabilities, I was able to build a system where the AI hands off the context to another agent. This keeps system prompts clean, concise, and highly specialized. Aryabhata doesn't need to know how to teach English grammar, and Vidya Vani doesn't need to know how to grade arithmetic!\n\nDynamic Content Generation vs. Static Banks\n\nInitially, I thought about loading a JSON file with 100 math problems. But that gets boring quickly. Instead, I prompted the LLMs to act as dynamic content generators. Instead of a hardcoded list of questions, the agent dynamically generates custom word problems based on what the user needs to practice right at that moment. No two study sessions are ever the same!\n\nLive Analytics & Real-Time Dashboarding\n\nWhat good is an educational tool if teachers can't track progress? I built a Next.js frontend with Recharts that displays live call outcomes. Every time an agent grades an exercise (right or wrong), the LLM triggers a hidden Python function tool. This tool writes the score to the database, and the frontend dashboard updates in real-time so administrators can track success vs. failure rates across the platform.\n\nPersistent Memory state\n\nVoice agents usually suffer from \"Goldfish Syndrome\"—they forget who you are the second you hang up. By utilizing a local SQLite database and instructing the agent to trigger a lookup_caller_memory tool at the start of a call, Vidya Vani greets returning learners by their first name and asks if they want to continue where they left off.\n\n**The \"Pulling My Hair Out\" Moments (Challenges)**\n\nBuilding real-time voice agents forces you to unlearn everything you know about text-based chatbots. You have to actively manage \"turn-taking\", speech interruptions, Voice Activity Detection (VAD), and latency. Here are the biggest hurdles I faced:\n\n**Challenge 1: The Silent Handoff Bug**\n\nDuring the multi-agent handoff, I wrote a prompt instructing the main agent to say \"I will transfer you\" and then call the transfer function tool in the same turn. What happened: The LLM would often just generate the spoken text and \"forget\" to execute the hidden tool call. The user would hear \"I will transfer you...\" and then sit in absolute silence forever! The Fix: I updated the tool description to explicitly enforce the agent to do both simultaneously. Furthermore, when the new agent (Aryabhata) took over, he would wait silently for the user to speak first. I had to write a script in the backend to auto-trigger Aryabhata's greeting 1.5 seconds after the handoff was complete!\n\n**Challenge 2: The Overly Polite Specialist **\n\nAfter Aryabhata graded a math problem and logged the score via a tool call, he would often go completely silent, assuming his turn was over and waiting for the user to prompt him again. The Fix: I had to explicitly write CRITICAL rules in the System Prompt: “After evaluating an answer and calling a tool, you MUST explicitly ask the user if they want to try another problem and wait for their confirmation! Do not stay silent.” In Voice AI, the agent must explicitly manage the conversational flow.\n\n**Challenge 3: Balancing Latency and Intelligence**\n\nThe faster the AI responds, the more natural the conversation feels. However, complex system prompts and heavy tool-calling slow down the LLM's Time-To-First-Token (TTFT). The Fix: I moved as much logic as possible out of the LLM prompt and into the Python backend. By letting the LLM focus solely on the conversation and using lightweight tools for database lookups and routing, combined with Murf Falcon's insane TTS speed, the latency dropped to a level where interruptions felt natural.\n\n**Show Me The Code! (How to Build Your Own)**\n\nWant to build your own real-time voice AI? You don't need a supercomputer—just a decent internet connection and some API keys. Here is how you can set up my project locally!\n\n**1. Clone the Repository & Set Up the Backend:** Ensure you have Python installed. We use uv for lightning-fast dependency management (if you haven't used uv yet, you are missing out!).\n\ngit clone [Your GitHub Repo URL]\n\ncd backend\n\nuv sync\n\n**2. Environment Variables:** Never hardcode your API keys! Create a .env file in the backend and frontend directories based on the provided .env.example files. You will need API keys for:\n\nLiveKit Cloud (For WebRTC Transport)\n\nOpenAI (For the LLM Brain)\n\nDeepgram (For STT)\n\nMurf AI (For Falcon TTS)\n\n**3. Run the Project:** Start the backend agent server first:\n\nuv run python src/agent.py dev\n\nThen, in a new terminal, spin up the Next.js frontend to see the UI and analytics dashboard:\n\ncd frontend\n\nnpm run dev\n\n**Code Highlight: The Multi-Agent Handoff**\n\nHere is a simplified snippet of how the main agent transfers the live call to the Math Specialist using LiveKit's function tools. Notice how we use asyncio.create_task to force the new agent to speak immediately!\n\n**@function_tool(description=\"Call this tool to hand off the conversation to the Maths Practice Specialist when the user asks for Math.\")\n\nasync def transfer_to_maths_specialist(self, context: RunContext) -> Agent:\n\n# Initialize the new specialist agent\n\nmaths_agent = MathsSpecialist(\n\nchat_ctx=self.chat_ctx.copy(exclude_instructions=True),\n\nparent_agent=self\n\n)\n\n```\n# Trigger an automatic greeting so the new agent speaks immediately!\nasync def trigger_intro():\n    await asyncio.sleep(1.5)\n    if hasattr(self, \"session\") and self.session:\n        await self.session.generate_reply(\n            instructions=\"Introduce yourself as Aryabhata the Maths Specialist and IMMEDIATELY ask the first beginner math problem.\"\n        )\n\nimport asyncio\nasyncio.create_task(trigger_intro())\n\n# Return the new agent to hand over the session\nreturn maths_agent**\n```\n\nWhat's Next for Vidya Vani?\n\nTen days is a short amount of time, and there is so much more I want to build. If I had another month, I would love to add:\n\nTrue Multilingual Support: Allow users to seamlessly switch between Hindi, Marathi, Bengali, and English mid-sentence, with the agent dynamically responding in the same language.\n\nGamification: Add a persistent points, streaks, and badges system to the frontend analytics dashboard to keep young learners motivated to return every day!\n\nSMS Integration: Send a text message summary of the day's lesson to the parents after the call ends.\n\n🔗 Links\n\nGitHub Repository: [https://github.com/satwik146/voice-for-Bharat](https://dev.tourl)", "url": "https://wpnews.pro/news/10-days-to-build-a-voice-ai-tutor-the-good-the-bad-and-the-why-is-it-silent", "canonical_source": "https://dev.to/satwik_mn/10-days-to-build-a-voice-ai-tutor-the-good-the-bad-and-the-why-is-it-silent-7ii", "published_at": "2026-08-15 09:53:35+00:00", "updated_at": "2026-08-15 10:11:56.575338+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-products", "ai-agents", "developer-tools"], "entities": ["Murf AI", "Vidya Vani", "LiveKit", "Deepgram", "OpenAI", "Anthropic", "Murf Falcon"], "alternates": {"html": "https://wpnews.pro/news/10-days-to-build-a-voice-ai-tutor-the-good-the-bad-and-the-why-is-it-silent", "markdown": "https://wpnews.pro/news/10-days-to-build-a-voice-ai-tutor-the-good-the-bad-and-the-why-is-it-silent.md", "text": "https://wpnews.pro/news/10-days-to-build-a-voice-ai-tutor-the-good-the-bad-and-the-why-is-it-silent.txt", "jsonld": "https://wpnews.pro/news/10-days-to-build-a-voice-ai-tutor-the-good-the-bad-and-the-why-is-it-silent.jsonld"}}