{"slug": "building-bharat-buddy-my-10-day-voice-ai-journey-with-murf-falcon", "title": "Building Bharat Buddy: My 10-Day Voice AI Journey with Murf Falcon", "summary": "A developer built Bharat Buddy, an AI-powered voice tutor for students, during the 10 Days of Voice Agents challenge. The agent supports Hindi, English, and Hinglish, and uses LiveKit for real-time voice, an LLM for reasoning, and Murf Falcon for text-to-speech. It includes memory, tools, specialist handoffs, and human escalation, aiming to make learning more natural and conversational.", "body_md": "What if learning didn't require typing?\n\nWhat if a student could simply speak to an AI tutor in Hindi, English, or Hinglish, ask questions naturally, practice problems, and even get connected to a specialist when needed?\n\nThat was the idea behind Bharat Buddy, the voice agent I built during 10 Days of Voice Agents — VoiceForBharat Edition.\n\nOver these 10 days, I went from a basic voice assistant to an agent that can remember users, use tools, make outbound calls, escalate problems to humans, track call analytics, and hand conversations over to specialist agents.\n\nThis post is about what I built, what went wrong, what I learned, and how someone else can start building their own voice agent.\n\n**🚀 What is Bharat Buddy?**\n\nBharat Buddy is an AI-powered voice tutor designed for students.\n\nThe goal was to make learning feel more like talking to a helpful teacher rather than interacting with a traditional chatbot.\n\nA student can speak naturally:\n\n\"Mujhe percentage samajh nahi aa raha.\"\n\nor:\n\n\"Can you explain this maths problem?\"\n\nor even:\n\n\"Bhai, mujhe Hinglish mein samjha.\"\n\nBharat Buddy can respond conversationally and adapt to the user's language.\n\nThe project focuses on:\n\n🇮🇳 Hindi\n\n🇬🇧 English\n\n🗣️ Hinglish\n\n🎙️ Voice-first interaction\n\n📚 Learning assistance\n\n🧠 User memory\n\n🧮 Maths practice\n\n👨💻 Specialist handoffs\n\n🧑💼 Human escalation\n\n📊 Call analytics\n\n**The idea was simple:**\n\nMake AI learning more natural, accessible and conversational.\n\n**🎯 Why Voice?**\n\nMost AI applications still expect users to:\n\nType → Wait → Read → Type again\n\nBut speaking is much more natural.\n\nFor a student, saying:\n\n\"What is 20 percent of 500?\"\n\nis much easier than typing a detailed prompt.\n\nVoice also makes the interaction feel more human.\n\nInstead of staring at a chatbot, you can actually talk to your tutor.\n\nThat's what I wanted Bharat Buddy to feel like.\n\n**🏗️ How Bharat Buddy Works**\n\nThe basic architecture looks like this:\n\n```\n            USER\n              │\n              │ Voice\n              ▼\n    ┌──────────────────┐\n    │   Speech-to-Text │\n    └────────┬─────────┘\n             │\n             ▼\n    ┌──────────────────┐\n    │       LLM        │\n    │  Bharat Buddy    │\n    └────────┬─────────┘\n             │\n   ┌─────────┼──────────┐\n   │         │          │\n   ▼         ▼          ▼\nMemory     Tools    Specialist\n                      Agent\n   │         │          │\n   └─────────┼──────────┘\n             │\n             ▼\n    ┌──────────────────┐\n    │   Text-to-Speech │\n    │   Murf Falcon    │\n    └────────┬─────────┘\n             │\n             ▼\n           USER\n```\n\nThe project uses LiveKit for real-time voice communication, an LLM for reasoning, and Murf Falcon for voice generation.\n\n**Murf AI**\n\n🎙️ Murf Falcon\n\nOne of the most important parts of the project was making the AI actually sound conversational.\n\nI used Murf Falcon, the fastest TTS API mentioned in the challenge, to generate the agent's voice.\n\nThis made the interaction much more natural than simply displaying text.\n\nThe agent can speak responses such as:\n\n\"Namaste! I'm Bharat Buddy. What would you like to learn today?\"\n\nand continue the conversation naturally.\n\nThe voice layer became a major part of the experience rather than just an additional feature.\n\n**🧠 Memory**\n\nA useful assistant shouldn't completely forget you after every conversation.\n\nSo I added memory capabilities to Bharat Buddy.\n\nThis allows the system to maintain useful user context and make future conversations more meaningful.\n\nThe important lesson here was that memory should be useful, not excessive.\n\nA voice agent doesn't need to remember every single sentence.\n\nIt should remember information that actually improves future interactions.\n\n**🛠️ Tools**\n\nAnother important part of the project was giving the agent access to tools.\n\nInstead of making the LLM pretend it knows everything, tools allow the agent to perform specific actions.\n\nFor example:\n\nUser\n\n↓\n\nBharat Buddy\n\n↓\n\nDoes this require a tool?\n\n↓\n\nYes → Call tool\n\n↓\n\nGet result\n\n↓\n\nExplain result to user\n\nThis makes the agent more reliable and gives it capabilities beyond simple conversation.\n\n**📞 Outbound Calls**\n\nBharat Buddy also gained the ability to make outbound calls.\n\nThis was one of the more interesting parts of the challenge because the agent moved beyond:\n\n\"User calls AI\"\n\nto:\n\n\"AI can initiate a call.\"\n\nThis opened up possibilities such as:\n\nFollow-ups\n\nReminders\n\nStudent assistance\n\nNotifications\n\nFuture learning workflows\n\n**🧑💼 Human Escalatio** n\n\nOne of the biggest lessons I learned was:\n\nAn AI agent should know when it should stop being the expert.\n\nSo I added a human escalation system.\n\nFor example, if the user needs help that the agent cannot safely handle, the agent can create a request for a human.\n\nThe workflow became:\n\nUser needs human help\n\n↓\n\nAgent identifies situation\n\n↓\n\nAgent asks for permission\n\n↓\n\nUser agrees\n\n↓\n\nCreate escalation request\n\n↓\n\nGenerate reference ID\n\n↓\n\nTell user what happens next\n\nThe escalation request contains only useful information rather than dumping the entire conversation.\n\nThis was especially important from a privacy perspective.\n\n**📊 Call Analytics Dashboard**\n\nOn Day 8, I built a simple analytics system.\n\nThe dashboard tracks:\n\nTotal Calls\n\nHow many calls have been recorded.\n\nSuccessful Calls\n\nCalls where the defined success condition was achieved.\n\nFailed Calls\n\nCalls where the success condition wasn't reached.\n\nThe data is stored using SQLite and exposed through an API that the dashboard can consume.\n\nFor example:\n\n{\n\n\"total\": 7,\n\n\"successful\": 4,\n\n\"failed\": 0\n\n}\n\nThe important part is that these aren't hardcoded numbers.\n\nThey come from actual agent activity.\n\n🧮 Maths Practice Specialist\n\nOne of my favourite features was adding a specialist agent.\n\nInstead of making Bharat Buddy an expert at everything, I created a separate:\n\nMaths Practice Specialist\n\nIts job is focused specifically on:\n\nArithmetic\n\nPercentages\n\nFractions\n\nRatios\n\nAlgebra\n\nBasic geometry\n\nStep-by-step maths explanations\n\nThe flow looks like:\n\nStudent\n\n↓\n\nBharat Buddy\n\n↓\n\nMaths question?\n\n↓\n\nYes\n\n↓\n\n\"I'll connect you with our Maths Practice Specialist.\"\n\n↓\n\nMaths Specialist\n\n↓\n\nContinues the same conversation\n\nThe user doesn't have to explain the entire problem again.\n\nThis taught me an important architectural concept:\n\nA good AI system doesn't necessarily need one giant agent.\n\nMultiple focused agents can often be easier to control.\n\n**🌐 Hindi + English + Hinglish**\n\nAnother important part of Bharat Buddy was language flexibility.\n\nThe agent is designed to respond according to the user's language:\n\nEnglish → English\n\nHindi → Hindi\n\nHinglish → Hinglish\n\nFor example:\n\nUser:\n\n\"Percentage ka concept simple language mein samjhao.\"\n\nBharat Buddy can respond naturally in Hinglish rather than suddenly switching to formal English.\n\nFor an India-focused voice assistant, this makes a huge difference.\n\n**🎨 Frontend**\n\nThe frontend provides the interface through which users interact with the voice agent.\n\nThe goal wasn't to create an unnecessarily complicated UI.\n\nIt focuses on showing the agent's state and making the voice interaction obvious.\n\nThe overall experience is:\n\nOpen Agent\n\n↓\n\nConnect\n\n↓\n\nSpeak\n\n↓\n\nAgent Processes\n\n↓\n\nAgent Responds\n\n😵 The Hardest Part\n\nHonestly, the project didn't work perfectly on the first try.\n\nOne of the most frustrating problems happened while implementing the specialist handoff.\n\nI initially received an error related to the tool schema:\n\ninvalid JSON schema for tool\n\nhandoff_to_maths_specialist\n\nAfter fixing that, another problem appeared:\n\nTypeError:\n\nmissing a required argument: 'reason'\n\nThe model was trying to call the handoff function, but the function definition and the arguments expected by the Python function didn't match.\n\nThe important lesson:\n\nWhen using function calling, the tool schema and the actual Python function signature must agree.\n\nFor example, if the function expects:\n\nasync def handoff_to_maths_specialist(reason: str):\n\nthen the model needs to provide a reason argument.\n\nThis sounds small, but it can completely break the voice pipeline.\n\n**🔐 Another Important Lesson: Secrets**\n\nWhile building the project, I also learned how important it is to keep API credentials outside the source code.\n\nAPI keys should be stored in environment variables such as:\n\n.env.local\n\nand should never be committed to GitHub.\n\nMy .gitignore contains rules for environment files and database files so that sensitive/local data isn't accidentally uploaded.\n\nThis is especially important when publishing an AI project publicly.\n\n**🧰 Tech Stack**\n\nThe main technologies used in Bharat Buddy include:\n\nTechnology Purpose\n\nPython Backend / agent logic\n\nLiveKit Real-time voice communication\n\nMurf Falcon Text-to-Speech\n\nLLM Reasoning and conversation\n\nSQLite Local data storage\n\nHTML/CSS/JS Dashboard/frontend\n\nGit/GitHub Version control\n\n🚀 How to Run the Project\n\nIf you want to experiment with the project, start by cloning the repository:\n\nBharat Buddy — GitHub Repository\n\nA multilingual AI voice assistant built during\n\n10 Days of Voice Agents — VoiceForBharat Edition, powered byMurf Falconand LiveKit.\n\nBharat Buddy is a real-time AI voice agent designed to make learning and everyday assistance more natural through voice.\n\nIt can understand and respond in **English, Hindi and Hinglish**, remember returning users, use tools, make outbound calls, escalate problems to humans, track call analytics and hand conversations to specialist agents.\n\nMany users are more comfortable speaking than typing, especially when using technology in their everyday language.\n\nBharat Buddy explores how a voice-first AI assistant can make digital learning and assistance more accessible by allowing users to simply **speak naturally instead of typing everything**.\n\nThe project focuses on:\n\nThen create your environment and install the project dependencies.\n\nFor example:\n\ngit clone [https://github.com/Rishabhpandey2007/murf-livekit-starter.git](https://github.com/Rishabhpandey2007/murf-livekit-starter.git)\n\ncd murf-livekit-starter\n\ncd backend\n\nuv sync\n\nCreate your local environment file:\n\n.env.local\n\nAdd the required API credentials there.\n\nDo not put your real API keys directly into Python files or commit .env.local to GitHub.\n\nThen start the agent according to the project instructions.\n\nOnce the agent is running, open the browser client, connect to the voice session, and start speaking.\n\n🧪 Testing the Agent\n\nI tested different paths during development.\n\nNormal conversation\n\nUser:\n\nExplain percentage.\n\nBharat Buddy:\n\nExplains percentage.\n\nSpecialist conversation\n\nUser:\n\nI want to practice maths percentage problems.\n\nBharat Buddy:\n\nI'll connect you with our Maths Practice Specialist.\n\nMaths Specialist:\n\nHi! I'm Bharat Buddy's Maths Practice Specialist...\n\nHuman escalation\n\nUser needs human help\n\n↓\n\nAgent asks permission\n\n↓\n\nUser approves\n\n↓\n\nEscalation created\n\n↓\n\nReference ID returned\n\nAnalytics\n\nCall completed\n\n↓\n\nOutcome stored\n\n↓\n\nDashboard updated\n\n📈 What I Learned in 10 Days\n\nBefore this challenge, I mostly thought about AI assistants as:\n\nInput → LLM → Output\n\nAfter building Bharat Buddy, I realized a production-style voice agent is much more than that.\n\nYou need to think about:\n\nReal-time audio\n\nSpeech recognition\n\nTTS latency\n\nPrompt design\n\nTool calling\n\nMemory\n\nSafety\n\nHuman escalation\n\nSpecialist routing\n\nAnalytics\n\nPrivacy\n\nError handling\n\nThe LLM is only one component.\n\nThe real engineering challenge is making all the components work together reliably.\n\n🔮 What I Would Build Next\n\nBharat Buddy is still a work in progress.\n\nSome things I'd like to add next:\n\n📱 Better mobile experience\n\nMake the voice experience easier to use on low-end smartphones.\n\n📚 Study Material Integration\n\nAllow students to upload notes and ask questions about them.\n\n📊 Better Analytics\n\nTrack:\n\nSuccess rate\n\nCall duration\n\nMost common questions\n\nFailure reasons\n\nSpecialist handoffs\n\n👨🏫 More Specialist Agents\n\nFor example:\n\nBharat Buddy\n\n│\n\n├── Maths Specialist\n\n├── English Specialist\n\n├── Coding Specialist\n\n└── Exam Preparation Specialist\n\n🌍 More Indian Languages\n\nEventually, I'd like Bharat Buddy to support more Indian languages and make voice-based learning accessible to more students.\n\n💡 Final Thoughts\n\nThe biggest thing I learned from this challenge is that building a voice agent isn't just about making an AI talk.\n\nIt's about designing a system that knows:\n\nwhen to answer,\n\nwhen to use a tool,\n\nwhen to remember,\n\nwhen to call a specialist,\n\nwhen to ask a human,\n\nand most importantly,\n\nwhen not to pretend it knows something.\n\nBharat Buddy started as an idea for a voice tutor.\n\nAfter 10 days, it became a much bigger experiment in building real-time, conversational AI systems.\n\nI'm still learning, but this challenge gave me a much better understanding of what it takes to turn an AI idea into a working voice product.\n\n🔗 Project\n\nGitHub:\n\nBharat Buddy Repository\n\nBuilt during:\n\n10 Days of Voice Agents — VoiceForBharat Edition\n\nVoice: Murf Falcon\n\nTech: Python + LiveKit + LLM + Murf Falcon + SQLite\n\n🙌 Thanks\n\nA big thanks to Murf AI for organizing the 10 Days of Voice Agents — VoiceForBharat Edition challenge.\n\nBuilding something every day for 10 days, debugging it, breaking it, fixing it, and finally documenting the journey was a genuinely valuable experience.\n\n10 days. One voice agent. A lot of debugging. 🚀", "url": "https://wpnews.pro/news/building-bharat-buddy-my-10-day-voice-ai-journey-with-murf-falcon", "canonical_source": "https://dev.to/rishabh_pandey_03ea87aa4c/building-bharat-buddy-my-10-day-voice-ai-journey-with-murf-falcon-2d7g", "published_at": "2026-08-15 05:53:01+00:00", "updated_at": "2026-08-15 06:11:30.943412+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-products", "developer-tools"], "entities": ["Bharat Buddy", "Murf Falcon", "LiveKit", "VoiceForBharat"], "alternates": {"html": "https://wpnews.pro/news/building-bharat-buddy-my-10-day-voice-ai-journey-with-murf-falcon", "markdown": "https://wpnews.pro/news/building-bharat-buddy-my-10-day-voice-ai-journey-with-murf-falcon.md", "text": "https://wpnews.pro/news/building-bharat-buddy-my-10-day-voice-ai-journey-with-murf-falcon.txt", "jsonld": "https://wpnews.pro/news/building-bharat-buddy-my-10-day-voice-ai-journey-with-murf-falcon.jsonld"}}