{"slug": "building-bhasha-academy-a-multi-agent-hinglish-voice-tutor-with-murf-falcon", "title": "Building Bhasha Academy: A Multi-Agent Hinglish Voice Tutor with Murf Falcon & LiveKit", "summary": "An engineer built Bhasha Academy, an AI-powered voice tutor that helps Indian learners practice English and math in Hinglish. The system uses LiveKit for real-time voice, Deepgram Nova-3 for speech recognition, Google Gemini for conversation, and Murf Falcon TTS for natural-sounding speech. A key fix involved prompting the model to write Hindi words in Devanagari script to improve pronunciation.", "body_md": "Language is meant to be spoken, not just read.\n\nFor millions of students and job seekers in India, being confident in English and basic math can open the door to better career opportunities. But many learners don't have access to a personal tutor. Even when they do, the fear of making mistakes in front of someone can make practicing uncomfortable.\n\nSo I built **Bhasha Academy**, an AI-powered voice tutor designed to give learners a private, judgment-free space to practice.\n\nIt can have conversations in **Hinglish**, help users practice English, solve basic math problems, remember learning progress, and even connect the learner with a human teacher when needed.\n\nThe interesting part is that the entire experience is voice-first.\n\nHere's how I built it.\n\nI wanted Bhasha Academy to feel less like talking to a chatbot and more like talking to a patient tutor.\n\nThe learner should be able to simply say:\n\n\"Mujhe English practice karni hai.\"\n\nAnd the agent should understand the Hinglish, respond naturally, and continue the conversation.\n\nFor math, the learner can say:\n\n\"Samar, 25 percent of 200 kitna hai?\"\n\nThe system can then switch to a math specialist and continue from the same conversation.\n\nThe main goals were:\n\nThe system is built using several services, with LiveKit sitting at the center of the real-time voice experience.\n\n```\nUser\n  ↓\nBrowser / SIP\n  ↓\nLiveKit\n  ↓\nSpeech-to-Text\n  ↓\nDeepgram Nova-3\n  ↓\nGemini\n  ↓\nAgent / Tools\n  ↓\nMurf Falcon TTS\n  ↓\nLiveKit\n  ↓\nUser\n```\n\nThere are also a few supporting components:\n\n```\nSQLite\n  ├── Student profiles\n  ├── Learning progress\n  ├── Call analytics\n  └── Escalation tickets\n\nDiscord Webhook\n  └── Human teacher notifications\n\nNext.js\n  └── Admin dashboard\n```\n\nI use **Deepgram Nova-3** for speech recognition.\n\nThe important part here is multilingual support.\n\nA learner might say:\n\n\"Yesterday I went market, but mujhe wahan kuch samajh nahi aaya.\"\n\nThe system needs to understand both languages without forcing the user to speak only English or only Hindi.\n\nFor the conversational layer, I use **Google Gemini 3.5 Flash Lite**.\n\nThe model is responsible for:\n\nFor voice generation, I use **Murf Falcon TTS** through LiveKit.\n\nBhasha Academy currently uses two voices:\n\nBoth are designed to sound natural for Indian users.\n\n**LiveKit Agents SDK** handles the real-time communication layer.\n\nIt takes care of things like:\n\nThis makes it possible to have a real conversation instead of the traditional:\n\n```\nRecord → Upload → Wait → Get Response → Play\n```\n\nInstead, audio can be streamed continuously.\n\nOne of the first problems I noticed was surprisingly simple.\n\nThe AI understood Hinglish perfectly, but the voice didn't always pronounce it naturally.\n\nFor example, if the model generated:\n\n```\nBahut achha, let's try again!\n```\n\nthe TTS system could interpret \"Bahut achha\" using English pronunciation.\n\nThe result sounded robotic.\n\nI changed the system prompt so that Hindi words should be written in **Devanagari**.\n\nInstead of:\n\n```\nBahut achha, let's try again!\n```\n\nthe model generates:\n\n```\nबहुत अच्छा, let's try again!\n```\n\nThis made a huge difference.\n\nMurf Falcon can handle mixed scripts, so it can naturally switch between Hindi and English.\n\nFor example:\n\n```\nबहुत बढ़िया! Let's try another word.\n```\n\nThis is a small prompt change, but it had a big impact on the voice experience.\n\nA tutor should remember its students.\n\nBhasha Academy uses SQLite to store learner profiles.\n\nWhen someone calls for the first time, the agent doesn't automatically save their information.\n\nInstead, it asks for permission.\n\nFor example:\n\n\"Would you like me to remember your name and learning progress for your next session?\"\n\nIf the learner agrees, the system can store information such as:\n\nThere is also a `forget_caller`\n\ntool.\n\nA learner can ask the system to forget them, and their stored profile can be removed.\n\nThis was important to me because personalization shouldn't come at the cost of user control.\n\nAnother interesting part of the project is the specialist handoff system.\n\nInstead of making one huge agent handle everything, I created separate agents.\n\nAnisha is the general language tutor.\n\nShe handles:\n\nSamar is the math specialist.\n\nHe handles:\n\nIf the learner says:\n\n\"Can we do some maths?\"\n\nAnisha can transfer the conversation to Samar.\n\nThe important part is that Samar shouldn't start from zero.\n\nHe should already know who the learner is and what was discussed.\n\nThe handoff looks roughly like this:\n\n``` python\n@function_tool\nasync def transfer_to_math_specialist(\n    self,\n    context: RunContext\n) -> tuple[Agent, str]:\n\n    math_agent = MathPracticeAgent(\n        chat_ctx=self.chat_ctx.copy(\n            exclude_instructions=True\n        )\n    )\n\n    return (\n        math_agent,\n        \"Transferring you to Samar, our maths practice specialist.\"\n    )\n```\n\nThe key idea is passing a copy of the existing `ChatContext`\n\n.\n\nThis preserves the conversation while allowing the new agent to have its own instructions and personality.\n\nBhasha Academy isn't limited to the browser.\n\nUsing LiveKit SIP integrations, the system can also make outbound calls.\n\nThis opens up the possibility of scheduled lessons.\n\nFor example:\n\n```\nScheduled lesson\n      ↓\nSystem calls student\n      ↓\nStudent answers\n      ↓\nAI tutor starts lesson\n```\n\nBut there is another problem with outbound calling: voicemail.\n\nThere is no reason to keep expensive AI services running when the call is answered by an answering machine.\n\nSo I added voicemail detection.\n\nIf the system detects a typical voicemail greeting, it can:\n\n`hang_up`\n\ntool.AI shouldn't try to solve every problem.\n\nIf a learner repeatedly struggles or becomes frustrated, Bhasha Academy can ask whether they want to speak with a human teacher.\n\nIf the learner agrees, the system creates an escalation ticket.\n\nThe ticket contains information such as:\n\n```\nReference ID\nStudent Name\nUrgency\nReason\n```\n\nA Discord webhook then sends the notification to the teacher/admin channel.\n\nFor example:\n\n```\npayload = {\n    \"embeds\": [{\n        \"title\": f\"🚨 Human Help Request Raised ({urgency.upper()})\",\n        \"color\": 15158332 if urgency == \"emergency\" else 3447003,\n        \"fields\": [\n            {\n                \"name\": \"Reference ID\",\n                \"value\": ref_id,\n                \"inline\": True\n            },\n            {\n                \"name\": \"Student Name\",\n                \"value\": name,\n                \"inline\": True\n            },\n            {\n                \"name\": \"Urgency\",\n                \"value\": urgency,\n                \"inline\": True\n            },\n            {\n                \"name\": \"Reason\",\n                \"value\": reason,\n                \"inline\": False\n            }\n        ]\n    }]\n}\n```\n\nThe idea is simple:\n\n**AI handles the routine conversations. Humans step in when the learner needs more help.**\n\nEvery call is also logged.\n\nThe system tracks things like:\n\nThe data is stored in SQLite and exposed through a Next.js dashboard.\n\nThis makes it easier to understand how the system is performing instead of relying only on individual conversations.\n\nGetting a good text response is only one part of the problem.\n\nA voice agent also needs:\n\nSmall improvements in any of these areas can make the experience feel much more natural.\n\nHinglish isn't simply English with Hindi words.\n\nThe way those words are written can directly affect how the TTS system pronounces them.\n\nSwitching Hindi words to Devanagari was one of the simplest and most effective improvements I made.\n\nAgent handoffs sound simple:\n\n```\nAgent A → Agent B\n```\n\nBut without preserving context, it becomes:\n\n```\nAgent A → Agent B → \"What's your name?\"\n```\n\nPassing the conversation context makes the handoff feel like one continuous conversation.\n\nA good tutor isn't necessarily one that answers everything.\n\nSometimes the best action is:\n\n\"Would you like me to connect you with a teacher?\"\n\nThat human fallback makes the system more useful and trustworthy.\n\nThe project is based on the open-source **Murf LiveKit Starter** repository.\n\nRepository:\n\n[https://github.com/bharatbushan03/murf-livekit-starter](https://github.com/bharatbushan03/murf-livekit-starter)\n\nYou'll need:\n\n`uv`\n\n`pnpm`\n\n```\ngit clone https://github.com/bharatbushan03/murf-livekit-starter.git\ncd murf-livekit-starter\n```\n\nInstall the backend:\n\n```\ncd backend\n\nuv sync\n\nuv run python src/agent.py download-files\n```\n\nInstall the frontend:\n\n```\ncd ../frontend\n\npnpm install\n```\n\nConfigure your environment variables with the required API credentials:\n\n```\nLIVEKIT_URL\nLIVEKIT_API_KEY\nLIVEKIT_API_SECRET\n\nMURF_API_KEY\nDEEPGRAM_API_KEY\nGOOGLE_API_KEY\n\nDISCORD_WEBHOOK_URL\n```\n\nThe Discord webhook is optional.\n\n```\ncd backend\n\nuv run python src/agent.py dev\n```\n\nThen start the frontend in another terminal:\n\n```\ncd frontend\n\npnpm dev\n```\n\nOpen:\n\n```\nhttp://localhost:3000\n```\n\nClick **Start Learning with Anisha** and start talking.\n\nThere are still several things I want to improve.\n\nReal users aren't always sitting in a quiet room.\n\nA student might be practicing from:\n\nI want to tune the voice activity detection system to work better in noisy environments.\n\nAnother feature I'm planning is real-time math progress.\n\nWhile Samar is teaching, the frontend could show something like:\n\n```\nMath Practice\n\nQuestions: 8\nCorrect: 6\nNeeds Practice: 2\n\nTopic:\nPercentages\n```\n\nThis would make the voice interaction feel more connected to the visual interface.\n\nBuilding Bhasha Academy taught me that voice AI can be much more than a voice chatbot.\n\nWhen you combine:\n\nyou can start building something that feels closer to a real tutor.\n\nThere are still many things to improve, but the core idea is simple:\n\n**Give learners a patient tutor they can talk to whenever they want, without being afraid of making mistakes.**\n\nThat's what I'm trying to build with Bhasha Academy.\n\nIf you're building something similar with LiveKit, Murf, Deepgram, or Gemini, I'd love to hear what you're working on.", "url": "https://wpnews.pro/news/building-bhasha-academy-a-multi-agent-hinglish-voice-tutor-with-murf-falcon", "canonical_source": "https://dev.to/bharat03/building-bhasha-academy-a-multi-agent-hinglish-voice-tutor-with-murf-falcon-livekit-njp", "published_at": "2026-08-15 08:51:09+00:00", "updated_at": "2026-08-15 09:11:47.908196+00:00", "lang": "en", "topics": ["artificial-intelligence", "natural-language-processing", "ai-products", "ai-agents", "developer-tools"], "entities": ["Bhasha Academy", "LiveKit", "Deepgram Nova-3", "Google Gemini", "Murf Falcon TTS", "SQLite", "Next.js", "Discord"], "alternates": {"html": "https://wpnews.pro/news/building-bhasha-academy-a-multi-agent-hinglish-voice-tutor-with-murf-falcon", "markdown": "https://wpnews.pro/news/building-bhasha-academy-a-multi-agent-hinglish-voice-tutor-with-murf-falcon.md", "text": "https://wpnews.pro/news/building-bhasha-academy-a-multi-agent-hinglish-voice-tutor-with-murf-falcon.txt", "jsonld": "https://wpnews.pro/news/building-bhasha-academy-a-multi-agent-hinglish-voice-tutor-with-murf-falcon.jsonld"}}