{"slug": "from-voice-agent-to-multi-agent-learning-assistant-my-10-days-of-voiceforbharat", "title": "From Voice Agent to Multi-Agent Learning Assistant: My 10 Days of VoiceForBharat", "summary": "A developer built an AI Voice Learning Assistant during Murf AI's 10 Days of Voice Agents — VoiceForBharat Edition, under the Learning & Literacy track. The system uses LiveKit, Deepgram Nova-3, Google Gemini, and Murf Falcon to enable real-time voice interaction, with features like persistent memory, interactive exercises, human escalation, and handoff to a Maths Specialist Agent.", "body_md": "As part of Murf AI's 10 Days of Voice Agents — VoiceForBharat Edition, I built an AI Voice Learning Assistant under the Learning & Literacy track.\n\nThe goal was to create a voice-first learning experience where learners can interact naturally with an AI assistant, practice learning exercises, receive personalized assistance, request human help, and even be connected to a specialized Maths agent.\n\nInstead of building an AI that simply answers questions, I wanted to build a complete voice learning system with memory, tools, analytics, human escalation, and multi-agent handoff.\n\nMany digital learning platforms depend heavily on typing and reading. This can make interaction less natural, especially for learners who are more comfortable communicating through speech.\n\nI wanted to explore how real-time voice AI could make learning more interactive and conversational.\n\nThe project focuses on creating a voice-based learning assistant that can:\n\n| Technology | Purpose | \n|---|---|\n| Python | Backend and AI agent | \n| LiveKit | Real-time voice communication | \n| Deepgram Nova-3 | Speech-to-Text | \n| Google Gemini | LLM / reasoning | \n| Murf Falcon | Text-to-Speech | \n| Silero VAD | Voice activity detection | \n| Next.js | Frontend | \n| React + TypeScript | User interface | \n| Tailwind CSS | Styling | \n| SQLite | Learner memory | \n\nThe voice interaction follows this pipeline:\n\nUser Speech\n\n    ↓\n\nLiveKit\n\n    ↓\n\nDeepgram Nova-3\n\n    ↓\n\nGoogle Gemini\n\n    ↓\n\nAgent Tools / Memory / Specialist\n\n    ↓\n\nMurf Falcon\n\n    ↓\n\nUser hears response\n\nBuilt the basic real-time voice agent using LiveKit, Deepgram, Google Gemini, and Murf Falcon.\n\nDefined the agent's personality, behavior, role, and safety boundaries.\n\nCustomized the frontend and added visual feedback for the voice interaction.\n\nWorked on language-aware conversations and Indian language/code-mixed interaction.\n\nAdded persistent learner memory and interactive learning exercises.\n\nExperimented with connecting the voice agent to phone calls using Twilio and LiveKit.\n\nAdded the ability to request human/teacher assistance and generate an escalation ID.\n\nBuilt a dashboard showing total, successful, and failed learning sessions.\n\nImplemented handoff from the main learning assistant to a dedicated Maths Specialist Agent.\n\nDocumented the complete project, architecture, challenges, solutions, and lessons learned.\n\nThe AI Voice Learning Assistant can remember useful information shared by the learner during conversations. The memory system uses SQLite to store learner-related information and preferences, allowing the assistant to provide a more personalized learning experience. The stored information can also be updated when the learner's preferences change.\n\nThe assistant provides interactive learning exercises through voice. Learners can answer questions verbally and receive immediate feedback from the AI. Exercises are stored in `exercises.json`, and learner responses are evaluated to determine whether they are correct or incorrect. The exercise results are also used to track the outcome of learning sessions.\n\nWhen a learner needs help from a human teacher or the AI is unable to appropriately handle a request, the learner can request human assistance. The system creates a unique escalation ID for the request, which can be used to track the learner's need for human support through the dashboard.\n\nThe analytics dashboard provides an overview of voice learning sessions by displaying **Total Calls, Successful Calls, and Failed Calls**. Session outcomes are recorded based on the learner's interaction with the learning exercises, providing useful insights into the effectiveness of learning sessions.\n\nThe system includes a dedicated Maths Practice Specialist Agent for mathematics-related learning requests. When the learner asks for help with a maths problem, the main learning assistant can hand off the conversation to the specialist agent. Relevant conversation context is passed during the handoff so that the learner does not need to repeat the entire problem.\n\nThe application provides real-time voice interaction, allowing learners to communicate naturally through speech. **Deepgram Nova-3** is used for speech-to-text, **Google Gemini** handles AI reasoning and conversation, and **Murf Falcon** provides text-to-speech. **LiveKit** enables real-time voice communication, while Silero VAD and LiveKit Turn Detector help detect speech and conversation turns.\n\nThe assistant follows defined instructions and safety boundaries to provide a focused learning experience. It stays within its educational role, asks for clarification when requests are unclear, avoids exposing sensitive information or API credentials, and uses human escalation when AI assistance is not sufficient. Guardrails also help the assistant decide when to use learning tools and when to hand off mathematics-related requests to the specialist agent.\n\nBuilding the project was not completely straightforward.\n\nInitially, the agent remembered a learner's first preference but did not correctly update it when the learner changed their preference. I had to improve the memory workflow so that information could be retrieved and updated correctly.\n\nOne challenge was correctly distinguishing successful and failed learning sessions. I improved the session outcome logic so that the actual exercise evaluation determines the final analytics result.\n\nInitially, generating an escalation ID did not immediately provide the expected dashboard visibility. I worked on connecting the escalation request with the frontend dashboard.\n\nThe main challenge was making the transition between agents feel natural while preserving enough conversation context so the learner would not need to repeat the problem.\n\nThis challenge helped me understand that building a voice agent is much more than connecting an LLM to a text-to-speech API.\n\nI learned how to work with:\n\nSome improvements I would like to make include:\n\nThe 10 Days of Voice Agents challenge gave me the opportunity to move from building a simple conversational voice agent to designing a more complete AI learning system.\n\nThe biggest lesson I learned is that a useful voice agent needs more than a good voice. It needs memory, tools, reliable state management, safety boundaries, analytics, and the ability to involve humans or specialized agents when necessary.\n\nThis project helped me understand how real-time voice AI can be applied to learning and how technologies such as LiveKit, Deepgram, Google Gemini, and Murf Falcon can work together to create a natural voice-first experience.\n\nGithub Repository -[https://github.com/NariboyinaRajaVaishnavi7/murf-livekit-starter](https://github.com/NariboyinaRajaVaishnavi7/murf-livekit-starter)", "url": "https://wpnews.pro/news/from-voice-agent-to-multi-agent-learning-assistant-my-10-days-of-voiceforbharat", "canonical_source": "https://dev.to/raja_vaishnavi_931fbf23c6/from-voice-agent-to-multi-agent-learning-assistant-my-10-days-of-voiceforbharat-3f5n", "published_at": "2026-09-09 14:20:36+00:00", "updated_at": "2026-09-09 14:41:39.752165+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-products", "developer-tools"], "entities": ["Murf AI", "LiveKit", "Deepgram Nova-3", "Google Gemini", "Murf Falcon", "Silero VAD", "Next.js", "Twilio"], "alternates": {"html": "https://wpnews.pro/news/from-voice-agent-to-multi-agent-learning-assistant-my-10-days-of-voiceforbharat", "markdown": "https://wpnews.pro/news/from-voice-agent-to-multi-agent-learning-assistant-my-10-days-of-voiceforbharat.md", "text": "https://wpnews.pro/news/from-voice-agent-to-multi-agent-learning-assistant-my-10-days-of-voiceforbharat.txt", "jsonld": "https://wpnews.pro/news/from-voice-agent-to-multi-agent-learning-assistant-my-10-days-of-voiceforbharat.jsonld"}}