{"slug": "shiksha-vani-shikssaa-vaannii-building-a-voice-ai-literacy-agent-with-murf", "title": "Shiksha Vani (शिक्षा वाणी): Building a Voice AI Literacy Agent with Murf Falcon", "summary": "A developer built Shiksha Vani, a voice AI literacy tutor for spoken English, as part of the 10 Days of Voice Agents challenge. The system uses LiveKit, Deepgram Nova-3, Gemini 1.5 Flash, and Murf Falcon TTS to support Hinglish conversations, with features like SQLite memory, consent-based data saving, and human escalation.", "body_md": "For millions of learners across India, mastering spoken English is a gateway to better educational and career opportunities. However, classroom learning is often passive, and practicing speaking requires a judgment-free partner. Learners face two distinct hurdles:\n\nTo solve this, I built **Shiksha Vani (शिक्षा वाणी)**—an interactive, Hinglish-speaking AI Spoken English and Literacy Tutor. Developed as part of the **10 Days of Voice Agents (#VoiceForBharat Edition)** challenge, Shiksha Vani is designed to act as a supportive practice buddy. It listens to a student's speech, corrects their pronunciation and grammar gently, and engages in daily spoken exercises.\n\n``` php\nmermaid\ngraph TD\n    User([User Voice]) <-->|WebRTC / SIP| LT[LiveKit Real-time Transport]\n    LT <-->|Audio Streams| Agent[Voice Agent Session]\n    Agent -->|Speech-to-Text| STT[Deepgram Nova-3]\n    Agent -->|LLM Reasoner| LLM[Gemini 1.5 Flash]\n    Agent -->|Text-to-Speech| TTS[Murf Falcon API]\n    LLM <-->|Tool Calling| DB[(SQLite Persistent Memory)]\n    LLM <-->|Context Handoff| Specialist[Returns Specialist Agent]\n3. Why Voice?\nText-based chat apps fail to capture the nuances of speech—pronunciation, hesitation, and cadence. Spoken interaction is essential for building verbal confidence. In India, where mobile internet is ubiquitous but literacy rates vary, voice is the most natural, accessible, and frictionless interface.\n\nBy utilizing Murf Falcon, a high-fidelity, low-latency Text-to-Speech (TTS) engine, the agent communicates in warm, authentic Indian accents (like the default hi-IN-kabir). This minimizes the intimidation barrier often associated with dry, robotic western accents.\n\n4. How the System Works\nThe architecture utilizes a real-time media transport layer combined with decoupled cognitive and speech engines:\n\nFrontend: A responsive glassmorphic dashboard (index.html) featuring real-time visualizers, call statistics, and interactive controls.\nReal-time Transport: LiveKit WebRTC serves as the audio pipeline, handling sub-100ms bidirectional streaming.\nSpeech-to-Text: Deepgram Nova-3 translates Hinglish voice inputs to text.\nBrain (LLM): Gemini 1.5 Flash manages dialogue flow, assesses learner level, and invokes database tools.\nText-to-Speech: Murf Falcon synthesizes natural, speech-tuned Hinglish responses.\nData & State Layer: A local SQLite database (shiksha_vani_memory.db) storing learner profiles, call analytics, and human escalation tickets.\n5. Key Features\nIndian Accents & Code-Mixing: Supports Hinglish code-mixing. If a learner says \"Mujhe English grammar seekhna hai,\" the agent replies in a corresponding encouraging tone: \"Bohot achha! Hinglish me baat karke English seekhna easy ho jata hai. Let's practice!\"\nSQLite Memory & Consent-Before-Save: The agent checks if the user has spoken before (lookup_learner). It remembers names and previous topics but strictly requests consent before saving updates (\"क्या मैं यह याद रख लूँ अगली बार के लिए?\").\nAPI Tool Chaining: Integrates a live dictionary and weather service. The agent checks the learner's district (e.g., Patna), fetches live weather via Open-Meteo, and creates a localized translation challenge (\"The weather in Patna is rainy and 29 degrees today. Can you translate this to Hindi?\").\nTelephony & Outbound Retries: Initiates automated practice calls via LiveKit SIP. If the callee is busy (SIP 486/603) or doesn't answer (SIP 408/480), the agent logs the outcome and schedules retries in SQLite.\nHuman-in-the-Loop & Consent Escalation: If a learner expresses distress (\"English is too hard, I want to quit\"), the agent requests permission to alert a human teacher, redacts any sensitive PII (like credit cards, PINs, or bank details), and files a ticket with a date-based Reference ID (HF-YYYYMMDD-XXX).\nMulti-Agent Handoff: If a user shifts from practicing English to asking about order returns from a sponsor store, the agent cleanly transfers the call to a CommerceSpecialistAgent using LiveKit's dynamic agent update method, returning them when done.\n6. The Hard Parts\nChallenge 1: Windows Path Length Limits on External Packages\nProblem: While installing Twilio and LiveKit dependencies inside the Python environment, the build failed due to the Windows 260-character file path limit.\nInvestigation: Long nested namespaces in the dependencies exceeded Windows' default path thresholds.\nSolution: Moved dependencies to a local self-contained libs directory under backend/src/libs and appended it to the system path at runtime using:\npython\n\ncurrent_dir = os.path.dirname(os.path.abspath(__file__))\nlibs_path = os.path.join(current_dir, \"backend\", \"src\", \"libs\")\nif libs_path not in sys.path:\n    sys.path.insert(0, libs_path)\nLesson: When deploying voice agents on Windows, always budget for path limit workarounds or containerize your backend early.\nChallenge 2: SSL Certification Errors on Public API Calls\nProblem: In the middle of dictionary and weather tool calling, Python's urllib crashed with SSL: CERTIFICATE_VERIFY_FAILED errors on Windows.\nInvestigation: Python on Windows does not automatically use the OS's root certificates, causing SSL handshakes with public endpoints (api.dictionaryapi.dev) to fail.\nSolution: Configured a custom SSL context bypass directly in the lookup function:\npython\n\nimport ssl\nctx = ssl.create_default_context()\nctx.check_hostname = False\nctx.verify_mode = ssl.CERT_NONE\n# Pass ctx to urllib.request.urlopen(req, context=ctx)\nLesson: External tool calls must fail gracefully. Bypassing validation (in dev) or bundling certifi (in prod) is essential to avoid blocking the voice loop.\n7. How You Can Build One\nTo build your own agent, follow the real-time audio pipeline flow:\n\nTransport: Set up a WebRTC session using LiveKit.\nSTT: Capture user input and transcribe it with a model that supports multi-lingual/code-mixed audio.\nLLM: Build system prompts that enforce short, voice-friendly replies (max 1–2 sentences) and structure your functions as tools.\nTTS: Connect to Murf Falcon to generate fast, low-latency audio chunks.\nEnvironment Setup\nCreate a .env.local file in the root directory:\n\nenv\n\nMURF_API_KEY=your_murf_api_key_here\nLIVEKIT_URL=your_livekit_url_here\nLIVEKIT_API_KEY=your_livekit_api_key_here\nLIVEKIT_API_SECRET=your_livekit_api_secret_here\n# Optional Telephony Config\nSIP_TRUNK_ID=your_sip_trunk_id\nTWILIO_ACCOUNT_SID=your_twilio_sid\nTWILIO_AUTH_TOKEN=your_twilio_token\nIMPORTANT\n\nNever commit your .env.local or .env files to git. Add them to your .gitignore to prevent exposing API keys.\n\n8. Running the Project\nClone the repository:\nbash\n\ngit clone [YOUR_GITHUB_REPOSITORY_URL]\ncd voice-for-bharat-challenge-2026\nInstall dependencies:\nbash\n\npip install -r requirements.txt\nRun the web simulator (REST & Web interface):\nbash\n\npython web_demo.py\nRun the LiveKit Voice Agent pipeline:\nbash\n\npython backend/src/agent.py dev\n9. Testing the Agent\nOpen http://localhost:8085 in your browser.\nSelect a voice accent (e.g., \"Kabir\").\nClick Connect and allow microphone access.\nTest Case 1 (Hinglish Greeting): Say \"Hello, mera naam Rohan hai.\"\nExpected behavior: The agent greets you by name and asks if it can save your profile. Click Accept or say \"Yes\".\nTest Case 2 (Tool Calling): Say \"Define perseverance.\"\nExpected behavior: The agent returns the definition, and a dictionary card slides down on the screen.\nTest Case 3 (Guardrail Refusal): Say \"Write a 500-word essay for my homework.\"\nExpected behavior: The agent refuses to write the essay, and guides you to construct sentences yourself.\n10. What I Would Improve Next\nOn-Device VAD Tuning: Fine-tune Silero VAD parameters to better handle ambient background noise in loud classroom settings.\nOffline/On-Device TTS Fallback: Cache common voice prompts locally to ensure standard greetings can play instantly even during internet drops.\n11. Repository and Demo\nCode Repository: [YOUR_GITHUB_REPOSITORY_URL]\nLive Web Demo: [YOUR_DEMO_URL] (if applicable)\n12. What I Learned\nBuilding Shiksha Vani taught me that engineering a voice agent is vastly different from building a chatbot. Turn-taking, speech pacing, VAD tuning, and TTS latency (~55ms with Murf Falcon) represent the difference between a natural human conversation and a disjointed, frustrating interaction. Designing with guardrails, strict consent checks, and robust fallbacks is crucial for building systems that Indian learners can trust.\n\n![ ](https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/xijo98gkep6qtr51z48s.png)\n```\n\n", "url": "https://wpnews.pro/news/shiksha-vani-shikssaa-vaannii-building-a-voice-ai-literacy-agent-with-murf", "canonical_source": "https://dev.to/mamta_sahu_/shiksha-vani-shikssaa-vaannii-building-a-voice-ai-literacy-agent-with-murf-falcon-535c", "published_at": "2026-08-15 16:33:07+00:00", "updated_at": "2026-08-15 17:12:12.910796+00:00", "lang": "en", "topics": ["artificial-intelligence", "natural-language-processing", "ai-agents", "ai-products", "developer-tools"], "entities": ["Shiksha Vani", "Murf Falcon", "LiveKit", "Deepgram Nova-3", "Gemini 1.5 Flash", "Open-Meteo", "SQLite"], "alternates": {"html": "https://wpnews.pro/news/shiksha-vani-shikssaa-vaannii-building-a-voice-ai-literacy-agent-with-murf", "markdown": "https://wpnews.pro/news/shiksha-vani-shikssaa-vaannii-building-a-voice-ai-literacy-agent-with-murf.md", "text": "https://wpnews.pro/news/shiksha-vani-shikssaa-vaannii-building-a-voice-ai-literacy-agent-with-murf.txt", "jsonld": "https://wpnews.pro/news/shiksha-vani-shikssaa-vaannii-building-a-voice-ai-literacy-agent-with-murf.jsonld"}}