Shiksha Vani (शिक्षा वाणी): Building a Voice AI Literacy Agent with Murf Falcon

A developer built Shiksha Vani, a voice AI literacy tutor for spoken English, as part of the 10 Days of Voice Agents challenge. The system uses LiveKit, Deepgram Nova-3, Gemini 1.5 Flash, and Murf Falcon TTS to support Hinglish conversations, with features like SQLite memory, consent-based data saving, and human escalation.

For millions of learners across India, mastering spoken English is a gateway to better educational and career opportunities. However, classroom learning is often passive, and practicing speaking requires a judgment-free partner. Learners face two distinct hurdles: To solve this, I built Shiksha Vani शिक्षा वाणी —an interactive, Hinglish-speaking AI Spoken English and Literacy Tutor. Developed as part of the 10 Days of Voice Agents VoiceForBharat Edition challenge, Shiksha Vani is designed to act as a supportive practice buddy. It listens to a student's speech, corrects their pronunciation and grammar gently, and engages in daily spoken exercises. php mermaid graph TD User User Voice <-- |WebRTC / SIP| LT LiveKit Real-time Transport LT <-- |Audio Streams| Agent Voice Agent Session Agent -- |Speech-to-Text| STT Deepgram Nova-3 Agent -- |LLM Reasoner| LLM Gemini 1.5 Flash Agent -- |Text-to-Speech| TTS Murf Falcon API LLM <-- |Tool Calling| DB SQLite Persistent Memory LLM <-- |Context Handoff| Specialist Returns Specialist Agent 3. Why Voice? Text-based chat apps fail to capture the nuances of speech—pronunciation, hesitation, and cadence. Spoken interaction is essential for building verbal confidence. In India, where mobile internet is ubiquitous but literacy rates vary, voice is the most natural, accessible, and frictionless interface. By utilizing Murf Falcon, a high-fidelity, low-latency Text-to-Speech TTS engine, the agent communicates in warm, authentic Indian accents like the default hi-IN-kabir . This minimizes the intimidation barrier often associated with dry, robotic western accents. 4. How the System Works The architecture utilizes a real-time media transport layer combined with decoupled cognitive and speech engines: Frontend: A responsive glassmorphic dashboard index.html featuring real-time visualizers, call statistics, and interactive controls. Real-time Transport: LiveKit WebRTC serves as the audio pipeline, handling sub-100ms bidirectional streaming. Speech-to-Text: Deepgram Nova-3 translates Hinglish voice inputs to text. Brain LLM : Gemini 1.5 Flash manages dialogue flow, assesses learner level, and invokes database tools. Text-to-Speech: Murf Falcon synthesizes natural, speech-tuned Hinglish responses. Data & State Layer: A local SQLite database shiksha vani memory.db storing learner profiles, call analytics, and human escalation tickets. 5. Key Features Indian Accents & Code-Mixing: Supports Hinglish code-mixing. If a learner says "Mujhe English grammar seekhna hai," the agent replies in a corresponding encouraging tone: "Bohot achha Hinglish me baat karke English seekhna easy ho jata hai. Let's practice " SQLite Memory & Consent-Before-Save: The agent checks if the user has spoken before lookup learner . It remembers names and previous topics but strictly requests consent before saving updates "क्या मैं यह याद रख लूँ अगली बार के लिए?" . API Tool Chaining: Integrates a live dictionary and weather service. The agent checks the learner's district e.g., Patna , fetches live weather via Open-Meteo, and creates a localized translation challenge "The weather in Patna is rainy and 29 degrees today. Can you translate this to Hindi?" . Telephony & Outbound Retries: Initiates automated practice calls via LiveKit SIP. If the callee is busy SIP 486/603 or doesn't answer SIP 408/480 , the agent logs the outcome and schedules retries in SQLite. Human-in-the-Loop & Consent Escalation: If a learner expresses distress "English is too hard, I want to quit" , the agent requests permission to alert a human teacher, redacts any sensitive PII like credit cards, PINs, or bank details , and files a ticket with a date-based Reference ID HF-YYYYMMDD-XXX . Multi-Agent Handoff: If a user shifts from practicing English to asking about order returns from a sponsor store, the agent cleanly transfers the call to a CommerceSpecialistAgent using LiveKit's dynamic agent update method, returning them when done. 6. The Hard Parts Challenge 1: Windows Path Length Limits on External Packages Problem: While installing Twilio and LiveKit dependencies inside the Python environment, the build failed due to the Windows 260-character file path limit. Investigation: Long nested namespaces in the dependencies exceeded Windows' default path thresholds. Solution: Moved dependencies to a local self-contained libs directory under backend/src/libs and appended it to the system path at runtime using: python current dir = os.path.dirname os.path.abspath file libs path = os.path.join current dir, "backend", "src", "libs" if libs path not in sys.path: sys.path.insert 0, libs path Lesson: When deploying voice agents on Windows, always budget for path limit workarounds or containerize your backend early. Challenge 2: SSL Certification Errors on Public API Calls Problem: In the middle of dictionary and weather tool calling, Python's urllib crashed with SSL: CERTIFICATE VERIFY FAILED errors on Windows. Investigation: Python on Windows does not automatically use the OS's root certificates, causing SSL handshakes with public endpoints api.dictionaryapi.dev to fail. Solution: Configured a custom SSL context bypass directly in the lookup function: python import ssl ctx = ssl.create default context ctx.check hostname = False ctx.verify mode = ssl.CERT NONE Pass ctx to urllib.request.urlopen req, context=ctx Lesson: External tool calls must fail gracefully. Bypassing validation in dev or bundling certifi in prod is essential to avoid blocking the voice loop. 7. How You Can Build One To build your own agent, follow the real-time audio pipeline flow: Transport: Set up a WebRTC session using LiveKit. STT: Capture user input and transcribe it with a model that supports multi-lingual/code-mixed audio. LLM: Build system prompts that enforce short, voice-friendly replies max 1–2 sentences and structure your functions as tools. TTS: Connect to Murf Falcon to generate fast, low-latency audio chunks. Environment Setup Create a .env.local file in the root directory: env MURF API KEY=your murf api key here LIVEKIT URL=your livekit url here LIVEKIT API KEY=your livekit api key here LIVEKIT API SECRET=your livekit api secret here Optional Telephony Config SIP TRUNK ID=your sip trunk id TWILIO ACCOUNT SID=your twilio sid TWILIO AUTH TOKEN=your twilio token IMPORTANT Never commit your .env.local or .env files to git. Add them to your .gitignore to prevent exposing API keys. 8. Running the Project Clone the repository: bash git clone YOUR GITHUB REPOSITORY URL cd voice-for-bharat-challenge-2026 Install dependencies: bash pip install -r requirements.txt Run the web simulator REST & Web interface : bash python web demo.py Run the LiveKit Voice Agent pipeline: bash python backend/src/agent.py dev 9. Testing the Agent Open http://localhost:8085 in your browser. Select a voice accent e.g., "Kabir" . Click Connect and allow microphone access. Test Case 1 Hinglish Greeting : Say "Hello, mera naam Rohan hai." Expected behavior: The agent greets you by name and asks if it can save your profile. Click Accept or say "Yes". Test Case 2 Tool Calling : Say "Define perseverance." Expected behavior: The agent returns the definition, and a dictionary card slides down on the screen. Test Case 3 Guardrail Refusal : Say "Write a 500-word essay for my homework." Expected behavior: The agent refuses to write the essay, and guides you to construct sentences yourself. 10. What I Would Improve Next On-Device VAD Tuning: Fine-tune Silero VAD parameters to better handle ambient background noise in loud classroom settings. Offline/On-Device TTS Fallback: Cache common voice prompts locally to ensure standard greetings can play instantly even during internet drops. 11. Repository and Demo Code Repository: YOUR GITHUB REPOSITORY URL Live Web Demo: YOUR DEMO URL if applicable 12. What I Learned Building Shiksha Vani taught me that engineering a voice agent is vastly different from building a chatbot. Turn-taking, speech pacing, VAD tuning, and TTS latency ~55ms with Murf Falcon represent the difference between a natural human conversation and a disjointed, frustrating interaction. Designing with guardrails, strict consent checks, and robust fallbacks is crucial for building systems that Indian learners can trust. https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/xijo98gkep6qtr51z48s.png