{"slug": "building-vidya-an-ultra-fast-bilingual-voice-ai-tutor-with-murf-falcon-livekit", "title": "Building Vidya: An Ultra-Fast Bilingual Voice AI Tutor with Murf Falcon & LiveKit (10 Days of Voice Agents)", "summary": "A developer built Vidya, a real-time bilingual (English/Hindi) AI voice tutor, as part of the #VoiceForBharat 10 Days of Voice Agents Challenge. The system uses Murf Falcon TTS, LiveKit Agents, Deepgram STT, and Google Gemini to enable ultra-low latency, natural conversations, with features like memory, tools, telephony, and safety guardrails. The project demonstrates a production-ready voice agent architecture for educational use in India.", "body_md": "Over the past 10 days, as part of the **#VoiceForBharat 10 Days of Voice Agents Challenge**, I built **Vidya** — a real-time, bilingual (English/Hindi) AI Voice Tutor designed to make learning interactive, accessible, and human-like for students across India.\n\nIn this post, I’ll share the story of how Vidya came to life, dive into the architecture behind ultra-low latency voice agents, highlight the key features built over the 10 days, discuss the toughest challenges faced, and walk you through building your own production-ready voice agent using **Murf Falcon TTS**, **LiveKit Agents**, **Deepgram STT**, and **Google Gemini**.\n\nIn India, text-based educational platforms often face a steep digital literacy and language barrier. Millions of learners feel intimidated by typing long queries or struggling through English-only user interfaces. Voice unlocks immediate, natural, and hands-free learning — allowing students to speak naturally in **English, Hindi, or code-mixed Hinglish**.\n\n**Vidya** serves as a personal AI learning companion:\n\nOver 10 intensive days, Vidya grew from a basic echo bot into a multi-agent system equipped with tools, memory, telephony, analytics, and safety guardrails:\n\nUsing **Murf Falcon TTS** (`livekit-murf`\n\n), Vidya speaks with a natural, conversational Indian voice (`Anisha`\n\n). Streaming TTS with sentence tokenization (`min_sentence_len=2`\n\n) and text pacing delivers speech chunks with sub-second latency, giving the agent a human-like flow.\n\nPowered by **Deepgram Nova-3 STT** (`language=\"multi\"`\n\n) and **Google Gemini**, Vidya fluently handles English, Hindi in Devanagari script, and code-mixed Hinglish phrases (e.g., *\"Namaste! Aaj hum beginner reading practice karenge.\"*).\n\nVidya remembers returning students! Using a persistent profile store (`user_store.py`\n\n), Vidya recalls the user's name, preferred language, current learning level, and last interaction date, greeting them warmly:\n\n\"Namaste Aarav, welcome back! You were working on beginner exercises. Last seen on August 14.\"\n\nVidya is equipped with specialized function tools:\n\n`fetch_next_exercise`\n\n: Retrieves level-appropriate practice prompts tagged with data freshness timestamps (`last_updated`\n\n).`score_spoken_answer`\n\n: Evaluates spoken pronunciations and answers on a 0–100 scale.`award_learning_star`\n\n: Awards virtual gold stars (🌟) to keep learners motivated.`scrape_website`\n\n: Fetches live web pages in real-time (`web_scraper.py`\n\n) for live context extraction.Integrated with LiveKit's SIP Trunking (`telephony/outbound/dial.py`\n\n), Vidya can initiate active outbound phone calls to learners' mobile phones for daily study check-ins and practice sessions.\n\nIf a learner is stuck, frustrated, or requests a human teacher, `create_escalation`\n\nlogs an escalation ticket (`ESC-12345`\n\n) and alerts support staff. Vidya follows strict privacy guardrails — asking for **explicit permission** before saving any personal details or submitting tickets.\n\nSession outcomes are tracked in `call_store.py`\n\n— logging call duration, agent type (browser vs. telephony), completion status (`successful`\n\n/ `failed`\n\n), and success reasons (`exercise_scored`\n\n, `star_awarded`\n\n, `escalated_to_human`\n\n).\n\nWhen a student asks a physics question (e.g., *\"Why does an apple fall from a tree?\"*), Vidya seamlessly hands off the conversation to **Dr. Homi (Physics Specialist)** using LiveKit's `context.session.update_agent()`\n\n. When physics practice ends, Dr. Homi hands the student back to Vidya for reading practice!\n\nBuilding a real-time voice agent isn't just about linking APIs together. Here are three major hurdles faced and solved:\n\n`min_sentence_len=2`\n\n).`session.update_agent()`\n\ncombined with WebRTC data channel events (`agent_handoff`\n\n) to update the Next.js frontend UI live without dropping the WebRTC room session.\n\n```\n+------------------+      WebRTC Audio Stream     +---------------------+\n|                  |  ------------------------->  |  Deepgram Nova-3    |\n|   Learner / UI   |                              |  Streaming STT      |\n|  (Next.js App)   |  <-------------------------  +----------+----------+\n+--------+---------+      Real-time Audio Out                |\n         ^                                                   v\n         | RTC Data Channel                       +---------------------+\n         | (State & Handoffs)                     |  Google Gemini LLM  |\n         |                                        | (Flash Lite Model)  |\n         +--------------------------------------  +----------+----------+\n                                                             |\n                                                             v\n                                                  +---------------------+\n                                                  |  Murf Falcon TTS    |\n                                                  | (Streaming Indian)  |\n                                                  +---------------------+\n```\n\n`agent.py`\n\n)\n\n``` python\nfrom livekit.agents import AgentSession, AgentServer, room_io\nfrom livekit.plugins import deepgram, google, murf, silero, noise_cancellation\n\nsession = AgentSession(\n    stt=deepgram.STT(model=\"nova-3\", language=\"multi\"),\n    llm=google.LLM(model=\"gemini-3.5-flash-lite\"),\n    tts=murf.TTS(\n        voice=\"Anisha\",          # Murf Falcon Indian accent voice\n        style=\"Conversation\",\n        tokenizer=tokenize.basic.SentenceTokenizer(min_sentence_len=2),\n        text_pacing=True,\n    ),\n    turn_detection=MultilingualModel(),\n    vad=ctx.proc.userdata[\"vad\"],\n    preemptive_generation=True,\n)\n```\n\n`agent.py`\n\n)\n\n``` python\n@function_tool\nasync def transfer_to_physics_specialist(self, context: RunContext, reason: str) -> str:\n    \"\"\"Hand off the conversation to Dr. Homi when user asks physics questions.\"\"\"\n    logger.info(\"Handing off conversation to PhysicsSpecialist. Reason: %s\", reason)\n    specialist = PhysicsSpecialist()\n    context.session.update_agent(specialist)\n\n    # Notify Next.js frontend UI via WebRTC data channel\n    payload = json.dumps({\n        \"type\": \"agent_handoff\",\n        \"from_agent\": \"Vidya (Literacy Tutor)\",\n        \"to_agent\": \"Dr. Homi (Physics Specialist)\",\n        \"message\": \"🔄 Switched conversation to Physics Specialist (Dr. Homi)\"\n    })\n    await context.room.local_participant.publish_data(payload=payload.encode(\"utf-8\"))\n\n    return \"I will connect you to our physics specialist.\"\n```\n\nWant to build your own voice AI agent? You can clone and run our open-source repository in minutes!\n\n```\ngit clone https://github.com/hotokeAtlast/murf-livekit-starter.git\ncd murf-livekit-starter\n```\n\nCopy `backend/.env.example`\n\nto `backend/.env.local`\n\nand fill in your keys:\n\n```\nLIVEKIT_URL=wss://your-livekit-project.livekit.cloud\nLIVEKIT_API_KEY=your_key\nLIVEKIT_API_SECRET=your_secret\nMURF_API_KEY=your_murf_api_key\nDEEPGRAM_API_KEY=your_deepgram_api_key\nGOOGLE_API_KEY=your_google_gemini_api_key\ncd backend\nuv sync\nuv run python src/agent.py dev\n```\n\nIn a new terminal:\n\n```\ncd frontend\npnpm install\npnpm dev\n```\n\nOpen `http://localhost:3000`\n\n, click **Connect**, and start talking to your voice agent!", "url": "https://wpnews.pro/news/building-vidya-an-ultra-fast-bilingual-voice-ai-tutor-with-murf-falcon-livekit", "canonical_source": "https://dev.to/stoichotoke/building-vidya-an-ultra-fast-bilingual-voice-ai-tutor-with-murf-falcon-livekit-10-days-of-voice-1fcf", "published_at": "2026-08-15 10:04:02+00:00", "updated_at": "2026-08-15 10:11:53.650396+00:00", "lang": "en", "topics": ["artificial-intelligence", "generative-ai", "ai-products", "ai-agents", "developer-tools"], "entities": ["Vidya", "Murf Falcon", "LiveKit", "Deepgram", "Google Gemini", "Deepgram Nova-3", "LiveKit Agents", "Dr. Homi"], "alternates": {"html": "https://wpnews.pro/news/building-vidya-an-ultra-fast-bilingual-voice-ai-tutor-with-murf-falcon-livekit", "markdown": "https://wpnews.pro/news/building-vidya-an-ultra-fast-bilingual-voice-ai-tutor-with-murf-falcon-livekit.md", "text": "https://wpnews.pro/news/building-vidya-an-ultra-fast-bilingual-voice-ai-tutor-with-murf-falcon-livekit.txt", "jsonld": "https://wpnews.pro/news/building-vidya-an-ultra-fast-bilingual-voice-ai-tutor-with-murf-falcon-livekit.jsonld"}}