Building Vidya: An Ultra-Fast Bilingual Voice AI Tutor with Murf Falcon & LiveKit (10 Days of Voice Agents)

A developer built Vidya, a real-time bilingual (English/Hindi) AI voice tutor, as part of the #VoiceForBharat 10 Days of Voice Agents Challenge. The system uses Murf Falcon TTS, LiveKit Agents, Deepgram STT, and Google Gemini to enable ultra-low latency, natural conversations, with features like memory, tools, telephony, and safety guardrails. The project demonstrates a production-ready voice agent architecture for educational use in India.

Over the past 10 days, as part of the VoiceForBharat 10 Days of Voice Agents Challenge , I built Vidya — a real-time, bilingual English/Hindi AI Voice Tutor designed to make learning interactive, accessible, and human-like for students across India. In this post, I’ll share the story of how Vidya came to life, dive into the architecture behind ultra-low latency voice agents, highlight the key features built over the 10 days, discuss the toughest challenges faced, and walk you through building your own production-ready voice agent using Murf Falcon TTS , LiveKit Agents , Deepgram STT , and Google Gemini . In India, text-based educational platforms often face a steep digital literacy and language barrier. Millions of learners feel intimidated by typing long queries or struggling through English-only user interfaces. Voice unlocks immediate, natural, and hands-free learning — allowing students to speak naturally in English, Hindi, or code-mixed Hinglish . Vidya serves as a personal AI learning companion: Over 10 intensive days, Vidya grew from a basic echo bot into a multi-agent system equipped with tools, memory, telephony, analytics, and safety guardrails: Using Murf Falcon TTS livekit-murf , Vidya speaks with a natural, conversational Indian voice Anisha . Streaming TTS with sentence tokenization min sentence len=2 and text pacing delivers speech chunks with sub-second latency, giving the agent a human-like flow. Powered by Deepgram Nova-3 STT language="multi" and Google Gemini , Vidya fluently handles English, Hindi in Devanagari script, and code-mixed Hinglish phrases e.g., "Namaste Aaj hum beginner reading practice karenge." . Vidya remembers returning students Using a persistent profile store user store.py , Vidya recalls the user's name, preferred language, current learning level, and last interaction date, greeting them warmly: "Namaste Aarav, welcome back You were working on beginner exercises. Last seen on August 14." Vidya is equipped with specialized function tools: fetch next exercise : Retrieves level-appropriate practice prompts tagged with data freshness timestamps last updated . score spoken answer : Evaluates spoken pronunciations and answers on a 0–100 scale. award learning star : Awards virtual gold stars 🌟 to keep learners motivated. scrape website : Fetches live web pages in real-time web scraper.py for live context extraction.Integrated with LiveKit's SIP Trunking telephony/outbound/dial.py , Vidya can initiate active outbound phone calls to learners' mobile phones for daily study check-ins and practice sessions. If a learner is stuck, frustrated, or requests a human teacher, create escalation logs an escalation ticket ESC-12345 and alerts support staff. Vidya follows strict privacy guardrails — asking for explicit permission before saving any personal details or submitting tickets. Session outcomes are tracked in call store.py — logging call duration, agent type browser vs. telephony , completion status successful / failed , and success reasons exercise scored , star awarded , escalated to human . When a student asks a physics question e.g., "Why does an apple fall from a tree?" , Vidya seamlessly hands off the conversation to Dr. Homi Physics Specialist using LiveKit's context.session.update agent . When physics practice ends, Dr. Homi hands the student back to Vidya for reading practice Building a real-time voice agent isn't just about linking APIs together. Here are three major hurdles faced and solved: min sentence len=2 . session.update agent combined with WebRTC data channel events agent handoff to update the Next.js frontend UI live without dropping the WebRTC room session. +------------------+ WebRTC Audio Stream +---------------------+ | | ------------------------- | Deepgram Nova-3 | | Learner / UI | | Streaming STT | | Next.js App | <------------------------- +----------+----------+ +--------+---------+ Real-time Audio Out | ^ v | RTC Data Channel +---------------------+ | State & Handoffs | Google Gemini LLM | | | Flash Lite Model | +-------------------------------------- +----------+----------+ | v +---------------------+ | Murf Falcon TTS | | Streaming Indian | +---------------------+ agent.py python from livekit.agents import AgentSession, AgentServer, room io from livekit.plugins import deepgram, google, murf, silero, noise cancellation session = AgentSession stt=deepgram.STT model="nova-3", language="multi" , llm=google.LLM model="gemini-3.5-flash-lite" , tts=murf.TTS voice="Anisha", Murf Falcon Indian accent voice style="Conversation", tokenizer=tokenize.basic.SentenceTokenizer min sentence len=2 , text pacing=True, , turn detection=MultilingualModel , vad=ctx.proc.userdata "vad" , preemptive generation=True, agent.py python @function tool async def transfer to physics specialist self, context: RunContext, reason: str - str: """Hand off the conversation to Dr. Homi when user asks physics questions.""" logger.info "Handing off conversation to PhysicsSpecialist. Reason: %s", reason specialist = PhysicsSpecialist context.session.update agent specialist Notify Next.js frontend UI via WebRTC data channel payload = json.dumps { "type": "agent handoff", "from agent": "Vidya Literacy Tutor ", "to agent": "Dr. Homi Physics Specialist ", "message": "🔄 Switched conversation to Physics Specialist Dr. Homi " } await context.room.local participant.publish data payload=payload.encode "utf-8" return "I will connect you to our physics specialist." Want to build your own voice AI agent? You can clone and run our open-source repository in minutes git clone https://github.com/hotokeAtlast/murf-livekit-starter.git cd murf-livekit-starter Copy backend/.env.example to backend/.env.local and fill in your keys: LIVEKIT URL=wss://your-livekit-project.livekit.cloud LIVEKIT API KEY=your key LIVEKIT API SECRET=your secret MURF API KEY=your murf api key DEEPGRAM API KEY=your deepgram api key GOOGLE API KEY=your google gemini api key cd backend uv sync uv run python src/agent.py dev In a new terminal: cd frontend pnpm install pnpm dev Open http://localhost:3000 , click Connect , and start talking to your voice agent