Over the past 10 days, as part of the #VoiceForBharat 10 Days of Voice Agents Challenge, I built Vidya — a real-time, bilingual (English/Hindi) AI Voice Tutor designed to make learning interactive, accessible, and human-like for students across India.
In this post, I’ll share the story of how Vidya came to life, dive into the architecture behind ultra-low latency voice agents, highlight the key features built over the 10 days, discuss the toughest challenges faced, and walk you through building your own production-ready voice agent using Murf Falcon TTS, LiveKit Agents, Deepgram STT, and Google Gemini.
In India, text-based educational platforms often face a steep digital literacy and language barrier. Millions of learners feel intimidated by typing long queries or struggling through English-only user interfaces. Voice unlocks immediate, natural, and hands-free learning — allowing students to speak naturally in English, Hindi, or code-mixed Hinglish.
Vidya serves as a personal AI learning companion:
Over 10 intensive days, Vidya grew from a basic echo bot into a multi-agent system equipped with tools, memory, telephony, analytics, and safety guardrails:
Using Murf Falcon TTS (livekit-murf
), Vidya speaks with a natural, conversational Indian voice (Anisha
). Streaming TTS with sentence tokenization (min_sentence_len=2
) and text pacing delivers speech chunks with sub-second latency, giving the agent a human-like flow.
Powered by Deepgram Nova-3 STT (language="multi"
) and Google Gemini, Vidya fluently handles English, Hindi in Devanagari script, and code-mixed Hinglish phrases (e.g., "Namaste! Aaj hum beginner reading practice karenge.").
Vidya remembers returning students! Using a persistent profile store (user_store.py
), Vidya recalls the user's name, preferred language, current learning level, and last interaction date, greeting them warmly:
"Namaste Aarav, welcome back! You were working on beginner exercises. Last seen on August 14."
Vidya is equipped with specialized function tools:
fetch_next_exercise
: Retrieves level-appropriate practice prompts tagged with data freshness timestamps (last_updated
).score_spoken_answer
: Evaluates spoken pronunciations and answers on a 0–100 scale.award_learning_star
: Awards virtual gold stars (🌟) to keep learners motivated.scrape_website
: Fetches live web pages in real-time (web_scraper.py
) for live context extraction.Integrated with LiveKit's SIP Trunking (telephony/outbound/dial.py
), Vidya can initiate active outbound phone calls to learners' mobile phones for daily study check-ins and practice sessions.
If a learner is stuck, frustrated, or requests a human teacher, create_escalation
logs an escalation ticket (ESC-12345
) and alerts support staff. Vidya follows strict privacy guardrails — asking for explicit permission before saving any personal details or submitting tickets.
Session outcomes are tracked in call_store.py
— logging call duration, agent type (browser vs. telephony), completion status (successful
/ failed
), and success reasons (exercise_scored
, star_awarded
, escalated_to_human
).
When a student asks a physics question (e.g., "Why does an apple fall from a tree?"), Vidya seamlessly hands off the conversation to Dr. Homi (Physics Specialist) using LiveKit's context.session.update_agent()
. When physics practice ends, Dr. Homi hands the student back to Vidya for reading practice!
Building a real-time voice agent isn't just about linking APIs together. Here are three major hurdles faced and solved:
min_sentence_len=2
).session.update_agent()
combined with WebRTC data channel events (agent_handoff
) to update the Next.js frontend UI live without dropping the WebRTC room session.
+------------------+ WebRTC Audio Stream +---------------------+
| | -------------------------> | Deepgram Nova-3 |
| Learner / UI | | Streaming STT |
| (Next.js App) | <------------------------- +----------+----------+
+--------+---------+ Real-time Audio Out |
^ v
| RTC Data Channel +---------------------+
| (State & Handoffs) | Google Gemini LLM |
| | (Flash Lite Model) |
+-------------------------------------- +----------+----------+
|
v
+---------------------+
| Murf Falcon TTS |
| (Streaming Indian) |
+---------------------+
agent.py
)
from livekit.agents import AgentSession, AgentServer, room_io
from livekit.plugins import deepgram, google, murf, silero, noise_cancellation
session = AgentSession(
stt=deepgram.STT(model="nova-3", language="multi"),
llm=google.LLM(model="gemini-3.5-flash-lite"),
tts=murf.TTS(
voice="Anisha", # Murf Falcon Indian accent voice
style="Conversation",
tokenizer=tokenize.basic.SentenceTokenizer(min_sentence_len=2),
text_pacing=True,
),
turn_detection=MultilingualModel(),
vad=ctx.proc.userdata["vad"],
preemptive_generation=True,
)
agent.py
)
@function_tool
async def transfer_to_physics_specialist(self, context: RunContext, reason: str) -> str:
"""Hand off the conversation to Dr. Homi when user asks physics questions."""
logger.info("Handing off conversation to PhysicsSpecialist. Reason: %s", reason)
specialist = PhysicsSpecialist()
context.session.update_agent(specialist)
payload = json.dumps({
"type": "agent_handoff",
"from_agent": "Vidya (Literacy Tutor)",
"to_agent": "Dr. Homi (Physics Specialist)",
"message": "🔄 Switched conversation to Physics Specialist (Dr. Homi)"
})
await context.room.local_participant.publish_data(payload=payload.encode("utf-8"))
return "I will connect you to our physics specialist."
Want to build your own voice AI agent? You can clone and run our open-source repository in minutes!
git clone https://github.com/hotokeAtlast/murf-livekit-starter.git
cd murf-livekit-starter
Copy backend/.env.example
to backend/.env.local
and fill in your keys:
LIVEKIT_URL=wss://your-livekit-project.livekit.cloud
LIVEKIT_API_KEY=your_key
LIVEKIT_API_SECRET=your_secret
MURF_API_KEY=your_murf_api_key
DEEPGRAM_API_KEY=your_deepgram_api_key
GOOGLE_API_KEY=your_google_gemini_api_key
cd backend
uv sync
uv run python src/agent.py dev
In a new terminal:
cd frontend
pnpm install
pnpm dev
Open http://localhost:3000
, click Connect, and start talking to your voice agent!