Language is meant to be spoken, not just read.
For millions of students and job seekers in India, being confident in English and basic math can open the door to better career opportunities. But many learners don't have access to a personal tutor. Even when they do, the fear of making mistakes in front of someone can make practicing uncomfortable.
So I built Bhasha Academy, an AI-powered voice tutor designed to give learners a private, judgment-free space to practice.
It can have conversations in Hinglish, help users practice English, solve basic math problems, remember learning progress, and even connect the learner with a human teacher when needed.
The interesting part is that the entire experience is voice-first.
Here's how I built it.
I wanted Bhasha Academy to feel less like talking to a chatbot and more like talking to a patient tutor.
The learner should be able to simply say:
"Mujhe English practice karni hai."
And the agent should understand the Hinglish, respond naturally, and continue the conversation.
For math, the learner can say:
"Samar, 25 percent of 200 kitna hai?"
The system can then switch to a math specialist and continue from the same conversation.
The main goals were:
The system is built using several services, with LiveKit sitting at the center of the real-time voice experience.
User
↓
Browser / SIP
↓
LiveKit
↓
Speech-to-Text
↓
Deepgram Nova-3
↓
Gemini
↓
Agent / Tools
↓
Murf Falcon TTS
↓
LiveKit
↓
User
There are also a few supporting components:
SQLite
├── Student profiles
├── Learning progress
├── Call analytics
└── Escalation tickets
Discord Webhook
└── Human teacher notifications
Next.js
└── Admin dashboard
I use Deepgram Nova-3 for speech recognition.
The important part here is multilingual support.
A learner might say:
"Yesterday I went market, but mujhe wahan kuch samajh nahi aaya."
The system needs to understand both languages without forcing the user to speak only English or only Hindi.
For the conversational layer, I use Google Gemini 3.5 Flash Lite.
The model is responsible for:
For voice generation, I use Murf Falcon TTS through LiveKit.
Bhasha Academy currently uses two voices:
Both are designed to sound natural for Indian users.
LiveKit Agents SDK handles the real-time communication layer.
It takes care of things like:
This makes it possible to have a real conversation instead of the traditional:
Record → Upload → Wait → Get Response → Play
Instead, audio can be streamed continuously.
One of the first problems I noticed was surprisingly simple.
The AI understood Hinglish perfectly, but the voice didn't always pronounce it naturally.
For example, if the model generated:
Bahut achha, let's try again!
the TTS system could interpret "Bahut achha" using English pronunciation.
The result sounded robotic.
I changed the system prompt so that Hindi words should be written in Devanagari.
Instead of:
Bahut achha, let's try again!
the model generates:
बहुत अच्छा, let's try again!
This made a huge difference.
Murf Falcon can handle mixed scripts, so it can naturally switch between Hindi and English.
For example:
बहुत बढ़िया! Let's try another word.
This is a small prompt change, but it had a big impact on the voice experience.
A tutor should remember its students.
Bhasha Academy uses SQLite to store learner profiles.
When someone calls for the first time, the agent doesn't automatically save their information.
Instead, it asks for permission.
For example:
"Would you like me to remember your name and learning progress for your next session?"
If the learner agrees, the system can store information such as:
There is also a forget_caller
tool.
A learner can ask the system to forget them, and their stored profile can be removed.
This was important to me because personalization shouldn't come at the cost of user control.
Another interesting part of the project is the specialist handoff system.
Instead of making one huge agent handle everything, I created separate agents.
Anisha is the general language tutor.
She handles:
Samar is the math specialist.
He handles:
If the learner says:
"Can we do some maths?"
Anisha can transfer the conversation to Samar.
The important part is that Samar shouldn't start from zero.
He should already know who the learner is and what was discussed.
The handoff looks roughly like this:
@function_tool
async def transfer_to_math_specialist(
self,
context: RunContext
) -> tuple[Agent, str]:
math_agent = MathPracticeAgent(
chat_ctx=self.chat_ctx.copy(
exclude_instructions=True
)
)
return (
math_agent,
"Transferring you to Samar, our maths practice specialist."
)
The key idea is passing a copy of the existing ChatContext
.
This preserves the conversation while allowing the new agent to have its own instructions and personality.
Bhasha Academy isn't limited to the browser.
Using LiveKit SIP integrations, the system can also make outbound calls.
This opens up the possibility of scheduled lessons.
For example:
Scheduled lesson
↓
System calls student
↓
Student answers
↓
AI tutor starts lesson
But there is another problem with outbound calling: voicemail.
There is no reason to keep expensive AI services running when the call is answered by an answering machine.
So I added voicemail detection.
If the system detects a typical voicemail greeting, it can:
hang_up
tool.AI shouldn't try to solve every problem.
If a learner repeatedly struggles or becomes frustrated, Bhasha Academy can ask whether they want to speak with a human teacher.
If the learner agrees, the system creates an escalation ticket.
The ticket contains information such as:
Reference ID
Student Name
Urgency
Reason
A Discord webhook then sends the notification to the teacher/admin channel.
For example:
payload = {
"embeds": [{
"title": f"🚨 Human Help Request Raised ({urgency.upper()})",
"color": 15158332 if urgency == "emergency" else 3447003,
"fields": [
{
"name": "Reference ID",
"value": ref_id,
"inline": True
},
{
"name": "Student Name",
"value": name,
"inline": True
},
{
"name": "Urgency",
"value": urgency,
"inline": True
},
{
"name": "Reason",
"value": reason,
"inline": False
}
]
}]
}
The idea is simple:
AI handles the routine conversations. Humans step in when the learner needs more help.
Every call is also logged.
The system tracks things like:
The data is stored in SQLite and exposed through a Next.js dashboard.
This makes it easier to understand how the system is performing instead of relying only on individual conversations.
Getting a good text response is only one part of the problem.
A voice agent also needs:
Small improvements in any of these areas can make the experience feel much more natural.
Hinglish isn't simply English with Hindi words.
The way those words are written can directly affect how the TTS system pronounces them.
Switching Hindi words to Devanagari was one of the simplest and most effective improvements I made.
Agent handoffs sound simple:
Agent A → Agent B
But without preserving context, it becomes:
Agent A → Agent B → "What's your name?"
Passing the conversation context makes the handoff feel like one continuous conversation.
A good tutor isn't necessarily one that answers everything.
Sometimes the best action is:
"Would you like me to connect you with a teacher?"
That human fallback makes the system more useful and trustworthy.
The project is based on the open-source Murf LiveKit Starter repository.
Repository:
https://github.com/bharatbushan03/murf-livekit-starter
You'll need:
uv
pnpm
git clone https://github.com/bharatbushan03/murf-livekit-starter.git
cd murf-livekit-starter
Install the backend:
cd backend
uv sync
uv run python src/agent.py download-files
Install the frontend:
cd ../frontend
pnpm install
Configure your environment variables with the required API credentials:
LIVEKIT_URL
LIVEKIT_API_KEY
LIVEKIT_API_SECRET
MURF_API_KEY
DEEPGRAM_API_KEY
GOOGLE_API_KEY
DISCORD_WEBHOOK_URL
The Discord webhook is optional.
cd backend
uv run python src/agent.py dev
Then start the frontend in another terminal:
cd frontend
pnpm dev
Open:
http://localhost:3000
Click Start Learning with Anisha and start talking.
There are still several things I want to improve.
Real users aren't always sitting in a quiet room.
A student might be practicing from:
I want to tune the voice activity detection system to work better in noisy environments.
Another feature I'm planning is real-time math progress.
While Samar is teaching, the frontend could show something like:
Math Practice
Questions: 8
Correct: 6
Needs Practice: 2
Topic:
Percentages
This would make the voice interaction feel more connected to the visual interface.
Building Bhasha Academy taught me that voice AI can be much more than a voice chatbot.
When you combine:
you can start building something that feels closer to a real tutor.
There are still many things to improve, but the core idea is simple:
Give learners a patient tutor they can talk to whenever they want, without being afraid of making mistakes.
That's what I'm trying to build with Bhasha Academy.
If you're building something similar with LiveKit, Murf, Deepgram, or Gemini, I'd love to hear what you're working on.