#VoiceForBharat| Built with the fastest TTS API — Murf Falcon | 10 Days of Voice Agents
Ten days ago, I started with a blank repo and a challenge: build a production-ready voice AI agent for Indian users — one that could speak naturally in English, Hindi, and Hinglish; remember returning callers; escalate to humans when things got serious; and hand off conversations to specialist agents without ever making the caller repeat themselves.
What came out the other side is RupeeGPT — a conversational AI financial assistant that helps any Indian user navigate banking, UPI, government welfare schemes, loans, and financial safety. Here's everything I built, what broke, how I fixed it, and how you can build your own.
India has over 500 million smartphone users, but financial literacy remains a barrier for hundreds of millions of people — especially in tier-2 and tier-3 cities and rural areas. The information exists: government scheme portals, RBI guidelines, banking apps. But it's buried in bureaucratic language, English-only interfaces, and long PDF documents.
A voice agent changes that. You don't need to read anything. You don't need to know the right portal URL. You just talk.
Who it's for: First-generation bank account holders, rural farmers checking PM Kisan eligibility, street vendors exploring PM SVANidhi loans, anyone who's ever been told to "read the fine print" and couldn't.
Why voice: Voice meets people where they are. It removes the literacy barrier, it's faster than navigating apps, and for many rural users, calling is the most intuitive interface they know.
🎙️ User speaks
→ Deepgram STT (nova-3, multilingual)
→ Gemini LLM (gemini-3.5-flash-lite via LiveKit Inference)
→ Murf Falcon TTS (Anisha — Indian English, en-IN)
→ LiveKit real-time transport
→ 🔊 User hears
The stack:
uv
for dependency managementThe default for most TTS-backed voice agents is a US English voice. For an Indian user asking about PM Kisan Samman Nidhi, hearing a generic American accent reading scheme names in English phonetics feels jarring and impersonal.
Murf Falcon's Anisha voice — Indian English, en-IN
, Conversation style — changes this completely. But there was a subtlety: even with an Indian voice, scheme names like "PM Kisan Samman Nidhi" or "Pradhan Mantri Jan Dhan Yojana" are read with English phonetics when spelled in Roman script.
My fix: a TTS pronunciation layer (tts_hindi.py
)
_ENGLISH_TO_HINDI: tuple[tuple[str, str], ...] = (
("pm kisan samman nidhi", "पीएम किसान सम्मान निधि"),
("pm jan dhan yojana", "पीएम जन धन योजना"),
("pradhan mantri jan dhan yojana", "प्रधानमंत्री जन धन योजना"),
("pm svanidhi", "पीएम स्वनिधि"),
("aadhaar", "आधार"),
("yojana", "योजना"),
)
Before any text reaches Murf Falcon, it passes through this whitelist rewriter. Known Hindi/Indian terms are converted to Devanagari, so the voice says "पीएम किसान सम्मान निधि" — exactly as a native speaker would say it on TV — instead of "P M Kisan Samman Nidhi" with English stress patterns.
The rewriter is safe to apply for every language mode: a pure-English sentence with none of these terms passes through byte-for-byte unchanged. I also built a detect_language()
function that classifies each user utterance as english
, hindi
, or hinglish
using Devanagari character detection and a curated Hinglish marker word list:
HINGLISH_MARKERS = ("mujhe", "kaise", "kya", "chahiye", "baat", "namaste",
"yojana", "sarkari", "paise", "rupaye", "bharat", ...)
The agent mirrors the caller's language — answers in Hindi if they speak Hindi, Hinglish if they code-switch — without ever asking them to repeat.
The system prompt defines the entire character of RupeeGPT: what it will help with, what it refuses, and how it escalates.
Key guardrails baked into the system prompt:
For the two escalation scenarios, the agent must:
create_escalation()
only after consentThis pattern — ask before acting, require a clear YES — became a design principle throughout the whole project.
The TTS node hooks into the LiveKit Agents pipeline using Agent.default.tts_node
:
async def tts_node(self, text, model_settings):
language = self._tts_language()
async def _tracked():
async for part in tts_hindi.stream_for_tts(text, language=language):
yield part
async for frame in Agent.default.tts_node(self, _tracked(), model_settings):
yield frame
The stream_for_tts
function accumulates the LLM's streaming text output into complete sentences before passing each sentence through the Devanagari rewriter. This is important: if a scheme name like "PM Kisan Samman Nidhi" were split across two streamed chunks, the phrase-level rewriter would miss it.
Every caller gets a persistent browser ID (stored in localStorage
and passed as a LiveKit participant attribute). The agent reads this at the start of every session and calls lookup_user()
to fetch any saved profile from MongoDB.
But here's the part that took the most iteration: consent architecture.
The agent is not allowed to save any personal fact without:
grant_user_memory_consent()
with that exact valuesave_user_memory()
with the same value
The save_user_memory
tool actively checks the in-session consent store and blocks saves for anything that wasn't consented to in the current call. Returning callers are greeted naturally: "Namaste Rahul, welcome back. Would you like to continue from PM Jan Dhan Yojana?"
The MongoDB document looks like this:
{
"user_id": "abc123",
"name": "Rahul",
"language_preference": "Hinglish",
"facts": {
"schemes_checked": ["PM Jan Dhan Yojana"],
"eligibility_answers": { "income_bracket": "below 3 lakh", "farmer": true }
},
"last_interaction": "2026-08-14T10:30:00Z"
}
Three function-calling tools give the agent live (or near-live) data:
** find_eligible_schemes** — Matches the caller's profile (age, state, income, occupation, caste, residence, disability, BPL status) against a local dataset of Indian government welfare schemes. Returns preliminary matches with names, benefits, documents required, and official portal URLs.
** get_usd_inr_rate** — Live USD/INR exchange rate.
** get_lending_rates** — Current base lending rates and MCLR data.
The scheme-matching tool uses a careful LLM prompt to avoid hallucination:
Using LiveKit's SIP integration, the agent can place outbound calls to real phone numbers. The session pipeline automatically detects SIP participants and switches the noise cancellation model:
noise_cancellation=lambda params: (
noise_cancellation.BVCTelephony()
if params.participant.kind == rtc.ParticipantKind.PARTICIPANT_KIND_SIP
else noise_cancellation.BVC()
)
BVCTelephony
is optimized for the narrowband audio characteristics of SIP/PSTN calls. The call analytics dashboard logs whether each session was a web
or sip
call.
When a caller reports suspected fraud or requests a decision override, the agent collects their name, contact number, issue summary, and urgency level — all with explicit consent — then calls create_escalation()
. This writes to escalations.json
(read by the Next.js frontend) and POSTs to a webhook endpoint.
The Escalation Desk (/demo
route) shows open escalations in real time:
ESC-492716
) that the agent reads back to the callerEvery call session — web or SIP — is logged on close. The _on_close
handler fires when LiveKit closes the room:
def _on_close(ev) -> None:
call_record = {
"id": ctx.room.name,
"created_at": start_time,
"ended_at": datetime.now(timezone.utc).isoformat(),
"duration_seconds": round(duration, 2),
"success": userdata.get("success", False),
"success_reason": userdata.get("success_reason", ""),
"call_type": call_type, # 'web' or 'sip'
"user_id": userdata.get("user_id", "")
}
A call is marked successful when the caller either checks their government scheme eligibility or creates a human escalation.
The /dashboard
page shows:
This was Day 9, and probably the most elegant feature technically. When a caller needs deep, focused help with government schemes — step-by-step application guidance, documents checklist, portal navigation — the main assistant hands off to a dedicated GovernmentSchemeSpecialist
agent:
async def transfer_to_scheme_specialist(self, context: RunContext, ...) -> str:
specialist = GovernmentSchemeSpecialist(
chat_ctx=context.session.history.copy() # full conversation history
)
context.session.update_agent(specialist) # live transition, no interruption
return "Handoff complete."
The specialist receives the full chat_ctx
, so the caller never has to repeat themselves. The specialist introduces itself once, then continues the conversation in-context. It's focused: it only handles government scheme questions, and explicitly declines general banking/UPI questions.
When the LLM streams its reply in chunks, a phrase like "PM Kisan Samman Nidhi" might arrive as "PM Kisan"
in one chunk and " Samman Nidhi"
in the next. My first implementation fed each chunk directly through the regex rewriter — which meant phrase-boundary splits caused silent failures where terms stayed in Roman script.
Fix: Buffer streamed chunks and only emit text at sentence boundaries ("."
, "!"
, "?"
, "\n"
). Since scheme names never cross sentence boundaries, the rewriter always sees the full phrase. Added a safety flush at 512 characters for run-on sentences.
Early versions of the memory tools had a subtle problem: the LLM would call save_user_memory
in the same turn the caller first mentioned a fact, before any consent was sought. I fixed this with two layers:
save_user_memory
checks an in-session consent
dict before writing anything; if the consent key isn't there, it returns a detailed refusal explaining exactly what's missingThis meant the enforcement was in the code, not just in the LLM's instruction-following.
gemini-2.5-flash
mid-challenge
Around Day 7, calls started returning 404 errors. The model gemini-2.5-flash
had been deprecated. Migrating to gemini-3.5-flash-lite
via the LiveKit Inference plugin fixed it — but it required updating both agent.py
and the test harness configuration. Always pin your model versions.
My first Hinglish marker list was too broad — common words like "hai" appeared in some proper nouns — and too narrow — it missed many common code-switch patterns. I iterated through actual test conversations, adding and removing markers until detect_language()
was reliably stable across English, Hindi, and code-switched Hinglish inputs.
| Component | What it does | Used in this project |
|---|---|---|
| STT | Turns speech to text (the ears) | Deepgram nova-3, multilingual |
| LLM | Generates responses (the brain) | Gemini 3.5 Flash Lite |
| TTS | Turns text to speech (the voice) | Murf Falcon, Anisha (en-IN) |
| Transport | Real-time audio | LiveKit (WebRTC) |
The key insight: these four components are independent and swappable. You can use any STT, any LLM, any TTS — as long as they're wired through a common agent runtime (LiveKit Agents in this case).
Why Murf Falcon for TTS?
git clone https://github.com/murf-ai/murf-livekit-starter.git
cd murf-livekit-starter
Set up API keys — never commit these to git:
Create backend/.env.local
(copy from backend/.env.example
):
LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_API_KEY=your_key
LIVEKIT_API_SECRET=your_secret
MURF_API_KEY=your_murf_key # murf.ai/api/dashboard
DEEPGRAM_API_KEY=your_deepgram_key # deepgram.com
GOOGLE_API_KEY=your_google_key # aistudio.google.com
Create frontend/.env.local
(copy from frontend/.env.example
):
LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_API_KEY=your_key
LIVEKIT_API_SECRET=your_secret
Install and run:
cd backend
uv sync
uv run python src/agent.py download-files
cd ../frontend
pnpm install
chmod +x start_app.sh && ./start_app.sh
Open ** http://localhost:3000**, click
The entire personality lives in one constant at the top of backend/src/agent.py
:
SYSTEM_PROMPT = """You are RupeeGPT, a personal AI assistant for Indian users.
...
"""
Change that string and you have a completely different agent. Change the voice:
tts=murf.TTS(voice="Anisha", locale="en-IN", style="Conversation")
After ./start_app.sh
, check the backend terminal for:
[STT] user said: <transcript>
— what Deepgram heard[LLM] metrics model=gemini-3.5-flash-lite ttft=0.42s
— LLM latency[TTS] metrics ttfb=0.13s
— Murf Falcon time-to-first-byte[CALL LOG] Saved to file ...
— session logged to dashboard
Browser (Next.js)
↕ WebRTC (audio)
LiveKit Server
↕ WebRTC
Python Agent Worker
├─ Deepgram STT (nova-3, multilingual)
├─ Gemini LLM (gemini-3.5-flash-lite)
├─ Murf Falcon TTS (Anisha, en-IN)
├─ tts_hindi.py (Devanagari pronunciation rewriter)
├─ memory.py (MongoDB caller profiles)
├─ schemes.py (government scheme matching)
├─ telephony/ (SIP outbound calls)
└─ Function tools:
lookup_user()
save_user_memory()
grant_user_memory_consent()
find_eligible_schemes()
get_usd_inr_rate()
get_lending_rates()
create_escalation()
transfer_to_scheme_specialist()
Next.js frontend routes:
/ Voice agent UI
/demo Escalation Desk dashboard
/dashboard Call Analytics dashboard
Design strict tool schemas from Day 1. The OpenAI strict schema validator requires every object to declare additionalProperties: false
. Retrofitting this was painful. The _pick_arg()
helper pattern I built to handle both LLM invocations and test-harness direct calls is something I'd design in from the beginning.
Pin model versions immediately. gemini-2.5-flash
deprecating mid-challenge cost me debugging time.
Integration test with real audio early. Unit tests caught logic errors; only real voice sessions caught the chunk-splitting bug in the TTS rewriter.
🔗 GitHub: github.com/murf-ai/murf-livekit-starter
⚠️ Never publish API keys, phone numbers, caller data, or any private information. Use
.env.local
(gitignored) for all secrets.
Ten days. One voice agent. Nine features that went from zero to production-ready code:
The most important lesson: voice agents are not just chatbots with audio bolted on. The interaction model is fundamentally different — no markdown, no bullet points, short turns, immediate feedback. You have to design for listening, not reading.
And for Indian users specifically, language flexibility and natural pronunciation are the difference between a tool that feels foreign and one that feels like talking to someone who genuinely gets it.
If you're building in this space, I hope this gives you a useful foundation. The code is open, the architecture is documented, and the patterns — consent-gated memory, language-aware TTS rewriting, live handoffs — are all reusable.
Build something for Bharat. Ship it.
Built during 10 Days of Voice Agents — VoiceForBharat Edition, powered by the fastest TTS API: Murf Falcon.
Tag @MurfAI | #VoiceForBharat