How I built an end-to-end, multilingual financial voice AI using Murf Falcon, LiveKit Agents, Deepgram Nova-3, Google Gemini, and Next.js during the 10 Days of AI Voice Agents Challenge.
In India, financial inclusion has accelerated rapidly with UPI, digital banking, and government-backed credit initiatives. However, navigating complex interest rates, eligibility criteria for government schemes (like PM Mudra or Sukanya Samriddhi Yojana), and understanding formal banking terms remains intimidating for millions of citizensโespecially in regional and tier-2/3 heartlands where digital interfaces can be overwhelming.
Text-first interfaces fail where voice thrives.
When rural entrepreneurs or first-time bank customers have questions, they don't want to navigate complex web forms or read dense PDFs. They want to ask a direct question in their language and get an immediate, clear, spoken answer.
To solve this, I built Roshni AI (and her specialist counterpart, Vikram) โ an ultra-low latency, conversational financial assistant engineered for natural voice interactions in English, Hindi (Devanagari script), and Hinglish.
Building a real-time conversational agent requires synchronizing four core pipelines with sub-second latency:
[ ๐ค User Microphone ]
โ (WebRTC Audio Stream)
โผ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ LiveKit Agents Worker โ
โโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโ
โ
โโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโ
โผ โผ โผ
โโโโโโโโโโโโโโโ โโโโโโโโโโโโโโโ โโโโโโโโโโโโโโโโโโโ
โ Deepgram โ โโโโโบ โGoogle Geminiโ โโโโโบ โ Murf Falcon โ
โ Nova-3 โ โ (LLM) โ โ Fast TTS โ
โ (Fast STT) โ โ โ โ (Anisha / Samar)โ
โโโโโโโโโโโโโโโ โโโโโโโโฌโโโโโโโ โโโโโโโโโโฌโโโโโโโโโ
โ (Tool / Handoff) โ
โผ โผ
โโโโโโโโโโโโโโโโโ [ ๐ Audio Output ]
โ SQLite Memory โ
โ & Analytics โ
โโโโโโโโโโโโโโโโโ
language="multi"
) for instantaneous multi-language speech recognition.gemini-3.5-flash-lite
/ gemini-2.0-flash
) with strict prompt guardrails.Using Murf Falconโs conversational voices transformed the agent from a robotic IVR into a warm, approachable advisor. By pacing sentences naturally and configuring TTFB optimization, conversational latency stayed under 1 second.
A common flaw in bilingual voice bots is sending romanized Hindi (e.g., "namaste, aapka swagat hai") to TTS engines, causing English phonetic engines to mispronounce Indian syllables. I enforced strict system prompt constraints:
Always write Hindi in Devanagari script (e.g., "เคจเคฎเคธเฅเคคเฅ, เคเคชเคเคพ เคธเฅเคตเคพเคเคค เคนเฅ"). Never write romanized Hindi.
This simple prompt architecture ensured flawless Indian accent pronunciation every single turn.
Using SQLite (agent_memory.db
), Roshni stores caller identity, preferred languages, and previous queries. Returning users are greeted by name with tailored suggestions based on their recorded financial profile.
Roshni accesses custom tools like check_scheme_rates
to provide live, structured interest rate calculations for Fixed Deposits, Senior Citizen schemes, and Savings accounts without LLM hallucinations.
One agent should not try to be an expert in everything. I split responsibilities between two distinct agents:
When a caller asks about government schemes, Roshni announces: "I will connect you to Vikram, our Government Scheme Specialist", and hands over the full conversation history to Vikram in real time.
python
@function_tool
async def transfer_to_scheme_specialist(self, context: RunContext, query: str) -> str:
"""Handoff caller to Vikram for Government Scheme queries."""
self.session_state["is_success"] = True
record_call_outcome(self.call_id, "SUCCESS", "Transferred to Scheme Specialist")
return "I will connect you to Vikram, our Government Scheme Specialist."