Building RupeeGPT: A Multilingual Voice AI Financial Assistant for Bharat A developer built RupeeGPT, a multilingual voice AI financial assistant for Indian users, in ten days. The system uses Deepgram for speech-to-text, Gemini for language understanding, and Murf Falcon's Indian English voice for text-to-speech, with a custom pronunciation layer to correctly render Hindi scheme names. It supports English, Hindi, and Hinglish, and includes guardrails for financial safety. VoiceForBharat| Built with the fastest TTS API — Murf Falcon | 10 Days of Voice Agents Ten days ago, I started with a blank repo and a challenge: build a production-ready voice AI agent for Indian users — one that could speak naturally in English, Hindi, and Hinglish; remember returning callers; escalate to humans when things got serious; and hand off conversations to specialist agents without ever making the caller repeat themselves. What came out the other side is RupeeGPT — a conversational AI financial assistant that helps any Indian user navigate banking, UPI, government welfare schemes, loans, and financial safety. Here's everything I built, what broke, how I fixed it, and how you can build your own. India has over 500 million smartphone users, but financial literacy remains a barrier for hundreds of millions of people — especially in tier-2 and tier-3 cities and rural areas. The information exists: government scheme portals, RBI guidelines, banking apps. But it's buried in bureaucratic language, English-only interfaces, and long PDF documents. A voice agent changes that. You don't need to read anything. You don't need to know the right portal URL. You just talk . Who it's for: First-generation bank account holders, rural farmers checking PM Kisan eligibility, street vendors exploring PM SVANidhi loans, anyone who's ever been told to "read the fine print" and couldn't. Why voice: Voice meets people where they are. It removes the literacy barrier, it's faster than navigating apps, and for many rural users, calling is the most intuitive interface they know. 🎙️ User speaks → Deepgram STT nova-3, multilingual → Gemini LLM gemini-3.5-flash-lite via LiveKit Inference → Murf Falcon TTS Anisha — Indian English, en-IN → LiveKit real-time transport → 🔊 User hears The stack: uv for dependency managementThe default for most TTS-backed voice agents is a US English voice. For an Indian user asking about PM Kisan Samman Nidhi , hearing a generic American accent reading scheme names in English phonetics feels jarring and impersonal. Murf Falcon's Anisha voice — Indian English, en-IN , Conversation style — changes this completely. But there was a subtlety: even with an Indian voice, scheme names like "PM Kisan Samman Nidhi" or "Pradhan Mantri Jan Dhan Yojana" are read with English phonetics when spelled in Roman script. My fix: a TTS pronunciation layer tts hindi.py ENGLISH TO HINDI: tuple tuple str, str , ... = "pm kisan samman nidhi", "पीएम किसान सम्मान निधि" , "pm jan dhan yojana", "पीएम जन धन योजना" , "pradhan mantri jan dhan yojana", "प्रधानमंत्री जन धन योजना" , "pm svanidhi", "पीएम स्वनिधि" , "aadhaar", "आधार" , "yojana", "योजना" , ... more Before any text reaches Murf Falcon, it passes through this whitelist rewriter. Known Hindi/Indian terms are converted to Devanagari, so the voice says "पीएम किसान सम्मान निधि" — exactly as a native speaker would say it on TV — instead of "P M Kisan Samman Nidhi" with English stress patterns. The rewriter is safe to apply for every language mode: a pure-English sentence with none of these terms passes through byte-for-byte unchanged. I also built a detect language function that classifies each user utterance as english , hindi , or hinglish using Devanagari character detection and a curated Hinglish marker word list: HINGLISH MARKERS = "mujhe", "kaise", "kya", "chahiye", "baat", "namaste", "yojana", "sarkari", "paise", "rupaye", "bharat", ... The agent mirrors the caller's language — answers in Hindi if they speak Hindi, Hinglish if they code-switch — without ever asking them to repeat. The system prompt defines the entire character of RupeeGPT: what it will help with, what it refuses, and how it escalates. Key guardrails baked into the system prompt: For the two escalation scenarios, the agent must: create escalation only after consentThis pattern — ask before acting, require a clear YES — became a design principle throughout the whole project. The TTS node hooks into the LiveKit Agents pipeline using Agent.default.tts node : python async def tts node self, text, model settings : language = self. tts language async def tracked : async for part in tts hindi.stream for tts text, language=language : yield part async for frame in Agent.default.tts node self, tracked , model settings : yield frame The stream for tts function accumulates the LLM's streaming text output into complete sentences before passing each sentence through the Devanagari rewriter. This is important: if a scheme name like "PM Kisan Samman Nidhi" were split across two streamed chunks, the phrase-level rewriter would miss it. Every caller gets a persistent browser ID stored in localStorage and passed as a LiveKit participant attribute . The agent reads this at the start of every session and calls lookup user to fetch any saved profile from MongoDB. But here's the part that took the most iteration: consent architecture . The agent is not allowed to save any personal fact without: grant user memory consent with that exact value save user memory with the same value Tools must be called in sequence, only after explicit spoken consent: 1. grant user memory consent name="Rahul", ... 2. save user memory name="Rahul", ... The save user memory tool actively checks the in-session consent store and blocks saves for anything that wasn't consented to in the current call. Returning callers are greeted naturally: "Namaste Rahul, welcome back. Would you like to continue from PM Jan Dhan Yojana?" The MongoDB document looks like this: { "user id": "abc123", "name": "Rahul", "language preference": "Hinglish", "facts": { "schemes checked": "PM Jan Dhan Yojana" , "eligibility answers": { "income bracket": "below 3 lakh", "farmer": true } }, "last interaction": "2026-08-14T10:30:00Z" } Three function-calling tools give the agent live or near-live data: find eligible schemes — Matches the caller's profile age, state, income, occupation, caste, residence, disability, BPL status against a local dataset of Indian government welfare schemes. Returns preliminary matches with names, benefits, documents required, and official portal URLs. get usd inr rate — Live USD/INR exchange rate. get lending rates — Current base lending rates and MCLR data. The scheme-matching tool uses a careful LLM prompt to avoid hallucination: Using LiveKit's SIP integration, the agent can place outbound calls to real phone numbers. The session pipeline automatically detects SIP participants and switches the noise cancellation model: noise cancellation=lambda params: noise cancellation.BVCTelephony if params.participant.kind == rtc.ParticipantKind.PARTICIPANT KIND SIP else noise cancellation.BVC BVCTelephony is optimized for the narrowband audio characteristics of SIP/PSTN calls. The call analytics dashboard logs whether each session was a web or sip call. When a caller reports suspected fraud or requests a decision override, the agent collects their name, contact number, issue summary, and urgency level — all with explicit consent — then calls create escalation . This writes to escalations.json read by the Next.js frontend and POSTs to a webhook endpoint. The Escalation Desk /demo route shows open escalations in real time: ESC-492716 that the agent reads back to the callerEvery call session — web or SIP — is logged on close. The on close handler fires when LiveKit closes the room: php def on close ev - None: call record = { "id": ctx.room.name, "created at": start time, "ended at": datetime.now timezone.utc .isoformat , "duration seconds": round duration, 2 , "success": userdata.get "success", False , "success reason": userdata.get "success reason", "" , "call type": call type, 'web' or 'sip' "user id": userdata.get "user id", "" } Write to calls.json + POST to Next.js API A call is marked successful when the caller either checks their government scheme eligibility or creates a human escalation. The /dashboard page shows: This was Day 9, and probably the most elegant feature technically. When a caller needs deep, focused help with government schemes — step-by-step application guidance, documents checklist, portal navigation — the main assistant hands off to a dedicated GovernmentSchemeSpecialist agent: php async def transfer to scheme specialist self, context: RunContext, ... - str: specialist = GovernmentSchemeSpecialist chat ctx=context.session.history.copy full conversation history context.session.update agent specialist live transition, no interruption return "Handoff complete." The specialist receives the full chat ctx , so the caller never has to repeat themselves. The specialist introduces itself once, then continues the conversation in-context. It's focused: it only handles government scheme questions, and explicitly declines general banking/UPI questions. When the LLM streams its reply in chunks, a phrase like "PM Kisan Samman Nidhi" might arrive as "PM Kisan" in one chunk and " Samman Nidhi" in the next. My first implementation fed each chunk directly through the regex rewriter — which meant phrase-boundary splits caused silent failures where terms stayed in Roman script. Fix: Buffer streamed chunks and only emit text at sentence boundaries "." , " " , "?" , "\n" . Since scheme names never cross sentence boundaries, the rewriter always sees the full phrase. Added a safety flush at 512 characters for run-on sentences. Early versions of the memory tools had a subtle problem: the LLM would call save user memory in the same turn the caller first mentioned a fact, before any consent was sought. I fixed this with two layers: save user memory checks an in-session consent dict before writing anything; if the consent key isn't there, it returns a detailed refusal explaining exactly what's missingThis meant the enforcement was in the code, not just in the LLM's instruction-following. gemini-2.5-flash mid-challenge Around Day 7, calls started returning 404 errors. The model gemini-2.5-flash had been deprecated. Migrating to gemini-3.5-flash-lite via the LiveKit Inference plugin fixed it — but it required updating both agent.py and the test harness configuration. Always pin your model versions. My first Hinglish marker list was too broad — common words like "hai" appeared in some proper nouns — and too narrow — it missed many common code-switch patterns. I iterated through actual test conversations, adding and removing markers until detect language was reliably stable across English, Hindi, and code-switched Hinglish inputs. | Component | What it does | Used in this project | |---|---|---| | STT | Turns speech to text the ears | Deepgram nova-3, multilingual | | LLM | Generates responses the brain | Gemini 3.5 Flash Lite | | TTS | Turns text to speech the voice | Murf Falcon, Anisha en-IN | | Transport | Real-time audio | LiveKit WebRTC | The key insight: these four components are independent and swappable. You can use any STT, any LLM, any TTS — as long as they're wired through a common agent runtime LiveKit Agents in this case . Why Murf Falcon for TTS? git clone https://github.com/murf-ai/murf-livekit-starter.git cd murf-livekit-starter Set up API keys — never commit these to git: Create backend/.env.local copy from backend/.env.example : LIVEKIT URL=wss://your-project.livekit.cloud LIVEKIT API KEY=your key LIVEKIT API SECRET=your secret MURF API KEY=your murf key murf.ai/api/dashboard DEEPGRAM API KEY=your deepgram key deepgram.com GOOGLE API KEY=your google key aistudio.google.com Create frontend/.env.local copy from frontend/.env.example : LIVEKIT URL=wss://your-project.livekit.cloud LIVEKIT API KEY=your key LIVEKIT API SECRET=your secret Install and run: Backend Python cd backend uv sync uv run python src/agent.py download-files Frontend Node cd ../frontend pnpm install Run everything from repo root chmod +x start app.sh && ./start app.sh Open http://localhost:3000 , click The entire personality lives in one constant at the top of backend/src/agent.py : SYSTEM PROMPT = """You are RupeeGPT, a personal AI assistant for Indian users. ... """ Change that string and you have a completely different agent. Change the voice: tts=murf.TTS voice="Anisha", locale="en-IN", style="Conversation" Browse all voices: murf.ai/api/docs/voices-styles/voice-library After ./start app.sh , check the backend terminal for: STT user said: