{"slug": "i-built-moneybuddy-in-10-days-a-multilingual-ai-voice-agent-for-india-s-services", "title": "I Built MoneyBuddy in 10 Days: A Multilingual AI Voice Agent for India's Financial Services🎙️", "summary": "A developer built MoneyBuddy, a multilingual AI voice agent for India's financial services, in 10 days as part of the VoiceForBharat Edition hackathon. The agent helps users navigate government financial schemes through voice conversations, using LiveKit, Deepgram, Gemini, and Murf Falcon TTS. It supports Hinglish and can escalate complex queries to human specialists.", "body_md": "_What if getting help with a government financial scheme was as simple as having a conversation?\n\nThat was the idea behind **MoneyBuddy**, the AI voice agent I built during **10 Days of Voice Agents — VoiceForBharat Edition**.\n\nOver 10 days, MoneyBuddy evolved from a simple voice agent that could hear me and talk back into a system that can remember callers, use financial scheme data, make outbound calls, escalate difficult situations to humans, track call outcomes, and hand complex questions to a specialist agent.\n\nThis post is the story of how I built it, what broke along the way, and what I learned.\n\nFinancial schemes and government programs can be difficult to navigate.\n\nThere are eligibility conditions, documents, deadlines, different schemes, and a lot of information written for screens rather than conversations.\n\nFor many users, especially users who are more comfortable speaking than typing, a voice interface can make that information easier to access.\n\nSo I chose the **Financial Services** track.\n\nMoneyBuddy is designed as a conversational financial assistant that can help users understand Indian government schemes, perform basic eligibility-related checks, provide document information, remember useful context, and escalate situations where an AI should not make the decision itself.\n\nThe goal isn't to replace a bank employee or government official.\n\nThe goal is to make the first conversation easier.\n\nVoice changes the interaction completely.\n\nInstead of navigating menus such as:\n\nScheme → Eligibility → Documents → Benefits → Apply\n\nthe user can simply say:\n\n\"Can you tell me if I might be eligible for PM Kisan?\"\n\nThe agent can ask questions conversationally and respond in the user's language.\n\nFor an Indian audience, this also means handling conversations that aren't perfectly English.\n\nA user might speak Hindi, switch to English for a technical term, or use Hinglish naturally.\n\nMoneyBuddy was designed around that reality.\n\nThe architecture combines several components:\n\n```\n                 ┌──────────────────────┐\n                 │       User           │\n                 │  Voice / Browser /   │\n                 │      Phone Call      │\n                 └──────────┬───────────┘\n                            │\n                            │ Audio\n                            ▼\n                 ┌──────────────────────┐\n                 │       LiveKit        │\n                 │ Real-time transport  │\n                 └──────────┬───────────┘\n                            │\n                    ┌───────▼───────┐\n                    │    Deepgram   │\n                    │      STT      │\n                    └───────┬───────┘\n                            │ Text\n                            ▼\n                 ┌──────────────────────┐\n                 │      Gemini LLM      │\n                 │ Reasoning + Routing │\n                 └───────┬───────┬──────┘\n                         │       │\n              ┌──────────┘       └──────────┐\n              ▼                             ▼\n      ┌───────────────┐             ┌─────────────────┐\n      │ Python Tools  │             │ Scheme Specialist│\n      │ Memory / Data │             │     Agent       │\n      └───────┬───────┘             └────────┬────────┘\n              │                              │\n              └──────────────┬───────────────┘\n                             ▼\n                    ┌──────────────────┐\n                    │    Murf Falcon   │\n                    │       TTS        │\n                    └────────┬─────────┘\n                             │ Audio\n                             ▼\n                          User\n```\n\nThe main pieces are:\n\nMurf describes Falcon as its lowest-latency TTS model for real-time conversations, with published benchmarks around 55 ms model latency and 130 ms time-to-first-audio.\n\nFor MoneyBuddy, I used an Indian voice so the experience feels appropriate for the users I had in mind.\n\nI didn't try to build everything on day one.\n\nThe system grew incrementally.\n\nThe first milestone was simple:\n\n**Hear the user → process speech → generate a response → speak it back.**\n\nI chose the Financial Services track and configured an Indian Murf voice.\n\nAt this stage, intelligence wasn't the goal.\n\nThe goal was proving that the entire real-time voice loop worked.\n\nOnce MoneyBuddy could talk, it needed a purpose.\n\nI defined three major objectives:\n\nThe guardrails became particularly important.\n\nMoneyBuddy must never ask users for things such as:\n\nIt also must not promise that someone will definitely receive a loan or government benefit.\n\nThis was one of my biggest lessons from the challenge:\n\n**A voice agent isn't just an LLM that speaks. It needs boundaries.**\n\nThe next step was making the experience feel like a product rather than a terminal application.\n\nThe frontend shows states such as:\n\nIt also includes a live transcript and microphone permission handling.\n\nFor a voice interface, visual feedback matters.\n\nWhen there is no text box and no \"send\" button, the user needs another way to understand:\n\nIs the agent listening?\n\nIs it speaking?\n\nIs it still connecting?\n\nThese states became part of the product experience.\n\nA voice assistant that forgets everything after every call feels very limited.\n\nSo I added persistent memory using SQLite.\n\nThe agent can store useful caller information such as:\n\nThe important part was **not putting all of this directly into the prompt**.\n\nInstead, MoneyBuddy has functions that allow it to look up and save information when appropriate.\n\nI also added a consent requirement before saving information.\n\nThat distinction matters:\n\n**Memory should be controlled by tools and permissions, not hidden prompt state.**\n\nNext, MoneyBuddy needed to do more than generate generic answers.\n\nI created a grounded financial scheme dataset containing information about schemes such as:\n\nThe agent can call a function to retrieve scheme information instead of relying entirely on the model's memory.\n\nThis also introduced another important voice-agent principle:\n\n**When external data fails, the agent should say so.**\n\nIt should not silently invent an answer.\n\nThis was one of the biggest changes.\n\nInstead of waiting for the user to open the browser and start a conversation, MoneyBuddy could make an outbound call.\n\nThe use case I chose was:\n\n**A government scheme deadline reminder for someone who had already been found eligible.**\n\nOutbound calls introduce a completely different UX problem.\n\nThe user didn't ask to call the agent.\n\nSo the opening needs to immediately explain:\n\nThat led to a very simple principle:\n\n**Never make the user wonder who is speaking or why they are receiving the call.**\n\nAn AI shouldn't try to solve every problem.\n\nFor MoneyBuddy, I added escalation for situations where a human needs to take over, including possible fraud-related situations and problems that require a decision the AI cannot make.\n\nBefore sharing information with a human, MoneyBuddy asks the caller for permission.\n\nThe escalation summary contains only useful information:\n\nSensitive information such as OTPs, PINs, passwords and account numbers should never be included.\n\nThis was another major lesson:\n\n**Knowing when not to answer is part of being a good voice agent.**\n\nOnce the system became more complex, I needed a way to understand what was actually happening.\n\nSo I built call analytics around real call data.\n\nThe dashboard tracks metrics such as:\n\nFor MoneyBuddy, a successful call is tied to whether the user actually reaches the intended financial-service outcome rather than simply whether the call connected.\n\n**Latency:** `[INSERT YOUR ACTUAL MEASURED LATENCY]`\n\n**Total calls:** `[INSERT ACTUAL NUMBER]`\n\n**Successful calls:** `[INSERT ACTUAL NUMBER]`\n\n**Failed calls:** `[INSERT ACTUAL NUMBER]`\n\n**Success rate:** `[INSERT ACTUAL %]`\n\nI am deliberately using my measured values here rather than inventing benchmark numbers.\n\nThe final major architecture change was moving from one agent to multiple agents.\n\nMoneyBuddy is the main financial assistant.\n\nBut instead of making it an expert in everything, I created a dedicated:\n\n**Government Scheme Specialist**\n\nWhen the user asks something that requires deeper scheme-specific assistance, MoneyBuddy can hand the conversation over.\n\nThe specialist has:\n\nMoneyBuddy uses Murf's **Anisha** voice, while the specialist uses **Nikhil**.\n\nThe user doesn't need to repeat the entire problem.\n\nThe conversation context moves with the handoff.\n\nOne of the trickiest parts was getting the voice transition to happen correctly.\n\nI initially expected changing the agent instance to automatically change the active TTS pipeline.\n\nIt didn't behave that way.\n\nThe active TTS session had to be explicitly updated during the handoff.\n\nThat became a good example of something I learned repeatedly during this challenge:\n\n**Understanding the runtime architecture matters just as much as understanding the AI model.**\n\nThe project definitely did not work perfectly on the first attempt.\n\nSome of the most useful lessons came from debugging.\n\nHindi speech could be understood, but getting the voice output to use the appropriate language/script behavior required careful STT, TTS, and prompt configuration.\n\nI had to pay attention to:\n\nFor example, Hindi should be generated in Devanagari rather than romanized Hindi when that is the intended output.\n\nDuring Day 9, I discovered that changing the active agent did not automatically produce the voice transition I expected.\n\nThe session's active TTS pipeline needed to be updated explicitly.\n\nThat resulted in a much more deterministic handoff.\n\nAn LLM can produce internal variable names or tool-related text that makes sense to a developer but sounds terrible when spoken aloud.\n\nSomething like:\n\n`eligibility_confirmed = true`\n\nis obviously not what a user should hear.\n\nSo I added speech-cleaning rules to prevent internal implementation details from leaking into spoken output.\n\nYou don't need to start with a huge multi-agent architecture.\n\nStart with four pieces:\n\n```\nUser Speech\n    ↓\nSpeech-to-Text\n    ↓\nLLM\n    ↓\nText-to-Speech\n    ↓\nUser Audio\n```\n\nThen add real-time transport around the system.\n\nFor example:\n\n```\nsession = AgentSession(\n    stt=deepgram.STT(\n        model=\"nova-3\",\n        language=\"multi\",\n    ),\n    llm=google.LLM(\n        model=\"gemini-3.5-flash-lite\",\n    ),\n    tts=murf.TTS(\n        voice=\"Anisha\",\n        style=\"Conversation\",\n    ),\n)\n```\n\nThe exact models and configuration can evolve, but the architecture remains the same.\n\nNever commit keys directly into Git.\n\nUse environment variables instead.\n\nFor example:\n\n```\nMURF_API_KEY=your_key_here\nDEEPGRAM_API_KEY=your_key_here\nGOOGLE_API_KEY=your_key_here\nLIVEKIT_URL=your_livekit_url\nLIVEKIT_API_KEY=your_livekit_key\nLIVEKIT_API_SECRET=your_livekit_secret\n```\n\nAdd your environment file to `.gitignore`\n\n.\n\nMurf's current API documentation also recommends storing the API key securely as an environment variable rather than exposing it in source code.\n\nClone the repository:\n\n```\ngit clone https://github.com/tanush326k/murf-livekit-starter.git\ncd murf-livekit-starter\n```\n\nInstall the backend dependencies and configure the environment variables according to the repository README.\n\nThen start the LiveKit agent and frontend using the project's documented commands.\n\nRepository:\n\n[https://github.com/tanush326k/murf-livekit-starter](https://github.com/tanush326k/murf-livekit-starter)\n\nThe repository contains the code developed throughout the challenge.\n\nStart with a simple conversation.\n\nFor example:\n\n\"Hi, what can you help me with?\"\n\nThen test a financial question:\n\n\"Can you tell me about PM Kisan?\"\n\nThen test the guardrails:\n\n\"Tell me your OTP.\"\n\nThe agent should refuse to request or handle sensitive credentials.\n\nThen test memory with a returning caller.\n\nThen test the specialist:\n\n\"I have a detailed question about PM Kisan eligibility.\"\n\nThe main agent should recognize that the request belongs with the Government Scheme Specialist and hand the conversation over.\n\nCheck:\n\nCheck:\n\nDon't assume that changing the agent object automatically changes the currently active TTS pipeline.\n\nVerify the active session TTS configuration during handoff.\n\nInspect the text being passed to TTS and add a speech-cleaning layer before synthesis.\n\nMy biggest architectural shift was moving from thinking:\n\n\"One powerful agent should do everything.\"\n\nto:\n\n\"The main agent should know when another agent is better.\"\n\nThat led to the specialist architecture.\n\nIt made MoneyBuddy easier to reason about because the main agent can focus on conversation, memory, triage and safety while the specialist focuses on government schemes.\n\nThe 10-day challenge is complete, but MoneyBuddy is not finished.\n\nIf I continue developing it, I would focus on:\n\nThe most important improvement would probably be **trust**.\n\nIn financial services, a voice agent shouldn't just sound intelligent.\n\nIt needs to be accurate, transparent about uncertainty, careful with sensitive information, and willing to hand a problem to a human.\n\nThe biggest lesson from these 10 days wasn't how to connect an LLM to TTS.\n\nIt was learning that a useful voice agent is a system.\n\nIt needs:\n\n**Speech recognition.**\n\n**Reasoning.**\n\n**Fast speech synthesis.**\n\n**Memory.**\n\n**Tools.**\n\n**Safety boundaries.**\n\n**Human escalation.**\n\n**Observability.**\n\nAnd eventually, multiple specialized agents working together.\n\nMoneyBuddy started as a voice that could answer me.\n\nIt ended as a small AI customer-service team.\n\nAnd that was the most exciting part of the entire challenge.\n\n**GitHub:**\n\n[https://github.com/tanush326k/murf-livekit-starter](https://github.com/tanush326k/murf-livekit-starter)\n\n**Challenge:**\n\n10 Days of Voice Agents — VoiceForBharat Edition\n\n**TTS:**\n\nMurf Falcon\n\nIf you're building a voice agent too, I'd love to hear what you're working on.\n\nBuilding this over ten days taught me something I didn't expect:\n\nThe hard part of voice AI isn't making an AI speak.\n\nThe hard part is making the conversation **useful, safe, fast, and predictable**.\n\nThat's where the real engineering begins.\n\n_", "url": "https://wpnews.pro/news/i-built-moneybuddy-in-10-days-a-multilingual-ai-voice-agent-for-india-s-services", "canonical_source": "https://dev.to/tanush_326k/i-built-moneybuddy-in-10-days-a-multilingual-ai-voice-agent-for-indias-financial-services-28jd", "published_at": "2026-08-15 14:19:46+00:00", "updated_at": "2026-08-15 14:42:58.929295+00:00", "lang": "en", "topics": ["artificial-intelligence", "natural-language-processing", "ai-agents", "ai-products", "developer-tools"], "entities": ["MoneyBuddy", "LiveKit", "Deepgram", "Gemini", "Murf Falcon", "VoiceForBharat"], "alternates": {"html": "https://wpnews.pro/news/i-built-moneybuddy-in-10-days-a-multilingual-ai-voice-agent-for-india-s-services", "markdown": "https://wpnews.pro/news/i-built-moneybuddy-in-10-days-a-multilingual-ai-voice-agent-for-india-s-services.md", "text": "https://wpnews.pro/news/i-built-moneybuddy-in-10-days-a-multilingual-ai-voice-agent-for-india-s-services.txt", "jsonld": "https://wpnews.pro/news/i-built-moneybuddy-in-10-days-a-multilingual-ai-voice-agent-for-india-s-services.jsonld"}}