{"slug": "building-finsaathi-a-voice-first-financial-assistant-for-bharat-10-days-of-voice", "title": "Building FinSaathi: A Voice-First Financial Assistant for Bharat 🇮🇳 10 Days of Voice Agents — VoiceForBharat Edition", "summary": "A developer built FinSaathi, a voice-first AI assistant for financial services in India, as part of the VoiceForBharat challenge. The system uses LiveKit for real-time transport, Murf Falcon for text-to-speech, and supports Hindi/Hinglish conversations, with features including government scheme eligibility checks, user memory, outbound calling, human escalation, and call analytics.", "body_md": "Over the last 10 days, I built FinSaathi, a voice-first AI assistant for the Financial Services track of the VoiceForBharat challenge.\n\nThe goal was simple: build an assistant that can talk naturally with users, understand financial and government-scheme related queries, remember relevant information, use tools, and know when a human or specialist should take over.\n\nWhat started as a basic voice agent gradually became a complete system with memory, tools, outbound calling, human escalation, call analytics, and specialist-agent handoffs.\n\n💡 The Problem\n\nFinancial and government-scheme processes can involve eligibility requirements, documents, deadlines, and complicated terminology.\n\nFor users who are more comfortable speaking than typing, voice can make these interactions much more natural.\n\nFor example, a user can simply ask:\n\n\"PMJJBY ke liye main eligible hoon?\"\n\nInstead of navigating through multiple forms, FinSaathi can understand the request, collect the required information, perform an eligibility check, and explain the result conversationally.\n\nThe goal is not to replace banks or human support, but to provide a conversational first layer of assistance and escalate situations when human help is required.\n\n🏗️ Architecture\n\nUSER\n\n│\n\n▼\n\nLiveKit\n\n│\n\n▼\n\nSpeech-to-Text\n\n│\n\n▼\n\nLLM / Agent\n\n│\n\n┌────────────┼────────────┐\n\n▼ ▼ ▼\n\nMemory Tools Escalation\n\n│ │ │\n\n└────────────┼────────────┘\n\n▼\n\nSQLite DB\n\n│\n\n┌──────┴──────┐\n\n▼ ▼\n\nHuman Support Analytics\n\nDashboard Dashboard\n\n│\n\n▼\n\nMurf Falcon\n\n│\n\n▼\n\nUSER\n\nTechnology Stack\n\nComponent\n\nTechnology\n\nFrontend\n\nNext.js / React\n\nAI Agent\n\nLiveKit Agents\n\nReal-time Transport\n\nLiveKit\n\nText-to-Speech\n\nMurf Falcon\n\nBackend\n\nPython\n\nAPI\n\nFastAPI\n\nDatabase\n\nSQLite\n\nCalling\n\nSIP / LiveKit\n\n🎙️ Key Features\n\n- Indian Voice & Natural Conversations\nFinSaathi uses Murf Falcon for text-to-speech and supports natural Hindi/Hinglish conversations.\nThe goal was to make the interaction feel more like talking to an assistant rather than interacting with a traditional chatbot.\n- Safety Guardrails\nFinancial conversations require strong safety boundaries.\nFor example, when a user reports an unauthorized transaction, FinSaathi warns them not to share:\nOTP\nPIN\nPassword\nCVV\nCard details\nThe agent also clearly explains when it does not have direct access to a user's bank account or transaction system.\n- User Memory\nFinSaathi stores relevant user information in a SQLite database so it can be reused in future conversations.\nThe database logic is separated into its own layer:\nAgent\n↓\ndatabase.py\n↓\nSQLite\n↓\nusers table\nThis made it easier to extend the project without mixing database operations throughout the agent code.\n- Government Scheme Eligibility\nFinSaathi can use tools to perform government-scheme eligibility checks.\nUser asks about scheme\n↓\nCollect required information\n↓\nEligibility tool\n↓\nEligible / Not Eligible\n↓\nDatabase\n↓\nUser receives result\nThe eligibility result can also be used to determine whether the user's intended task was successfully completed.\n- Outbound Calling\nFinSaathi can proactively contact eligible users using outbound calling.\nDatabase\n↓\nCheck deadlines / eligible users\n↓\nOutbound calling logic\n↓\nSIP / LiveKit\n↓\nAI Agent\n↓\nUser\nSetting up and debugging the SIP and LiveKit calling flow was one of the most challenging parts of the project.\n- Human Escalation\nFinSaathi can recognize situations where human support is more appropriate.\nFor example, for an unauthorized transaction, the agent can ask:\n\"Agar aap chahein, toh main aapki situation ke liye ek human support request create kar sakti hoon.\"\nAfter the user gives consent, the system creates an escalation and generates a unique reference ID.\nExample:\nFS-A5323F\nThe request is stored with information such as:\nReason: possible_fraud\nUrgency: HIGH\nStatus: OPEN\nPreferred Follow-up: phone\nThe human-support dashboard then allows the request to move through:\nOPEN\n↓\nIN_PROGRESS\n↓\nRESOLVED\n- Call Analytics\nFinSaathi also records the outcome of conversations.\nFor this project, a successful interaction means that the user's intended task was completed, such as completing an eligibility check or receiving the required information.\nThe dashboard displays:\nTotal Calls\nSuccessful Calls\nFailed Calls\nThese values come from actual browser or SIP interactions rather than hardcoded numbers.\nActual Call\n↓\nDetermine Outcome\n↓\ncall_analytics\n↓\nFastAPI\n↓\nAnalytics Dashboard\nA failed call does not necessarily mean a technical failure. For example, if a user leaves before completing an eligibility check, the interaction can be recorded as unsuccessful.\n- Specialist Agent Handoff\nInstead of making one agent responsible for every type of question, FinSaathi can hand government-scheme related conversations to a Government Scheme Specialist.\nMain Agent\n│\nGovernment scheme?\n│\nYES\n↓\nGovernment Scheme Specialist\n│\n↓\nEligibility / Documents /\nBenefits / Scheme Information\nThis makes the architecture more modular and allows additional specialist agents to be added in the future.\n🧩 Challenges I Faced\nThe project definitely did not work perfectly on the first attempt.\nOne of the biggest challenges was real-time calling and LiveKit/SIP integration.\nDuring testing, I encountered errors such as:\nWinError 64\nConnectionResetError\nDuplexClosed\nI also faced an outbound-call issue where the AI would start speaking but the call could terminate before the conversation continued.\nDebugging this required checking:\nLiveKit worker lifecycle\nSIP configuration\nNetwork connections\nAgent processes\nCall state\nReal-time transport\nThis taught me that building a voice agent is not only about the LLM. The real-time infrastructure around the agent matters just as much.\nAnother important lesson was separating responsibilities between the agent, database, API, and frontend.\nFor example:\nAgent → database.py → SQLite\nwhile the human-support dashboard uses:\nFrontend → FastAPI → database.py → SQLite\nThis separation made the system easier to extend and debug.🚀 What I Learned\nBefore this challenge, I thought building a voice agent was mainly:\nSpeech → AI → Speech\nAfter these 10 days, I realized that a useful voice agent needs much more:\nVoice\n+\nLLM\n+\nMemory\n+\nTools\n+\nSafety\n+\nReal-time communication\n+\nDatabase\n+\nHuman escalation\n+\nAnalytics\n+\nSpecialist handoffs\nThe biggest lesson for me was:\nA useful voice agent isn't just an AI that can talk. It's a system that can understand, act, remember, measure its performance, and know when a human or specialist should take over.\nBuilding FinSaathi during the 10 Days of Voice Agents — VoiceForBharat Edition gave me hands-on experience with all of these pieces.", "url": "https://wpnews.pro/news/building-finsaathi-a-voice-first-financial-assistant-for-bharat-10-days-of-voice", "canonical_source": "https://dev.to/nipun_goel_720eefc9d5f127/building-finsaathi-a-voice-first-financial-assistant-for-bharat-10-days-of-voice-agents--2f94", "published_at": "2026-08-14 18:21:12+00:00", "updated_at": "2026-08-14 19:05:55.812751+00:00", "lang": "en", "topics": ["artificial-intelligence", "natural-language-processing", "ai-agents", "ai-products", "developer-tools"], "entities": ["FinSaathi", "LiveKit", "Murf Falcon", "VoiceForBharat", "Next.js", "FastAPI", "SQLite", "SIP"], "alternates": {"html": "https://wpnews.pro/news/building-finsaathi-a-voice-first-financial-assistant-for-bharat-10-days-of-voice", "markdown": "https://wpnews.pro/news/building-finsaathi-a-voice-first-financial-assistant-for-bharat-10-days-of-voice.md", "text": "https://wpnews.pro/news/building-finsaathi-a-voice-first-financial-assistant-for-bharat-10-days-of-voice.txt", "jsonld": "https://wpnews.pro/news/building-finsaathi-a-voice-first-financial-assistant-for-bharat-10-days-of-voice.jsonld"}}