{"slug": "from-midnight-power-cuts-to-multi-agent-voice-ai-how-i-built-raksha-in-10-days", "title": "From Midnight Power Cuts to Multi-Agent Voice AI: How I Built Raksha in 10 Days", "summary": "A Class 12 student built Raksha, a multi-agent voice assistant for the #VoiceForBharat Challenge, designed to protect Indian citizens from cyber scams and verify government schemes. The system, developed in 10 days, integrates speech-to-text, large language models, and telephony, with features like persistent caller memory, dynamic database queries, and human-in-the-loop escalation. Despite challenges including power cuts and hardware limitations, the developer successfully deployed the system to handle high-stress banking fraud scenarios in Hinglish.", "body_md": "Building voice AI sounds deceptively clean on paper: capture speech, stream it to an STT engine, prompt an LLM, and synthesize audio back in real time.\n\nIn reality, building a real-time, multilingual voice assistant that handles high-stress banking fraud while running on desktop hardware in 30°C heat with unstable power is a completely different story.\n\nAs a Class 12 student diving into my first-ever hackathon, I spent the last 10 days of the **#VoiceForBharat Challenge** building **Raksha**—an empathetic, multi-agent voice assistant designed to protect Indian citizens from cyber scams, verify official government schemes, and escalate active financial fraud to human coordinators.\n\nHere is the honest breakdown of how Raksha evolved from Day 1 to Day 10, the absurd hurdles along the way, and what it actually takes to build reliable voice systems.\n\nIndia's digital payments ecosystem is scaling at lightspeed, but so is financial fraud. Every day, people receive panic-inducing SMS messages claiming: *\"Your bank account is blocked. Update KYC immediately or share OTP.\"*\n\nWhen someone is panicked, reading a 5-page PDF advisory is the last thing they will do. They need immediate, reassuring spoken advice in accessible Hinglish: **\"Ghabraiye mat. Bank kabhi phone ya message par OTP nahi mangta. Yeh poori tarah fraud hai.\"**\n\nVoice is the most natural medium for this. If an AI agent can step in during those critical first 60 seconds, it can prevent immediate financial loss.\n\nDay 1 was pure chaos. I was battling 404s and gRPC hanging issues while connecting Gemini and LiveKit. Exactly 16 minutes before the midnight deadline, the power cut out completely. When it returned a minute past midnight, my desktop greeted me with a blue `SrtTrail.txt`\n\nWindows repair screen—and then another storm knocked the grid out again.\n\nI thought I was out on Day 1, but after seeing an extension on Discord, I locked back in for Day 2.\n\nBeing on a desktop without a dedicated studio mic—and honestly, being an introvert where speaking out loud repeatedly into a mic for demos felt unnatural—I found a workaround: I routed clean TTS audio from my phone directly into my earphone inline mic. It bypassed background fan noise, kept latency clean, and fed Deepgram crisp input. By Day 3, I had rebuilt the frontend with a custom slate-indigo theme, 5 distinct agent states, and LiveKit's audio-reactive WebGL aura visualizer.\n\nA voice assistant can't just be an open chatbot; it needs strict boundaries. On Day 4, I implemented persistent caller memory using SQLite. If a returning caller connects, Raksha recognizes them contextually. But in banking, privacy is non-negotiable: Raksha is strictly instructed to ask for explicit consent before saving any facts.\n\nOn Day 5, I hooked up dynamic database tools like `check_scheme_eligibility`\n\nwith spoken timestamping (*\"August 2026 ke nirdeshon ke anusaar...\"*), ensuring Raksha queries official SQLite records on the fly rather than hallucinating scheme details.\n\nDay 6 was built in 30°C+ heat with low voltage, a barely spinning ceiling fan, and a constantly beeping UPS. I integrated LiveKit SIP Telephony with Linphone to place outbound reminder calls for PMJJBY renewals, implementing an instant opt-out tool whenever STT caught *\"band karo\"*.\n\nOn Day 7, I added human-in-the-loop escalation. If a user reports active unauthorized deductions, Raksha stops acting alone. She asks: *\"Kya main aapki shikayat hamari human security team ko bhej sakti hoon?\"* Upon consent, a regex sanitizer strips 4–6 digit OTPs and 10–16 digit account numbers from the summary, dispatching a color-coded ticket (e.g., `ESC-96155`\n\n) directly to a Discord webhook desk.\n\nOn Day 8, I built an auto-refreshing telemetry command center using Flask and Chart.js, tracking call resolution rates live in SQLite without logging any private conversation text.\n\nOn Day 9, I addressed the biggest architectural bottleneck: **prompt overload**. Cramming fraud prevention, escalation rules, and insurance checklists into one system prompt caused hallucinations. I split the system into two focused agents:\n\nWhen a caller asks about insurance, Raksha initiates a tool-driven handoff (`transfer_to_scheme_specialist`\n\n), passing the conversation context across worker processes so the user never has to re-explain their situation.\n\nTo keep the conversation natural, end-to-end latency must stay well under a second. Here is the WebRTC pipeline powering Raksha:\n\n```\n[ Citizen Speaks ] ──► (WebRTC Audio Stream) ──► Deepgram Nova-3 STT\n                                                        │\n                                                        ▼\n                                             Gemini 3.5 Flash-Lite LLM\n                                                        │\n                                                  (Tool Dispatch)\n                                              ┌─────────┴─────────┐\n                                              ▼                   ▼\n                                      SQLite Database     Discord Webhook Desk\n                                     (Memory & Schemes)   (Human Escalations)\n                                              │\n                                              ▼\n                                     Murf Falcon TTS (Anisha / Samar)\n                                              │\n[ Spoken Response ] ◄─────────────────────────┘\n```\n\n`language=\"multi\"`\n\nfor code-mixed Hinglish).When transferring conversation history between agents, the incoming specialist reads the shared transcript. If system prompts lack hard identity boundaries, the incoming agent can get confused by previous greetings in the chat log. Ensuring that handoff tools explicitly define the incoming agent's persona is critical to preventing identity bleed.\n\nDuring a live recording, saying the acronym `\"PMJJBY\"`\n\nrapidly caused Deepgram to transcribe it phonetically as `\"PFJYWise\"`\n\n. Because the SQLite tool expected an exact string match, it returned *Scheme not found*.\n\n**The Lesson:** Voice AI systems cannot rely on rigid text matching. Prompts must encourage full spoken titles (e.g., *\"Pradhan Mantri Jeevan Jyoti Bima Yojana\"*), and backend tools need flexible alias mappings.\n\nTuning Voice Activity Detection (VAD) is a delicate balance. If turn detection is too sensitive, the agent interrupts natural pauses; if it is too slow, response latency feels sluggish. Pairing Silero VAD, Deepgram Nova-3, and Murf Falcon's fast streaming TTS delivered the low-latency responsiveness needed for real-time triage.\n\n```\ngit clone https://github.com/PrakhidhaChawdhury/voice-for-bharat-day1\ncd backend\npython -m venv .venv\nsource .venv/bin/activate  # On Windows: .venv\\Scripts\\activate\npip install -r requirements.txt\n```\n\nCreate a `.env.local`\n\nfile inside the `backend/`\n\ndirectory:\n\n```\nLIVEKIT_URL=wss://your-project.livekit.cloud\nLIVEKIT_API_KEY=your_key\nLIVEKIT_API_SECRET=your_secret\nDEEPGRAM_API_KEY=your_key\nMURF_API_KEY=your_key\nGOOGLE_API_KEY=your_key\nDISCORD_WEBHOOK_URL=your_discord_webhook\n# Terminal 1: Run Backend Agent\npython src/agent.py dev\n\n# Terminal 2: Run Frontend Client\ncd ../frontend\nnpm install\nnpm run dev\n```\n\nOpen `http://localhost:3000`\n\nin your browser and start a voice consultation.\n\nThis 10-day sprint proved that voice AI can be empathetic, secure, and privacy-conscious. Moving forward, the roadmap includes:\n\n*Built as part of the 10 Days of Voice Agents Challenge (#VoiceForBharat), powered by Murf AI and LiveKit.*\n\n", "url": "https://wpnews.pro/news/from-midnight-power-cuts-to-multi-agent-voice-ai-how-i-built-raksha-in-10-days", "canonical_source": "https://dev.to/prakhidha_chawdhury/from-midnight-power-cuts-to-multi-agent-voice-ai-how-i-built-raksha-in-10-days-22lf", "published_at": "2026-08-15 06:55:04+00:00", "updated_at": "2026-08-15 07:10:51.891560+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "natural-language-processing", "ai-agents", "ai-products"], "entities": ["Raksha", "Gemini", "LiveKit", "Deepgram", "SQLite", "Flask", "Chart.js", "Discord"], "alternates": {"html": "https://wpnews.pro/news/from-midnight-power-cuts-to-multi-agent-voice-ai-how-i-built-raksha-in-10-days", "markdown": "https://wpnews.pro/news/from-midnight-power-cuts-to-multi-agent-voice-ai-how-i-built-raksha-in-10-days.md", "text": "https://wpnews.pro/news/from-midnight-power-cuts-to-multi-agent-voice-ai-how-i-built-raksha-in-10-days.txt", "jsonld": "https://wpnews.pro/news/from-midnight-power-cuts-to-multi-agent-voice-ai-how-i-built-raksha-in-10-days.jsonld"}}