{"slug": "building-arogya-seva-how-i-built-an-ultra-low-latency-telehealth-voice-ai-for-in", "title": "Building Arogya Seva: How I Built an Ultra-Low Latency Telehealth Voice AI for Bharat in 10 Days", "summary": "A developer built Arogya Seva, an ultra-low-latency telehealth voice AI for India, in 10 days as part of the #VoiceForBharat challenge. The system uses LiveKit Agents SDK with Deepgram Nova-3 for speech-to-text, Google Gemini 2.0 Flash for intent processing, and Murf Falcon for text-to-speech, supporting multiple Indian languages and scripts. It includes strict clinical guardrails, privacy-first memory with explicit consent, and features like symptom triage, PHC lookup, and human escalation.", "body_md": "Arogya Seva was created to bridge this gap as part of the #VoiceForBharat challenge (Track: Health Access). It is an empathetic, multilingual, real-time voice assistant designed to interact naturally in Indian English, Hindi (Devanagari script), and regional scripts.\n\nWhy Voice? For millions of non-tech-savvy users or individuals in low-literacy regions, typing in an app or filling out complex forms is a friction point. Speaking directly over a phone call or web interface is the most accessible, natural, and human way to receive guidance.\n\nThe system is built on LiveKit Agents SDK with a modular pipeline:\n\nSpeech-to-Text (STT): Deepgram Nova-3 transcribes spoken voice in real time.\n\nBrain (LLM): Google Gemini 2.0 Flash processes intent, applies clinical guardrails, and decides on function tool calls.\n\nText-to-Speech (TTS): Murf Falcon (livekit-murf plugin, voice model en-IN-Anisha) streams ultra-low latency, human-like voice synthesis back to the user.\n\nReal-time Transport: LiveKit WebRTC (web frontend) and SIP Telephony (outbound/inbound phone calls).\n\nMemory & State: SQLite (agent_memory.db) for privacy-first caller persistence and escalation management.\n\nMermaid diagram\n\n🛡️ Feature 2: Strict Guardrails & Native Script Enforcement\n\nHealth AI requires absolute safety. Arogya Seva follows strict operational boundaries:\n\nRed-Flag Clinical Emergency Protocol: Immediately flags chest pain, dyspnea, heavy bleeding, or acute trauma, urging callers to dial emergency 108.\n\nNative Script Enforcement: To ensure proper acoustic synthesis and avoid awkward transliteration, responses in Hindi are strictly produced in native Devanagari script (e.g., नमस्ते, आप कैसे हैं?), avoiding romanized \"Hinglish\".\n\n💻 Feature 3: Dynamic Frontend State & Audio Visualizers\n\nBuilt with Next.js and LiveKit Agents UI, the frontend displays real-time agent states:\n\nListening (Visualized with dynamic frequency waveforms)\n\nThinking (Tool execution state)\n\nSpeaking (Fluid audio spectrum representation)\n\n🧠 Feature 4: Privacy-First Memory with Explicit Consent\n\nReturning callers don't need to re-explain their location or age band. However, privacy is paramount:\n\nThe agent explicitly asks: \"May I save your name and basic health details so I can remember you next time?\"\n\nFacts are stored only if explicit consent is given.\n\nUsers can say \"Forget me\" at any time to wipe their records via forget_caller.\n\n🛠️ Feature 5: Real-Domain Health Tools & Tool Chaining\n\nclassify_symptom_triage: Categorizes symptoms into Self-Care / Low, Moderate / Consult Nurse, or High Urgent / Red-Flag.\n\nlookup_nearest_phc: Searches Primary Health Centres based on district.\n\nTool Chaining: Automatically reuses district information saved in caller memory without re-asking the user.\n\nGraceful Failure: If the registry API is unreachable, the agent announces the offline status calmly and provides emergency helpline 104/108 numbers.\n\n📞 Feature 6: Outbound Telephony & Mandatory Opt-Out\n\nFor automated health reminders and follow-up calls:\n\nTwo-Sentence Mandatory Opening: State WHO is calling, WHY, and HOW to opt out in the first two sentences.\n\nInstant Opt-Out: Saying \"stop calling me\" or pressing 9 immediately executes opt_out_caller in SQLite and terminates the call.\n\n🆘 Feature 7: Human Escalation & Reference IDs\n\nWhen situations exceed AI scope:\n\nAgent detects clinical doctor requests or red-flag symptoms.\n\nAgent requests explicit permission to create an escalation ticket.\n\nUpon agreement, create_escalation stores a sanitized summary (no passwords/PINs/Aadhaar) and returns a unique reference ID (e.g., ESC-8492).\n\n📊 Feature 8: Call Analytics & Outcome Tracking\n\nEvery call session logs structured metrics into SQLite, including call duration, triage classifications, escalation status, and resolution codes (triage_completed, phc_found, escalated, handed_off).\n\n🔀 Feature 9: Multi-Agent Specialist Handoff\n\nWhen callers request to schedule, modify, or cancel OPD appointments, the main agent invokes transfer_to_clinic_specialist:\n\npython\n\n@function_tool\n\nasync def transfer_to_clinic_specialist(self, context: RunContext, reason: str) -> str:\n\nspecialist = ClinicAppointmentSpecialist()\n\ncontext.session.update_agent(specialist)\n\nreturn \"Handed off conversation to Clinic and Appointment Specialist.\"\n\nThe session dynamically updates to ClinicAppointmentSpecialist, seamlessly swapping persona and toolsets without dropping the audio call!\n\nStep 1: Prerequisites\n\nPython 3.10+ & uv package manager\n\nNode.js 18+ & pnpm\n\nLiveKit Cloud account (URL, API Key, API Secret)\n\nMurf AI API Key (for Falcon TTS)\n\nDeepgram API Key (for STT)\n\nGoogle Gemini API Key (for LLM)\n\nStep 2: Clone & Configure Backend\n\nbash\n\ngit clone [https://github.com/viral-1998/VoiceOfBharat.git](https://github.com/viral-1998/VoiceOfBharat.git)\n\ncd VoiceOfBharat/backend\n\ncp .env.example .env.local\n\nAdd your API keys to backend/.env.local:\n\nenv\n\nLIVEKIT_URL=wss://your-livekit-project.livekit.cloud\n\nLIVEKIT_API_KEY=your_key\n\nLIVEKIT_API_SECRET=your_secret\n\nMURF_API_KEY=your_murf_key\n\nDEEPGRAM_API_KEY=your_deepgram_key\n\nGOOGLE_API_KEY=your_google_key\n\nStep 3: Run Backend Agent\n\nbash\n\nuv sync\n\nuv run python src/agent.py download-files # First time model download\n\nuv run python src/agent.py dev # Start live dev server\n\nStep 4: Run Frontend UI\n\nbash\n\ncd ../frontend\n\npnpm install\n\npnpm dev\n\nOpen [http://localhost:3000](http://localhost:3000) in your browser, click Connect, and start speaking to your agent!\n\npython\n\n@function_tool\n\nasync def transfer_to_clinic_specialist(\n\nself,\n\ncontext: RunContext,\n\nreason: str = \"User requested appointment booking\",\n\n) -> str:\n\n\"\"\"Transfer caller to Clinic & Appointment Specialist agent.\"\"\"\n\nspecialist = ClinicAppointmentSpecialist()\n\ncontext.session.update_agent(specialist)\n\n```\ncall_id = getattr(getattr(context, \"session\", None), \"call_id\", \"\")\nif call_id:\n    db.mark_call_success(call_id, outcome_summary=f\"Handed off: {reason}\")\n\nreturn \"Handed off conversation to Clinic and Appointment Specialist.\"\n```\n\nMulti-lingual Voice Cloning: Adding localized voice accents across 10+ Indian regional languages using Murf Falcon's voice library.\n\nWhatsApp Telemetry Notifications: Sending automated SMS/WhatsApp appointment receipts following human escalations.\n\nEHR Integration: Connecting triage outcomes directly with ABDM (Ayushman Bharat Digital Mission) health IDs.", "url": "https://wpnews.pro/news/building-arogya-seva-how-i-built-an-ultra-low-latency-telehealth-voice-ai-for-in", "canonical_source": "https://dev.to/viral1998/building-arogya-seva-how-i-built-an-ultra-low-latency-telehealth-voice-ai-for-bharat-in-10-days-1cmg", "published_at": "2026-08-15 11:20:47+00:00", "updated_at": "2026-08-15 11:41:27.610280+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-products", "ai-infrastructure", "developer-tools"], "entities": ["Arogya Seva", "LiveKit Agents SDK", "Deepgram Nova-3", "Google Gemini 2.0 Flash", "Murf Falcon", "SQLite", "Next.js", "VoiceForBharat"], "alternates": {"html": "https://wpnews.pro/news/building-arogya-seva-how-i-built-an-ultra-low-latency-telehealth-voice-ai-for-in", "markdown": "https://wpnews.pro/news/building-arogya-seva-how-i-built-an-ultra-low-latency-telehealth-voice-ai-for-in.md", "text": "https://wpnews.pro/news/building-arogya-seva-how-i-built-an-ultra-low-latency-telehealth-voice-ai-for-in.txt", "jsonld": "https://wpnews.pro/news/building-arogya-seva-how-i-built-an-ultra-low-latency-telehealth-voice-ai-for-in.jsonld"}}